Virtual power plant online optimization scheduling method based on deep reinforcement learning algorithm

Abstract

The invention discloses a virtual power plant online optimization scheduling method based on a deep reinforcement learning algorithm. The method comprises the following steps: S1, collecting historical operation data of a virtual power plant; s2, establishing an inner and outer layer comprehensive optimization scheduling model, defining related parameters of a deep reinforcement learning DDPG algorithm at the outer layer, adding a Lyapunov drift distance, and establishing an optimization model at the inner layer; s3, using historical data to train a DDPG algorithm until convergence; s4, collecting real-time operation data of the virtual power plant; s5, on the basis of the collected real-time data, calling a maturely trained DDPG algorithm to carry out optimization scheduling on resources with high time sequence; s6, solving an inner-layer optimization model to schedule residual wind power photovoltaic resources, controllable loads and other resources; and S7, integrating the action of the DDPG algorithm and the result of the optimization model to obtain a final scheduling scheme, and obtaining the optimal economic benefit. According to the method, the schedulable resources in the virtual power plant are subjected to sequential decoupling, the deep reinforcement learning algorithm and the optimization model are combined, and the online optimization scheduling efficiency of the virtual power plant is improved.

Description

Technical Field

[0001] The present invention relates to a virtual power plant online optimization scheduling method, which combines a deep reinforcement learning algorithm with a traditional optimization model and takes into account the Lyapunov optimization method in queuing theory to achieve real-time optimal scheduling of various resources within the virtual power plant. The present invention belongs to a new method in the field of virtual power plant optimization scheduling. Background Art

[0002] With the depletion of traditional energy and the urgent need for environmental protection, coupled with the continuous upgrading of the national economy's demand for electricity, research on the development and utilization of renewable energy has been deepening in recent years. However, the traditional power system is facing many challenges. On the one hand, with the widespread access to renewable energy generation, its serious uncertainty has brought tremendous pressure to the optimization and dispatching of the power system. On the other hand, the increasingly diverse electricity demand has made how to continue to ensure power quality and safe power supply an important goal in power system construction.

[0003] As a method of integrating and managing regional distributed resources, Virtual Power Plant (VPP) can aggregate distributed power sources of different types and locations for virtual power generation through advanced measurement, communication and control technologies. At the same time, VPP optimizes the scheduling of various internal resources to achieve coordinated operation of the source and load sides, so that new energy power generation resources can be reasonably allocated and efficiently utilized.

[0004] However, the complexity of various resources within the VPP has brought tremendous pressure to its optimization and scheduling process. The efficient optimization and scheduling method of source-load coordination has important practical significance and application value. At present, there are two main methods for optimizing the scheduling of VPP. One is offline scheduling based on complete information conditions, which usually includes using optimization models for intraday optimization based on historical experience data, or using model predictive control (MPC) for rolling forecast optimization. The other is online optimization scheduling based on real-time information, such as online scheduling using deep reinforcement learning algorithms. Summary of the invention

[0005] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide an online optimization scheduling method that considers the temporal decoupling of various resources within the VPP. The present invention combines the traditional optimization model with the deep reinforcement learning method. Specifically, the relevant resources with temporal sequence are scheduled online using the DDPG algorithm in deep reinforcement learning, while the resources with less temporal correlation and flexible scheduling are directly optimized using the optimization model. This method aims to reduce the computational complexity of online optimization and improve the efficiency of online optimization scheduling.

[0006] To achieve the above object, the present invention proposes an online optimization scheduling method based on deep reinforcement learning, the steps of which include:

[0007] Step 1: Collect historical operation data of the virtual power plant, including historical power consumption, electricity prices, and wind and photovoltaic output of the virtual power plant. In order to train the DDPG algorithm of deep reinforcement learning, it is necessary to collect historical operation data of the virtual power plant, mainly including six data items: average wind speed, light intensity, controllable load, basic load, electricity purchase price and electricity sales price in the virtual power plant area. The intelligent agent of the DDPG algorithm will be trained based on historical data in the future.

[0008] Step 2: Establish an internal and external double-layer comprehensive optimization scheduling model. First, define the action set and action space, state set and state space, rewards, and discount rates of the deep reinforcement learning algorithm in the outer layer of the model. Secondly, add Lyapunov drift to the reward of the intelligent agent to maintain the long-term stability of online scheduling. Finally, define the optimization model in the inner layer of the model, and define the objective function, decision variables, and constraints;

[0009] Step 3: Collect real-time operation data of the virtual power plant, including power consumption, electricity prices, and wind and photovoltaic output in the current period. Similar to the information collected in step S1, before conducting deep reinforcement learning intelligent scheduling, it is necessary to collect real-time operation data of the virtual power plant in advance and make decisions through the mature model trained in step S2.

[0010] Step 4: Use the DDPG algorithm to optimize resources with high timing, including the charging and discharging actions of energy storage devices and thermal power units.

[0011] Step 5: Based on the optimization results of the DDPG algorithm, a traditional optimization model is established to dispatch the remaining wind power, photovoltaic power, controllable load and other resources.

[0012] Step 6: Integrate the actions of the DDPG algorithm with the results of the optimization model to obtain the final scheduling plan and achieve the optimal economic benefits.

[0013] Furthermore, the agent reward in step 2 includes cost 1 and cost 2; the objective function of the optimization model includes cost 3, cost 4, cost 5, and cost 6;

[0014] Cost 1: Thermal power dispatching cost

[0015] The dispatch cost of thermal power units C t d It consists of 3 parts:

[0016]

[0017] is the power generation cost of thermal power units, a t , b t 、c tThey are the constant term, linear term and quadratic term coefficients of power generation cost respectively. is the startup cost of the thermal power unit, k d1 is the startup cost coefficient, ΔT stop The time interval from the last shutdown. is the shutdown cost of the thermal power unit, k d2 is the downtime cost coefficient, ΔT start It is the interval time from the last startup, that is, the length of time the thermal power unit has been running.

[0018] Cost 2: Energy storage dispatch cost

[0019] Considering factors such as the life and depreciation of energy storage equipment, the dispatch cost of energy storage equipment needs to be considered. Its charging and discharging power with energy storage equipment It is a linear relationship and can be expressed as:

[0020]

[0021] Among them, k ess It is the electricity cost of the energy storage device.

[0022] Cost 3: Wind power generation cost

[0023] Wind power costs Including depreciation cost of wind turbines Scheduling costs Wind curtailment penalty costs

[0024]

[0025] Among them, r is the annual interest rate, is the installation cost per unit capacity of wind turbines, K w is the capacity factor of the wind turbine, n w is the operating life of the wind turbine, K wom is the maintenance and operation cost coefficient of the wind turbine. w,1 With λ w,2 They are the linear and quadratic coefficients of wind curtailment penalty respectively.

[0026] Cost 4: Photovoltaic power generation cost

[0027] Photovoltaic power generation cost C t p Including depreciation cost of wind turbines Scheduling costs Abandonment penalty cost

[0028]

[0029] Among them, r is the annual interest rate, is the installation cost per unit capacity of photovoltaic cells, K p is the photovoltaic capacity factor, n p is the operating life of the photovoltaic, K pom is the photovoltaic maintenance and operation cost coefficient. p,1 With λ p,2 are the linear and quadratic coefficients of the light abandonment penalty respectively.

[0030] Cost 5: External power grid interaction cost

[0031] During the VPP dispatching process, it can interact with the external power grid. When the internal load demand of the VPP is too large, it can choose to purchase electricity from the external power grid; when the internal power generation of the VPP is in excess, it can choose to sell electricity to the external power grid to obtain revenue. The cost of the interaction between the VPP and the external power grid can be expressed as:

[0032]

[0033] in, is the electricity purchase price at time t, is the electricity price at time t, is the interactive power. When VPP purchases electricity from the external power grid, When VPP sells electricity to external power grid.

[0034] Cost 6: Load reduction costs

[0035] VPP operators can reduce some of the load demand, but at the same time they need to give users corresponding compensation. The calculation can be expressed as:

[0036]

[0037] Among them, β is the compensation cost per unit reduction, is the load reduction amount.

[0038] Furthermore, in step 2, the action set and action space of the DDPG algorithm agent are defined.

[0039] Considering the decision-making with time sequence, the action of the agent is:

[0040]

[0041] in, is the power generation of the thermal power unit at time t. is the charge and discharge amount of the energy storage device at time t. When the energy storage device is charged, In order to facilitate subsequent analysis, the start and stop variables of the thermal power unit are defined as follows:

[0042]

[0043] use Represents the start and stop status of the thermal power unit at time t. When the thermal power unit is in the starting state, The thermal power unit is shut down.

[0044] Define the action space of the energy storage device in the above actions:

[0045]

[0046] in, It is the upper limit of the charging and discharging power of the energy storage device.

[0047] Further consider the action space of thermal power units:

[0048]

[0049] in, and are the lower and upper limits of the power generation capacity of thermal power units, DR d is the lower limit of the thermal power unit’s slow-down, UR d is the upper limit of the thermal power unit’s climbing slope. In the process of algorithm solution, the action space of the thermal power unit will change with the power generation power at time t-1. The dynamic changes mean that the agent has different action spaces when taking actions due to the different actions at the previous moment.

[0050] At the same time, define the state set and state transition of the DDPG algorithm agent in step three.

[0051] Agent state set

[0052] The state of the agent will have an important impact on its decision-making. According to the specific scenario studied in this problem, since the action of the agent is only a two-dimensional variable, the dynamic changes of various physical quantities can be fully considered in the state set. The state of the agent at time t is defined as:

[0053]

[0054] in, is the maximum wind power at time t, is the maximum photovoltaic power at time t, is the maximum controllable load at time t, is the basic load at time t, SOC t is the state of charge of the energy storage device at time t, is the electricity purchase price at time t, is the electricity price at time t

[0055] The agent will make decisions based on the 7 states in the state set. In the state set, except SOC t All other states will be passed to the inner part of the model, and the optimization model will be used to make further decisions and determine the actions of the flexibly scheduled equipment.

[0056] Energy storage device state transfer

[0057] Energy storage device SOC t The state transition process is:

[0058]

[0059] Among them, η ch and η dis are the charge and discharge efficiencies respectively, and C is the capacity of the energy storage device in the VPP.

[0060] Wind power and photovoltaic state transfer

[0061] On the source side, the maximum power generation of wind power and photovoltaic power is related to specific wind and solar data. Therefore, the public historical data set is used to predict the maximum power generation of wind power and photovoltaic power based on wind speed and light intensity.

[0062] For wind power generation, the wind speed v at time t is used. t , the upper limit of wind power generation power is calculated

[0063]

[0064] in, is the rated power of the fan, ν ci is the wind turbine cut-in wind speed, ν co is the fan cut-out wind speed, ν n is the rated wind speed of the fan.

[0065] For photovoltaic power generation, the light intensity θ at time t is used t , the upper limit of photovoltaic power generation can be calculated

[0066]

[0067] Among them, η PV is the photovoltaic conversion efficiency, S PV is the photovoltaic equipment area.

[0068] When training the DRL algorithm, the intraday scenery data will be randomly sampled to simulate the real environment.

[0069] In addition, define the reward function of the DDPG algorithm agent in step 3.

[0070] The reward of the agent includes the negative of cost 1 and cost 2 defined in step 1. At the same time, since the present invention belongs to an online scheduling method, the sequence stability problem of long-term scheduling also needs to be considered. For this reason, the Lyapunov drift indicator needs to be introduced in the reward function of the agent to ensure the stability of the sequence in the long-term scheduling process.

[0071] This paper defines a virtual queue based on the state of charge (SOC) of energy storage devices, and achieves the constraint of long-term state of charge stability by introducing a drift indicator of the queue length and minimizing it.

[0072] According to the energy storage device state transfer calculation formula, assuming that the initial state of charge of the energy storage device is SOC o At the same time, to facilitate subsequent derivation, the SOC change calculation formula is defined as:

[0073]

[0074] By long-term accumulation of the charge and discharge change formula, we can get:

[0075]

[0076] Due to SOC T Need to meet SOC T ∈[SOC min ,SOC max ], then finding the limit of T yields:

[0077]

[0078] The above formula indicates that the SOC can be stabilized within the constraint range only when the long-term total charge capacity of the energy storage device is equal to the long-term total discharge capacity. Therefore, the virtual queue can be further defined as:

[0079] Q t =SOC t -ξ

[0080] The constant ξ is:

[0081]

[0082] in, is the maximum charge and discharge power of the energy storage device, and V is the weight coefficient. Based on this virtual queue, the long-term stability condition of the energy storage device can be defined as:

[0083]

[0084] The Lyapunov function that defines the queue length is:

[0085]

[0086] Then the Lyapunov drift index can be introduced as:

[0087]

[0088] When minimizing the drift is used as the objective function, the long-term stability of the queue can be achieved. Therefore, by adding the Lyapunov drift to the reward function of the intelligent agent, the long-term stability of the online scheduling can be achieved.

[0089] Finally, in step 2, it is also necessary to define the reward discount rate of the DDPG algorithm agent. The present invention takes the discount rate γ = 0.98 during calculation.

[0090] Furthermore, in step 2, the decision variables, objective function, and constraint terms of the optimization model need to be defined.

[0091] Objective Function

[0092] The objective function of the inner optimization part is to minimize the scheduling cost of the remaining scheduling resources:

[0093]

[0094] in, The cost of wind power generation is is the photovoltaic power generation cost, is the cost of interacting with the external power grid, The cost of reducing the load. These four costs have been defined in detail in step 1.

[0095] Constraints

[0096] Power balance constraints

[0097] Considering the power balance of various resources within the VPP, VPP operators need to meet the following constraints during actual scheduling:

[0098]

[0099] The left side of the equation is the power generation part, and the right side of the equation is the power consumption part, where: and It is the action of the outer part. It is one of the states of the outer layer. When making optimization decisions for the inner layer, all three are given constants. is the decision variable for inner optimization.

[0100] Renewable energy generation ceiling constraints

[0101] Considering that wind power and photovoltaic power generation are both limited by factors such as wind speed and light intensity, their power generation must meet the following upper limit constraints:

[0102]

[0103] in, and They are the upper limits of wind power and photovoltaic power generation, that is, the current state transmitted from the outer layer to the inner layer.

[0104] Controllable load reduction limit

[0105] The upper limit of controllable load reduction satisfies the following formula:

[0106]

[0107] in, Likewise, the outer layer passes the current state to the inner layer.

[0108] Upper limit of wind and solar power curtailment

[0109] The upper limit of wind and solar power abandonment satisfies the following formula:

[0110]

[0111] in, and These are the maximum waste ratios of wind power and photovoltaic power generation respectively.

[0112] Interactive power limit

[0113] The amount of electricity exchanged with the external power grid must meet the following constraints:

[0114]

[0115] in, is the maximum value of electricity sold to the external power grid, It is the maximum amount of electricity purchased from the external power grid.

[0116] This paper uses the DDPG algorithm in deep reinforcement learning, combines the traditional optimization model with the Lyapunov optimization in queuing theory, and proposes an online comprehensive optimization scheduling method with both internal and external layers, aiming to efficiently schedule various resources within the VPP.

[0117] The advantages of the present invention are: by performing temporal decoupling of dispatchable resources in a virtual power plant and combining a deep reinforcement learning algorithm with an optimization model, the online optimization scheduling efficiency of the virtual power plant is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0118] Figure 1 It is a flow chart of applying the method of the present invention.

[0119] Figure 2 It is a structural block diagram of the overall comprehensive scheduling model based on the deep reinforcement learning algorithm and optimization model established by the present invention.

[0120] Figure 3 This is a training flow chart of the DDPG algorithm used in the present invention. DETAILED DESCRIPTION

[0121] The present invention is further described below in conjunction with the accompanying drawings.

[0122] Reference Figure 1 to Figure 3 The virtual power plant online optimization scheduling method based on deep reinforcement learning algorithm proposed in the present invention comprises the following steps:

[0123] S1: Collect historical operation data of virtual power plants, including historical power consumption, electricity prices, and wind and photovoltaic output of virtual power plants. In order to train the DDPG algorithm of deep reinforcement learning, it is necessary to collect historical operation data of virtual power plants, mainly including six data items: average wind speed, light intensity, controllable load, basic load, electricity purchase price and electricity sales price in the area of ​​virtual power plants. The intelligent agent of DDPG algorithm will be trained based on historical data in the future.

[0124] S2: Establish an internal and external double-layer comprehensive optimization scheduling model. First, define the action set and action space, state set and state space, rewards, and discount rates of the deep reinforcement learning algorithm in the outer layer of the model. Secondly, add Lyapunov drift to the agent's reward to maintain the long-term stability of online scheduling. Finally, define the optimization model in the inner layer of the model, and define the objective function, decision variables, and constraints;

[0125] S2-1: The scheduling characteristics of various resources within the VPP are classified according to their relationship with timing, and the resources with timing are scheduled externally.

[0126] Resources with time series include energy storage equipment and thermal power units. The scheduling of energy storage systems needs to take into account the dynamic changes in power supply and demand, and maximize the overall benefits over a longer time scale by charging and discharging at different times. For thermal power equipment such as small gas turbines, it is necessary to consider the constraints and additional costs during the ramp-up and ramp-down process. In view of the characteristics of such resources, deep reinforcement learning methods can be used to achieve sequential decision-making by constructing a Markov decision process (MDP).

[0127] Resources that are not time-sequential and have flexible scheduling include loads that can be cut, wind power equipment, photovoltaic equipment, etc. The economic benefits of such equipment are independent of the previous moment, and the overall benefits can be maximized based only on the current state, including electricity prices, wind and solar ceilings, etc. Therefore, the optimal scheduling of this part of resources can be directly solved by establishing an optimization model to obtain the optimal scheduling plan.

[0128] S2-2: Define the action set and action space of the DDPG algorithm agent.

[0129] Considering the decision-making with time sequence, the action of the agent is:

[0130]

[0131] in, is the power generation of the thermal power unit at time t. is the charge and discharge amount of the energy storage device at time t. When the energy storage device is charged, In order to facilitate subsequent analysis, the start and stop variables of the thermal power unit are defined as follows:

[0132]

[0133] use Represents the start and stop status of the thermal power unit at time t. When the thermal power unit is in the starting state, The thermal power unit is shut down.

[0134] Define the action space of the energy storage device in the above actions:

[0135]

[0136] in, It is the upper limit of the charging and discharging power of the energy storage device.

[0137] Further consider the action space of thermal power units:

[0138]

[0139] in, and are the lower and upper limits of the power generation capacity of thermal power units, DR d is the lower limit of the thermal power unit’s slow-down, UR d is the upper limit of the thermal power unit’s climbing slope. In the process of algorithm solution, the action space of the thermal power unit will change with the power generation power at time t-1. The dynamic changes mean that the agent has different action spaces when taking actions due to the different actions at the previous moment.

[0140] S2-3: Define the state set and state transition of the DDPG algorithm agent.

[0141] Agent state set

[0142] The state of the agent will have an important impact on its decision-making. According to the specific scenario studied in this problem, since the action of the agent is only a two-dimensional variable, the dynamic changes of various physical quantities can be fully considered in the state set. The state of the agent at time t is defined as:

[0143]

[0144] in, is the maximum wind power at time t, is the maximum photovoltaic power at time t, is the maximum controllable load at time t, is the basic load at time t, SOC t is the state of charge of the energy storage device at time t, is the electricity purchase price at time t, is the electricity price at time t

[0145] The agent will make decisions based on the 7 states in the state set. In the state set, except SOC t All other states will be passed to the inner part of the model, and the optimization model will be used to make further decisions and determine the actions of the flexibly scheduled equipment.

[0146] Energy storage device state transfer

[0147] Energy storage device SOC t The state transition process is:

[0148]

[0149] Among them, η ch and η dis are the charge and discharge efficiencies respectively, and C is the capacity of the energy storage device in the VPP.

[0150] Wind power and photovoltaic state transfer

[0151] On the source side, the maximum power generation of wind power and photovoltaic power is related to specific wind and solar data. Therefore, the public historical data set is used to predict the maximum power generation of wind power and photovoltaic power based on wind speed and light intensity.

[0152] For wind power generation, the wind speed v at time t is used. t , the upper limit of wind power generation power is calculated

[0153]

[0154] in, is the rated power of the fan, ν ci is the wind turbine cut-in wind speed, ν co is the fan cut-out wind speed, ν n is the rated wind speed of the fan.

[0155] For photovoltaic power generation, the light intensity θ at time t is used t , the upper limit of photovoltaic power generation can be calculated

[0156]

[0157] Among them, η PV is the photovoltaic conversion efficiency, S PV is the photovoltaic equipment area.

[0158] When training the DRL algorithm, the intraday scenery data will be randomly sampled to simulate the real environment.

[0159] S2-4: Define the reward function of the DDPG algorithm agent.

[0160] The reward function of the DDPG agent is the negative of the thermal power dispatching cost and the energy storage dispatching cost, plus the objective function value of the subsequent inner optimization model and the Lyapunov drift.

[0161] Thermal power dispatch cost

[0162] Dispatch cost of thermal power units It consists of 3 parts:

[0163]

[0164]

[0165]

[0166] is the power generation cost of thermal power units, a t , b t 、c t They are the constant term, linear term and quadratic term coefficients of power generation cost respectively. is the startup cost of the thermal power unit, k d1 is the startup cost coefficient, ΔT stop The time interval from the last shutdown. is the shutdown cost of the thermal power unit, k d2 is the downtime cost coefficient, ΔT start It is the interval time from the last startup, that is, the length of time the thermal power unit has been running.

[0167] Energy storage dispatch costs

[0168] Considering factors such as the life and depreciation of energy storage equipment, the dispatch cost of energy storage equipment needs to be considered. Its charging and discharging power with energy storage equipment It is a linear relationship and can be expressed as:

[0169]

[0170] Among them, k ess It is the electricity cost of the energy storage device.

[0171] S2-5: Define the reward discount rate for the DDPG algorithm agent.

[0172] In DRL, the discount rate is the attenuation of future rewards. The smaller the discount rate, the smaller the impact of future rewards on current decisions, and the agent will consider the current situation more when making decisions; the larger the discount rate, the greater the impact of future rewards on the agent when making decisions. Since in the comprehensive scheduling model established in this paper, the flexible scheduling resources are solved in the inner optimization model, the actions of the outer agent have a strong temporal nature, and the discount rate should be set as large as possible. Therefore, the method of the present invention takes the discount rate γ = 0.98 when calculating.

[0173] S2-6: Introduce the Lyapunov drift indicator into the agent’s reward function to ensure the stability of the sequence during long-term scheduling.

[0174] The virtual queue is defined by the state of charge (SOC) of the energy storage device, and the long-term state of charge stability constraint is achieved by introducing a drift indicator of the queue length and minimizing it.

[0175] According to the energy storage device state transfer calculation formula, assuming that the initial state of charge of the energy storage device is SOC o At the same time, to facilitate subsequent derivation, the SOC change calculation formula is defined as:

[0176]

[0177] By long-term accumulation of the charge and discharge change formula, we can get:

[0178]

[0179] Due to SOC T Need to meet SOC T ∈[SOC min ,SOC max ], then finding the limit of T yields:

[0180]

[0181] The above formula indicates that the SOC can be stabilized within the constraint range only when the long-term total charge capacity of the energy storage device is equal to the long-term total discharge capacity. Therefore, the virtual queue can be further defined as:

[0182] Q t =SOC t -ξ

[0183] The constant ξ is:

[0184]

[0185] in, is the maximum charge and discharge power of the energy storage device, and V is the weight coefficient. Based on this virtual queue, the long-term stability condition of the energy storage device can be defined as:

[0186]

[0187] The Lyapunov function that defines the queue length is:

[0188]

[0189] Then the Lyapunov drift index can be introduced as:

[0190]

[0191] When minimizing the drift is used as the objective function, the long-term stability of the queue can be achieved. Therefore, by adding the Lyapunov drift to the reward function of the intelligent agent, the long-term stability of the online scheduling can be achieved.

[0192] S2-7: Define the decision variables of the optimization model.

[0193] After the outer layer makes a decision, the inner layer will further solve the optimal action for the remaining resources in the VPP based on the actions taken by the agent. Specifically, the outer layer optimizes the scheduling of time-series resources, while the inner layer further optimizes the scheduling of relatively flexible resources.

[0194] Therefore, the decision variable for inner optimization is the actual wind power generation Photovoltaic power generation Interaction power with external power grid Load reduction of controllable load

[0195] S2-8: Define the objective function of the optimization model.

[0196] The objective function of the inner optimization part is to minimize the scheduling cost of the remaining scheduling resources:

[0197]

[0198] in, The cost of wind power generation is is the photovoltaic power generation cost, is the cost of interacting with the external power grid, To reduce the cost of load.

[0199] Wind power costs

[0200] Wind power costs Including depreciation cost of wind turbines Scheduling costs Wind curtailment penalty costs

[0201]

[0202]

[0203]

[0204] Among them, r is the annual interest rate, is the installation cost per unit capacity of wind turbines, K w is the capacity factor of the wind turbine, n w is the operating life of the wind turbine, K wom is the maintenance and operation cost coefficient of the wind turbine. w,1 With λ w,2 They are the linear and quadratic coefficients of wind curtailment penalty respectively.

[0205] Photovoltaic power generation costs

[0206] Photovoltaic power generation costs Including depreciation cost of wind turbines Scheduling costs Abandonment penalty cost

[0207]

[0208]

[0209]

[0210]

[0211] Among them, r is the annual interest rate, is the installation cost per unit capacity of photovoltaic cells, K p is the photovoltaic capacity factor, n p is the operating life of the photovoltaic, K pom is the photovoltaic maintenance and operation cost coefficient. p,1 With λ p,2are the linear and quadratic coefficients of the light abandonment penalty respectively.

[0212] External grid interaction costs

[0213] During the VPP dispatching process, it can interact with the external power grid. When the internal load demand of the VPP is too large, it can choose to purchase electricity from the external power grid; when the internal power generation of the VPP is in excess, it can choose to sell electricity to the external power grid to obtain revenue. The cost of the interaction between the VPP and the external power grid can be expressed as:

[0214]

[0215] in, is the electricity purchase price at time t, is the electricity price at time t, is the interactive power. When VPP purchases electricity from the external power grid, When VPP sells electricity to external power grid.

[0216] Load reduction costs

[0217] VPP operators can reduce some of the load demand, but at the same time they need to give users corresponding compensation. The calculation can be expressed as:

[0218]

[0219] Among them, β is the compensation cost per unit reduction, is the load reduction amount.

[0220] S2-9: Define the constraints of the optimization model.

[0221] Power balance constraints

[0222] Considering the power balance of various resources within the VPP, VPP operators need to meet the following constraints during actual scheduling:

[0223]

[0224] The left side of the equation is the power generation part, and the right side of the equation is the power consumption part, where: and It is the action of the outer part. It is one of the states of the outer layer. When making optimization decisions for the inner layer, all three are given constants. is the decision variable for inner optimization.

[0225] Renewable energy generation ceiling constraints

[0226] Considering that wind power and photovoltaic power generation are both limited by factors such as wind speed and light intensity, their power generation must meet the following upper limit constraints:

[0227]

[0228] in, and They are the upper limits of wind power and photovoltaic power generation, that is, the current state transmitted from the outer layer to the inner layer.

[0229] Controllable load reduction limit

[0230] The upper limit of controllable load reduction satisfies the following formula:

[0231]

[0232] in, Likewise, the outer layer passes the current state to the inner layer.

[0233] Upper limit of wind and solar power curtailment

[0234] The upper limit of wind and solar power abandonment satisfies the following formula:

[0235]

[0236] in, and These are the maximum waste ratios of wind power and photovoltaic power generation respectively.

[0237] Interactive power limit

[0238] The amount of electricity exchanged with the external power grid must meet the following constraints:

[0239]

[0240] in, is the maximum value of electricity sold to the external power grid, It is the maximum amount of electricity purchased from the external power grid.

[0241] S2-10. Train the DDPG algorithm. The internal structure of the algorithm is shown in the attached figure. Figure 3 As shown, it includes two strategy networks and two value networks.

[0242] S2-11. The training process of the DDPG algorithm is as follows:

[0243]

[0244]

[0245] S3: Collect real-time operation data of virtual power plants, including power consumption, electricity prices, and wind and photovoltaic output in the current period. Similar to the information collected in step S1, before conducting deep reinforcement learning intelligent scheduling, it is necessary to collect real-time operation data of virtual power plants in advance and make decisions through the mature model trained in step S2.

[0246] S4: Use the DDPG algorithm to optimize resources with high timing, including the charging and discharging actions of energy storage devices and thermal power units.

[0247] S5: Based on the optimization results of the DDPG algorithm, a traditional optimization model is established to dispatch the remaining wind power, photovoltaic power, controllable load and other resources.

[0248] S6: Integrate the actions of the DDPG algorithm with the results of the optimization model to obtain the final scheduling plan and achieve the optimal economic benefits.

[0249] In order to make researchers in this field better understand the method proposed by the present invention, the attached Figure 1 The flowchart shown is Figure 2 The model structure diagram shown is intuitive and easy to understand.

[0250] This patent involves an online optimization scheduling method for virtual power plants, which aims to combine the efficiency of online decision-making of deep reinforcement learning algorithms with the optimality of the solution results of traditional optimization models to efficiently schedule various resources within the VPP.

[0251] First, the present invention adopts the DDPG algorithm of deep reinforcement learning to deal with complex optimization and scheduling problems during the operation of virtual power plants. Training based on the historical data of virtual power plants can enable intelligent agents to better handle the uncertainty within virtual power plants when making decisions. Secondly, the present invention still retains part of the traditional optimization decision-making. In view of the fact that traditional optimization models can quickly solve the optimal solution when facing problems with low complexity and low spatial dimensions of decision variables, the present invention uses traditional optimization models to solve the scheduling problems of resources in virtual power plants that are highly decoupled from timing and have flexible scheduling time. Combined with deep reinforcement learning methods, it is highly efficient. Finally, the present invention introduces the Lyapunov optimization method in queuing theory to ensure the stability of long-term online decision-making, convert the charge state of energy storage equipment in virtual power plants into virtual queues, achieve long-term stability of queue length, and ensure the stability of long-term online scheduling.

[0252] In summary, the online optimization scheduling method of virtual power plants based on deep reinforcement learning proposed in this patent integrates modern intelligent algorithms, optimization models and control theories, effectively solves the problems faced by existing systems, and has high practical value and technological innovation.

[0253] The contents described in the embodiments of this specification are merely enumerations of implementation forms of the inventive concept and are for illustrative purposes only. The protection scope of the present invention should not be considered to be limited to the specific forms described in this embodiment, and the protection scope of the present invention also extends to equivalent technical means that can be thought of by ordinary technicians in this field based on the inventive concept.

Claims

1. A virtual power plant online optimization scheduling method based on deep reinforcement learning algorithm, characterized in that: The specific steps include: S1. Collect historical operation data of virtual power plants, including historical power consumption, electricity prices, and wind and photovoltaic output of virtual power plants; S2. Establish an internal and external double-layer comprehensive optimization scheduling model. First, define the action set and action space, state set and state space, reward, and discount rate of the deep reinforcement learning algorithm in the external layer of the model. Second, add the Lyapunov drift to the reward of the intelligent agent to maintain the long-term stability of the online scheduling. S3. Define the optimization model in the inner layer of the model, and define the objective function, decision variables, and constraints; S4. Use historical data to train the DDPG algorithm in the internal and external double-layer comprehensive optimization scheduling model until the model converges; S5. Collect real-time operation data of virtual power plants, including power consumption, electricity prices, wind power and photovoltaic output during the current period; S6. Input the real-time operation data of the virtual power plant, call the internal and external double-layer comprehensive optimization model, and use the mature DDPG algorithm of the external layer to optimize the scheduling of resources with high time sequence, including the charging and discharging actions of energy storage equipment and thermal power units; S7. After the intelligent agent of the DDPG algorithm in the outer layer of the model takes action and transfers state, the established inner layer optimization model is used to dispatch the remaining wind power, photovoltaic power, and controllable load resources; S8. Integrate the actions of the agent in the DDPG algorithm and the results of the optimization model to obtain the final scheduling plan and obtain the optimal economic benefits.

2. According to claim 1, a virtual power plant online optimization scheduling method based on deep reinforcement learning algorithm is characterized in that: The historical operation data of the virtual power plant described in step S1 include: the historical average wind speed in the virtual power plant area, the historical light intensity, the historical controllable load, the historical basic load, the historical electricity purchase price and the historical electricity sales price for interaction with the external power grid.

3. The online optimization scheduling method of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: Step S2 defines the action set and action space, state set and state space, reward, and discount rate of the deep reinforcement learning algorithm in the outer layer of the model, which consists of the following sub-steps: S2-1: Classify the scheduling characteristics of various resources within the VPP according to their relationship with timing, and perform external scheduling on the resources with timing. Resources with time sequence include energy storage equipment and thermal power units. These devices can use deep reinforcement learning methods to achieve sequential decision-making by constructing a Markov decision process (MDP). Resources that are not time-sequential and have flexible scheduling include curtailable loads, wind power equipment, and photovoltaic equipment. For such equipment, the optimal scheduling solution can be directly solved by establishing an optimization model. S2-2: Define the action set and action space of the DDPG algorithm agent; Considering the decision-making with time sequence, the action of the agent is: a t ={P t d ,P t ess } Among them, P t d is the power generation of the thermal power unit at time t; P t ess is the charge and discharge capacity of the energy storage device at time t. t ess ≥0, the energy storage device is charged, when P t ess When <0, the energy storage device discharges; in order to facilitate subsequent analysis, the start and stop variables of the thermal power unit are defined: Use V t d Represents the start and stop status of the thermal power unit at time t. t d =1, the thermal power unit is in the starting state. t d =0 when the thermal power unit is shut down; Define the action space of the energy storage device in the above actions: in, The upper limit of the charging and discharging power of the energy storage device; Further consider the action space of thermal power units: in, and are the lower and upper limits of the power generation capacity of thermal power units, DR d is the lower limit of the thermal power unit’s slow-down, UR d is the upper limit of the thermal power unit’s climbing slope; in the algorithm solution process, the action space of the thermal power unit will change with the power generation power at time t-1. The dynamic change means that the agent has different action spaces when taking actions due to the different actions at the previous moment; S2-3: Define the state set and state transition of the DDPG algorithm agent; 1) The state set of the agent Define the state of the agent at time t as: in, is the maximum wind power at time t, is the maximum photovoltaic power at time t, is the maximum controllable load at time t, P t base is the basic load at time t, SOC t is the state of charge of the energy storage device at time t, is the electricity purchase price at time t, is the electricity selling price at time t; The agent will make decisions based on the seven states in the state set; in the state set, except SOC t All other states outside the model will be passed to the inner part of the model, and the optimization model will be used to make further decisions to determine the actions that can flexibly schedule the equipment; 2) Energy storage device state transfer Energy storage device SOC t The state transition process is: Among them, η ch and η dis are the charge and discharge efficiencies, respectively, and C is the capacity of the energy storage device in the VPP; 3) Wind power and photovoltaic state transfer Using publicly available historical data sets, the maximum power generation of wind power and photovoltaic power is predicted based on wind speed and sunlight intensity; For wind power generation, the wind speed v at time t is used. t , the upper limit of wind power generation power is calculated in, is the rated power of the fan, ν ci is the wind turbine cut-in wind speed, ν co is the fan cut-out wind speed, ν n is the rated wind speed of the fan; For photovoltaic power generation, the light intensity θ at time t is used t , the upper limit of photovoltaic power generation can be calculated Among them, η PV is the photovoltaic conversion efficiency, S PV is the photovoltaic equipment area; When training the DRL algorithm, the intraday scenery data will be randomly sampled to simulate the real environment; S2-4: Define the reward function of the DDPG algorithm agent; The reward function of the DDPG agent is the negative of the thermal power dispatching cost and the energy storage dispatching cost, plus the objective function value of the subsequent inner optimization model and the Lyapunov drift; 1) Thermal power dispatching costs Dispatch cost of thermal power units It consists of 3 parts: is the power generation cost of thermal power units, a t , b t 、c t They are the coefficients of the constant term, linear term, and quadratic term of the power generation cost respectively; is the startup cost of the thermal power unit, k d1 is the startup cost coefficient, ΔT stop The interval time from the last shutdown; is the shutdown cost of the thermal power unit, k d2 is the downtime cost coefficient, ΔT start It is the interval time from the last startup, that is, the running time of the thermal power unit; 2) Energy storage dispatch costs Considering factors such as the life and depreciation of energy storage equipment, the dispatch cost of energy storage equipment needs to be considered. Its charging and discharging power P of the energy storage device t ess It is a linear relationship and can be expressed as: Among them, k ess is the electricity cost of the energy storage device; S2-5: Define the reward discount rate of the DDPG algorithm agent. The discount rate γ is taken as 0.98 in the calculation.

4. The online optimization scheduling method of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: Step S2 adds Lyapunov drift to the agent's reward to maintain long-term stability of online scheduling. The specific steps are: Introducing the Lyapunov drift indicator into the agent’s reward function to ensure the stability of the sequence during long-term scheduling; The virtual queue is defined by the state of charge (SOC) of the energy storage device. The drift index of the queue length is introduced and minimized to achieve the long-term state of charge stability constraint. According to the energy storage device state transfer calculation formula, assuming that the initial state of charge of the energy storage device is SOC o At the same time, to facilitate subsequent derivation, the SOC change calculation formula is defined as: By long-term accumulation of the charge and discharge change formula, we can get: Due to SOC T Need to meet SOC T ∈[SOC min ,SOC max ], then finding the limit of T yields: The above formula indicates that the SOC can be stabilized within the constraint range only when the long-term total charge capacity of the energy storage device is equal to the long-term total discharge capacity. Therefore, the virtual queue can be further defined as: Q t =SOC t -ξ The constant ξ is: in, is the maximum charge and discharge power of the energy storage device, and V is the weight coefficient. Based on this virtual queue, the long-term stability condition of the energy storage device can be defined as: The Lyapunov function that defines the queue length is: Then the Lyapunov drift index can be introduced as: When minimizing the drift is used as the objective function, the long-term stability of the queue can be achieved. Therefore, by adding the Lyapunov drift to the reward function of the intelligent agent, the long-term stability of the online scheduling can be achieved.

5. The online optimization scheduling method of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: The establishment of the optimization model described in step S3 includes the following sub-steps: S3-1: Define the decision variables of the optimization model; After the outer layer makes a decision, the inner layer will further solve the optimal action for the remaining resources in the VPP based on the actions taken by the agent. Specifically, the outer layer optimizes the scheduling of time-series resources, while the inner layer further optimizes the scheduling of relatively flexible resources. Therefore, the decision variable for inner optimization is the actual wind power generation P t w , Photovoltaic power generation power P t p , the amount of electricity exchanged with the external power grid P t tr , load reduction of controllable load P t con ; S3-2: Define the objective function of the optimization model; The objective function of the inner optimization part is to minimize the scheduling cost of the remaining scheduling resources: in, The cost of wind power generation is is the photovoltaic power generation cost, is the cost of interacting with the external power grid, Costs for load reduction; 1) Wind power generation costs Wind power costs Including depreciation cost of wind turbines Scheduling costs Wind curtailment penalty costs Among them, r is the annual interest rate, is the installation cost per unit capacity of wind turbines, K w is the capacity factor of the wind turbine, n w is the operating life of the wind turbine, K wom is the maintenance and operation cost coefficient of the wind turbine; w,1 With λ w,2 are the linear and quadratic coefficients of wind curtailment penalty respectively; 2) Photovoltaic power generation cost Photovoltaic power generation costs Including depreciation cost of wind turbines Scheduling costs Abandonment penalty cost Among them, r is the annual interest rate, is the unit capacity installation cost of photovoltaic cells, K p is the photovoltaic capacity factor, n p is the operating life of the photovoltaic, K pom is the photovoltaic maintenance and operation cost coefficient; p,1 With λ p,2 are the linear and quadratic coefficients of the light abandonment penalty respectively; 3) External power grid interaction costs During the VPP dispatching process, it can interact with the external power grid. When the internal load demand of the VPP is too large, it can choose to purchase electricity from the external power grid. When the internal power generation of the VPP is in excess, it can choose to sell electricity to the external power grid to obtain revenue. The cost of the interaction between the VPP and the external power grid can be expressed as: in, is the electricity purchase price at time t, is the electricity price at time t, P t tr is the interactive power; when P t tr >0 means VPP purchases electricity from external power grid. t tr When <0, it means that VPP sells electricity to the external power grid; 4) Load reduction costs VPP operators can reduce some load demand, but at the same time they need to give users corresponding compensation; load reduction costs The calculation can be expressed as: Among them, β is the compensation cost per unit reduction, P t con is the load reduction amount; S3-3: Define the constraints of the optimization model; 1) Power balance constraints Considering the power balance of various resources within the VPP, VPP operators need to meet the following constraints during actual scheduling: The left side of the equation is the power generation part, and the right side of the equation is the power consumption part, where P t d With P t ess is the action of the outer part, P t base is one of the states of the outer layer. When optimizing the inner layer, all three are given constants. t w , P t p , P t tr , P t con is the decision variable for inner optimization; 2) Upper limit constraints on renewable energy power generation Considering that wind power and photovoltaic power generation are both limited by wind speed and light intensity, their power generation must meet the following upper limit constraints: in, and are the upper limits of wind power and photovoltaic power generation, i.e., the current state transmitted from the outer layer to the inner layer; 3) Upper limit of controllable load reduction The upper limit of controllable load reduction satisfies the following formula: in, Likewise, the outer layer passes the current state to the inner layer; 4) Upper limit of wind and solar power curtailment The upper limit of wind and solar power abandonment satisfies the following formula: in, and They are the maximum waste ratios of wind power and photovoltaic power generation; 5) Interactive power limit The amount of electricity exchanged with the external power grid must meet the following constraints: in, is the maximum value of electricity sold to the external power grid, It is the maximum amount of electricity purchased from the external power grid.

6. The online optimization scheduling method of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: The real-time operating data described in step S5 is the same as the historical data in step S1, including the real-time average wind speed in the virtual power plant area, the real-time light intensity, the real-time controllable load, the real-time basic load, the real-time electricity purchase price interacting with the external power grid, and the real-time electricity selling price.

7. The method for online optimization and scheduling of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: The implementation steps of using the DDPG algorithm to optimize the resources with high timing characteristics in step S6 are as follows: In the inner and outer double-layer optimization scheduling model established in step S2, based on the real-time operation data collected in step S5, the mature DDPG deep reinforcement learning algorithm of the outer layer is used to optimize the scheduling of equipment with high timing, and obtain the optimized scheduling instructions for charging and discharging of thermal power units and energy storage equipment.

8. The online optimization scheduling method of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: The final dispatching plan described in step S8 includes the plan obtained by integrating the actions of the DDPG algorithm and the results of the optimization model, specifically, the power generation of the thermal power units, the charging and discharging of the energy storage equipment, the actual power generation of the wind power equipment, the actual power generation of the photovoltaic equipment, the controllable load reduction, and the interactive power with the external power grid.

9. The online optimization scheduling method of a virtual power plant based on a deep reinforcement learning algorithm according to claim 1 is characterized in that: The final economic benefit described in step S8 is obtained by adding the reward of the deep reinforcement learning algorithm and the objective function value of the optimization model, which means the optimal long-term stable profit obtained after the virtual power plant is optimized and scheduled online.

Images