Virtual power plant dynamic aggregation bidding method, system and equipment based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based virtual power plant dynamic aggregation bidding method, system and device, and aims to solve the problems that a complex and changeable electricity market environment enables a VPP to face many uncertainties, and a traditional bidding strategy is difficult to meet efficient operation requirements in a dynamic market; according to the method, the bidding process of the VPP is modeled as a Markov decision process, market supply and demand information, self resource states and the like are taken as state variables, bidding prices and electric quantity are taken as action spaces, and a reinforcement learning algorithm is utilized to enable the VPP to learn and optimize a bidding strategy in continuous interaction with a market environment so as to realize revenue maximization; the bidding strategy provided by the invention can effectively adapt to market changes, the bidding income and competitiveness of the virtual power plant in a dynamic market are remarkably improved, and a decision support with higher adaptability and intelligence is provided for a VPP operator.

Description

Technical Field

[0001] The present invention relates to a virtual power plant, and in particular to a dynamic aggregation bidding method, system and equipment for a virtual power plant based on reinforcement learning. Background Art

[0002] The power industry is currently undergoing profound changes. On the one hand, the influx of renewable energy sources, such as solar and wind power, into the power grid presents significant challenges to the stable operation of the power system due to their inherent intermittency and volatility. On the other hand, the gradual emergence of an open and competitive power market requires all types of power generation entities to participate in market transactions more flexibly and intelligently.

[0003] As an emerging energy aggregation system, virtual power plants (VPPs) effectively integrate diverse decentralized resources, such as distributed power sources, energy storage equipment, and controllable loads, through advanced information, communication, and control technologies, achieving coordinated and optimized resource operation. They not only enhance the capacity to absorb renewable energy but also enable integrated participation in electricity market transactions, providing innovative solutions for the stable operation and efficient management of power systems.

[0004] Although virtual power plants have made certain progress in both theory and practice, they still face many difficulties when participating in electricity market bidding. The electricity market environment is complex and changeable, and electricity prices are affected by many factors, such as energy supply and demand, weather conditions, etc., showing a high degree of uncertainty. Traditional static bidding strategies are difficult to cope with such a complex and dynamically changing market environment. They are unable to adjust bidding plans in time according to market fluctuations, resulting in virtual power plants being unable to fully utilize their resource aggregation advantages and maximize economic benefits. In addition, the internal resources of virtual power plants are of various types and have different characteristics. How to dynamically aggregate these resources under different market conditions and participate in bidding with the optimal combination is a key issue that needs to be solved urgently. The lack of an effective dynamic aggregation bidding strategy may result in waste of resources or irrational allocation, reducing the overall competitiveness of virtual power plants. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a dynamic aggregation bidding method, system and equipment for virtual power plants based on reinforcement learning, so as to improve the operational efficiency and profitability of virtual power plants in complex power markets and ensure the stable and economic operation of the power system.

[0006] Technical solution: The present invention provides a dynamic aggregation bidding method for virtual power plants based on reinforcement learning, comprising:

[0007] (1) Construct performance indicators of the flexible resource adjustable capacity in the virtual power plant, including maximum adjustment capacity, response delay, response duration and adjustment rate, maximum adjustment capacity Indicates the maximum power upward offset and maximum downward adjustment capability of the flexibility resource relative to the current operating point. Indicates the maximum power downward offset that the flexibility resource can make relative to the current operating point; response delay T R It indicates the time required for the flexibility resource to fully reach the required power after receiving the adjustment command, which corresponds to the start and stop time of the traditional unit; the response duration T D It indicates the maximum time that the flexibility resource can maintain the adjustment state after meeting the power adjustment requirements, which corresponds to the continuous start and stop time of the traditional unit; the adjustment rate R includes the upward adjustment rate R k,up and downward adjustment rate R k,down , respectively represent the maximum value of the upward and downward power adjustment of the flexibility resource per unit time, corresponding to the ramp rate of the traditional unit;

[0008] (2) Construct the reliability index of the adjustable capacity of flexible resources in the virtual power plant, including the probability of response time of insufficient power, the expected value of response time of insufficient power, and the expected value of response time of insufficient power; the probability of response time of insufficient power γ LOLP represents the probability that the flexible resource cannot reach the specified adjustment range due to uncertain factors after receiving the adjustment instruction; the expected time of response to insufficient power γ LOLE It represents the expected value of the time when the adjustable capacity shortage of flexible resources is greater than or equal to the backup capacity; the expected value of the response power shortage γ EENS It indicates that the flexibility resources are affected by uncertainty factors, resulting in insufficient response, thereby reducing the expected value of power supply;

[0009] (3) Construct a virtual power plant day-ahead and real-time bidding model based on constraint-aware reinforcement learning, including a virtual power plant day-ahead bidding model and a virtual power plant real-time bidding model;

[0010] (4) Construct the objective function and constraints of the virtual power plant day-ahead bidding model. The objective function of the virtual power plant day-ahead bidding model is to maximize its own market revenue from participating in the day-ahead market. The constraints of the virtual power plant day-ahead bidding model include the upper and lower limits of the virtual power plant's overall power and the ramp rate constraint.

[0011] (5) Construct the objective function and constraints of the virtual power plant real-time quotation model. The objective function of the virtual power plant real-time quotation model is to optimize the quotation results of the period. The constraints of the virtual power plant real-time quotation model are the same as those of the virtual power plant day-ahead quotation model, only the time scale is converted into the real-time scale;

[0012] (6) Mixed integer programming-deep Q network planning is used to solve the virtual power plant day-ahead-real-time bidding model.

[0013] Furthermore, step (1) includes:

[0014]

[0015] in, is the power upper limit of flexibility resource k; P k,t is the power of flexibility resource k at time t;

[0016]

[0017] in, is the lower power limit of flexibility resource k;

[0018]

[0019] in, and are the response delays of upward and downward adjustment of flexibility resource k, respectively; t ds,up and t ds,down are the moments when the flexibility resources complete upward and downward adjustments, respectively; t ss The moment when the flexibility resource receives the response instruction;

[0020]

[0021] in, and are the response durations of upward and downward regulation of flexibility resource k, respectively; t de,up and t de,down The maximum time for the flexibility resource to maintain the adjustment state after completing the upward and downward adjustments respectively;

[0022]

[0023] Furthermore, step (2) includes:

[0024]

[0025] Among them, l is the system operation status; p l is the probability of insufficient response power when the system is in running state l; t l The duration of the response power shortage when the system is in running state l;

[0026]

[0027] Among them, I t is the number of time periods a day is divided into; t is the time within the time period; T i is the number of hours included in the i-th period; C i is the theoretical maximum adjustment capacity of the flexibility resource in the i-th period; L i,tis the peak load power at time t in the i-th period; P[X≥C i -L i,t ] is the probability that the adjustable capacity shortage of flexible resources is greater than or equal to the spare capacity;

[0028]

[0029] Where N is the number of simulation hours; C I is the theoretical maximum adjustment capacity of the flexibility resource in the first simulation hour; L I is the load power in the first simulation hour; P I (X) is the probability of insufficient response power in the first simulation hour.

[0030] Furthermore, step (3) includes: reporting the pre-dispatching strategy to the dispatching agency, and when optimizing the dispatching strategy on the cloud platform, in order to take uncertainty into account, generating several scenarios for random optimization, and predicting the clearing quantity and price of the real-time market, and selecting the strategy based on the highest combined day-ahead and real-time market returns under each scenario;

[0031] After the virtual power plant cloud platform completes the day-ahead market quotation, it obtains more accurate real-time forecast data in the intraday stage, allowing it to participate in the real-time market to correct its own reported electricity volume; among them, the electricity cleared in the day-ahead stage is settled according to the day-ahead price, and the correction amount for the day-ahead stage in the real-time stage is settled according to the real-time market clearing electricity price; the real-time stage uses a rolling time window for optimization, taking the current moment and the subsequent time period as the time window, and referring to the real-time forecast information to perform optimization operation within the time period to obtain the optimal operation scheduling result for the time period, but the virtual power plant cloud platform only reports the optimization result of the current moment when reporting.

[0032] Furthermore, in step (4), the optimization objective function of the virtual power plant's day-ahead quotation is as follows:

[0033]

[0034] Among them, π ω is the probability of the scenario ω occurring; T represents the time of a scheduling cycle; are the bidding power of the virtual power plant cloud platform at time t in the day-ahead phase, and ω is the decision variable of the day-ahead optimization scheduling problem; are the predicted values ​​of the clearing electricity prices of the day-ahead energy market and the reserve market by the virtual power plant cloud platform at time t; T RT is the number of time periods divided into each hour on the day-ahead time scale; are the winning bids in the real-time electric energy market and the reserve market under scenario ω, respectively. In the day-ahead phase, their values ​​are predicted values; They are the electricity market price and the reserve market price in the real-time stage respectively; is the overall operating cost function of the virtual power plant, which is expressed in the form of a quadratic function, and its parameters are the maximum values ​​in the flexibility resource cost function; is the total amount of electricity that the virtual power plant won in the spot market under the scenario ω, expressed as:

[0035]

[0036] Furthermore, the overall power limit of the virtual power plant is constrained as follows:

[0037]

[0038] in, The output power reported by the virtual power plant at time t when participating in the day-ahead electricity energy-reserve market; and They represent the maximum output power boundary and minimum power output boundary of the virtual power plant at time t respectively;

[0039] Virtual power plant ramp rate constraints:

[0040]

[0041] in, and They represent the lower and upper boundaries of the virtual power plant power change rate respectively.

[0042] Furthermore, step (5) includes: the objective function of the virtual power plant real-time quotation model is as follows:

[0043]

[0044] in, are the electricity reported by the virtual power plant cloud platform at real-time market moment n, and are the decision variables for optimization operation in the real-time stage; Δt' is the length of the rolling optimization time window, which is an integer value; are the clearing electricity prices of the energy market and reserve market predicted by the cloud platform at the real-time market moment n; They are the adjustment of the output power by the virtual power plant cloud platform in real time at time t; T RT The number of scheduling intervals for the real-time phase; They are the cleared quantities of the day-ahead electricity market and the reserve market after the cleared quantities are expanded to the real-time time scale.

[0045] Furthermore, step (6) includes: transforming the virtual power plant day-ahead-real-time bidding problem into a Markov decision process, wherein the basic characteristics of the process are {s, a, P, R}, which represent the state space, action space, state transition matrix and reward respectively;

[0046] 1) State Space

[0047] When the state space corresponds to the virtual power plant day-ahead real-time bidding model, the state set is expressed as:

[0048]

[0049] Among them, s represents the state set, which represents the set of multiple state information of the virtual power plant at a certain moment; P VPP,t-1 is the power of the virtual power plant at time t-1; E t-1 is the state of charge of the energy storage at time t-1; is time i at t-1 F The power of the devices in the collection; i F The collection includes photovoltaic PV, wind power W, energy storage ESS, distributed generator DG and grid AC; i b Represents the optimization cycle index, and N represents the total number of cycles;

[0050] 2) Action Space

[0051] The action space consists of the actions of each agent, which corresponds to the decision variables for the optimal operation of each flexibility resource in this problem;

[0052] α1={P VPP,t} (18)

[0053] 3) Rewards

[0054] The agent's reward set is the objective function value in the model.

[0055] Based on the same inventive concept, the present invention provides a virtual power plant dynamic aggregation bidding system based on reinforcement learning, including:

[0056] Evaluation index construction module, used to construct performance indicators and reliability indicators of the flexible resource adjustability in virtual power plants;

[0057] A model building module is used to build a virtual power plant day-ahead and real-time bidding model based on constraint-aware reinforcement learning, including a virtual power plant day-ahead bidding model, its objective function and constraints, and a virtual power plant real-time bidding model, its objective function and constraints;

[0058] The model solving module is used to solve the day-ahead and real-time bidding model of the virtual power plant using mixed integer programming-deep Q network planning.

[0059] Based on the same inventive concept, the present invention provides a virtual power plant dynamic aggregation bidding device based on reinforcement learning, including a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the above-mentioned virtual power plant dynamic aggregation bidding method based on reinforcement learning.

[0060] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0061] In the context of building a new power system and transitioning to a low-carbon energy system, virtual power plants (VPPs) leverage advanced technology to aggregate distributed resources and participate in power market transactions, significantly improving grid dispatch and renewable energy consumption. However, the complex and volatile power market environment presents VPPs with numerous uncertainties, making traditional bidding strategies unable to meet the requirements for efficient operation in this dynamic market.

[0062] This paper proposes a dynamic aggregation bidding strategy for virtual power plants (VPPs) based on reinforcement learning. This strategy models the VPP bidding process as a Markov decision process, using market supply and demand information, resource status, and other variables as state variables, and the bidding price and power consumption as the action space. Using a reinforcement learning algorithm, the VPP learns and optimizes its bidding strategy through continuous interaction with the market environment to maximize revenue. Simulation experiments in various market scenarios demonstrate that, compared to traditional bidding strategies, the proposed bidding strategy can effectively adapt to market changes, significantly improving the bidding revenue and competitiveness of VPPs in dynamic markets. This provides VPP operators with a more adaptable and intelligent decision-making platform. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a flowchart of a dynamic aggregation bidding method for a virtual power plant based on reinforcement learning provided by an embodiment of the present invention;

[0064] Figure 2 is the cumulative reward curve of the virtual power plant during the training process in an embodiment of the present invention;

[0065] Figure 3 is the trained Q table in the embodiment of the present invention. DETAILED DESCRIPTION

[0066] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It will be understood by those skilled in the art that the purposes and advantages that can be achieved by the present invention are not limited to the specific description of the above beneficial effects, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description.

[0067] It should be understood by those skilled in the art that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether hardware or software is used depends on the specific application and design and tree conditions of the technical solution. Professionals and technicians may use different methods to implement the functions described for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0068] References to "embodiments" in this disclosure mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the disclosure. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0069] Example 1

[0070] like Figure 1 As shown, a virtual power plant dynamic aggregation bidding method based on reinforcement learning provided in Example 1 may include the following steps:

[0071] (1) Constructing performance indicators for the adjustable capacity of flexible resources in virtual power plants;

[0072] The performance indicators of flexible resource adjustability include maximum adjustment capability, response delay, response duration, and adjustment rate, as follows:

[0073] 1) Maximum adjustment capability It represents the maximum power upward offset that the flexibility resource can make relative to the current operating point. The expression is as follows:

[0074]

[0075] in, is the power upper limit of flexibility resource k; P k,t is the power of flexibility resource k at time t.

[0076] 2) Maximum down-regulation capability It represents the maximum power downward offset that the flexibility resource can make relative to the current operating point. The expression is as follows:

[0077]

[0078] in, is the lower power limit of flexibility resource k.

[0079] 3) Response delay T R: It represents the time required for the flexible resource to fully reach the required power after receiving the adjustment command, which corresponds to the start and stop time of the traditional unit. The expression is as follows:

[0080]

[0081] in, and are the response delays of upward and downward adjustment of flexibility resource k, respectively; t ds,up and t ds,down are the moments when the flexibility resources complete upward and downward adjustments, respectively; t ss The moment when the flexibility resource receives the response instruction.

[0082] 4) Response duration T D : Indicates the maximum time that the flexibility resource can maintain the regulation state after meeting the power regulation requirements, corresponding to the continuous start and stop time of the traditional unit. The expression is as follows:

[0083]

[0084] in, and are the response durations of upward and downward regulation of flexibility resource k, respectively; t de,up and t de,down The maximum time that a flexibility resource can maintain its adjusted state after completing upward and downward adjustments, respectively.

[0085] 5) Adjustment rate R: including upward adjustment rate R k,up and downward adjustment rate R k,down , respectively, represent the maximum value of the upward and downward power adjustment of the flexible resource per unit time, corresponding to the ramp rate of the traditional unit. The expression is as follows:

[0086]

[0087] (2) Constructing reliability indicators for the adjustable capacity of flexible resources in virtual power plants;

[0088] The reliability indicators of the flexible resource adjustability include the probability of response time to insufficient power, the expected response time to insufficient power, and the expected value of response time to insufficient power, as follows:

[0089] 1) Probability of responding to insufficient battery time γ LOLP : It represents the probability that the flexibility resource cannot reach the specified adjustment range due to uncertain factors after receiving the adjustment instruction. The expression is:

[0090]

[0091] Among them, l is the system operation status; p l is the probability of insufficient response power when the system is in running state l; t l The duration of the response power shortage when the system is in the running state l.

[0092] 2) Expected response time to low battery level γ LOLE : Indicates the expected time that the adjustable capacity shortage of flexible resources is greater than or equal to the spare capacity. The expression is:

[0093]

[0094] Among them, I t is the number of time periods a day is divided into; t is the time within the time period; T i is the number of hours included in the i-th period; C i is the theoretical maximum adjustment capacity of the flexibility resource in the i-th period; L i,t is the peak load power at time t in the i-th period; P[X≥C i -L i,t ] is the probability that the adjustable capacity shortage of flexibility resources is greater than or equal to the spare capacity.

[0095] 3) Expected value of response to insufficient power γ EENS : Indicates that the flexibility resource is affected by uncertainty factors, resulting in insufficient response, thereby reducing the expected value of power supply. The expression is:

[0096]

[0097] Where N is the number of simulation hours; C I is the theoretical maximum adjustment capacity of the flexibility resource in the first simulation hour; L I is the load power in the first simulation hour; P I (X) is the probability of insufficient response power in the first simulation hour.

[0098] (3) Construct a day-ahead and real-time bidding model for virtual power plants based on constraint-aware reinforcement learning;

[0099] The virtual power plant day-ahead-real-time bidding model based on constraint-aware reinforcement learning includes a virtual power plant day-ahead bidding model and a virtual power plant real-time bidding model.

[0100] Declare the pre-dispatch strategy to the dispatching agency. When optimizing the dispatching strategy on the cloud platform, in order to take uncertainty into account, generate several scenarios for random optimization, and predict the real-time market clearing quantity and price. The strategy selection is based on the highest combined day-ahead and real-time market returns in each scenario.

[0101] After the virtual power plant cloud platform completes the day-ahead market quotation, it obtains more accurate real-time forecast data during the intraday phase, allowing it to participate in the real-time market to correct its reported electricity. The electricity cleared during the day-ahead phase is settled according to the day-ahead price, and the correction amount during the real-time phase is settled according to the real-time market clearing price. The real-time phase uses a rolling time window for optimization, taking the current moment and the subsequent period as the time window, and referring to the real-time forecast information to perform optimization operations within the period to obtain the optimal operation and scheduling results for the period. However, the virtual power plant cloud platform only reports the optimization results for the current moment when reporting.

[0102] (4) Construct the objective function and constraints of the virtual power plant day-ahead bidding model;

[0103] The objective function of the virtual power plant day-ahead bidding model is to maximize its own market revenue from participating in the day-ahead market, as follows:

[0104]

[0105] Among them, π ω is the probability of the scenario ω occurring; T represents the time of a scheduling cycle; are the bidding power of the virtual power plant cloud platform at time t in the day-ahead phase, and ω is the decision variable of the day-ahead optimization scheduling problem; are the predicted values ​​of the clearing electricity prices of the day-ahead energy market and the reserve market by the virtual power plant cloud platform at time t; T RT is the number of time periods per hour in the day-ahead time scale. Taking the granularity of 15 minutes as an example, its value is 4; are the winning bids in the real-time electricity market and reserve market under scenario ω, respectively. In the day-ahead phase, their values ​​are predicted values ​​(generated based on the scenario); They are the electricity market price and the reserve market price in the real-time stage respectively; is the overall operating cost function of the virtual power plant, which is expressed in the form of a quadratic function, and its parameters are the maximum values ​​in the flexibility resource cost function; is the total amount of electricity that the virtual power plant won in the spot market under the scenario ω, expressed as:

[0106]

[0107] When participating in the primary and secondary joint market, the virtual power plant cloud platform encapsulates internal resource information and presents the overall external characteristics of the flexible resource virtual power plant to the outside world. Therefore, during the day-ahead quotation process, the reported power information must meet the external characteristics of the virtual power plant itself, including the maximum adjustable power and ramp rate characteristics. The specific expression is as follows:

[0108] 1) Upper and lower limits of the overall power of the virtual power plant:

[0109]

[0110] in, The output power reported by the virtual power plant at time t when participating in the day-ahead electricity energy-reserve market; and They represent the maximum output power boundary and minimum power output boundary of the virtual power plant at time t respectively.

[0111] 2) Virtual power plant ramp rate constraints:

[0112]

[0113] in, and They represent the lower and upper boundaries of the virtual power plant power change rate respectively.

[0114] (5) Construct the objective function and constraints of the virtual power plant real-time bidding model;

[0115] The objective function of the virtual power plant real-time bidding model is to optimize the bidding results for that period, as follows:

[0116]

[0117] in, are the electricity reported by the virtual power plant cloud platform at real-time market moment n, and are the decision variables for the optimization operation in the real-time stage; Δt' is the length of the rolling optimization time window, which is an integer value; are the clearing electricity prices of the energy market and reserve market predicted by the cloud platform at the real-time market moment n; They are the adjustment of the output power by the virtual power plant cloud platform in real time at time t; T RT The number of scheduling intervals for the real-time phase; They are the cleared quantities of the day-ahead electricity market and the reserve market after the cleared quantities are expanded to the real-time time scale.

[0118] During real-time optimization, the virtual power plant cloud platform also needs to consider its own external constraints when reporting power consumption. Therefore, the constraints are the same as for day-ahead optimization, with only the time scale being converted to real-time.

[0119] (6) Mixed integer programming-deep Q network planning is used to solve the virtual power plant day-ahead-real-time bidding model.

[0120] First, the virtual power plant day-ahead real-time bidding problem is transformed into a Markov decision process (MDP). The basic characteristics of this process are {s, a, P, R}, which represent the state space, action space, state transition matrix, and reward, respectively. The specific characteristics for this problem are as follows:

[0121] 1) State Space

[0122] When the state space corresponds to the virtual power plant day-ahead real-time bidding model, it is the network topology, network flow information, the flexibility resource optimization operation results at the previous moment, the charge state of the energy storage, etc. Therefore, the state set can be expressed as:

[0123]

[0124] Among them, s represents the state set, which represents the set of multiple state information of the virtual power plant at a certain moment; P VPP,t-1 is the power of the virtual power plant at time t-1; E t-1 is the state of charge of the energy storage at time t-1; is time i at t-1 F The power of the devices in the collection; i F The collection includes photovoltaic PV, wind power W, energy storage ESS, distributed generator DG and grid AC; i b Represents the optimization cycle index, and N represents the total number of cycles.

[0125] 2) Action Space

[0126] The action space consists of the actions of each agent, which corresponds to the decision variables for the optimal operation of each flexibility resource in this problem;

[0127] α1={P VPP,t} (18)

[0128] 3) Rewards

[0129] The agent's reward set is the objective function value in the model. The above content corresponds to each abstract concept of the MDP problem that the present invention needs to solve.

[0130] The following is a specific test case that uses a reinforcement learning algorithm (Q-learning) to optimize the dynamic bidding strategy of a virtual power plant in the electricity market. A virtual power plant is a cluster of multiple distributed energy systems (such as wind, solar, and energy storage devices). Its goal is to dynamically adjust the amount of electricity sold based on real-time electricity market prices and demand to maximize revenue.

[0131] Action space: The amount of electricity a virtual power plant can bid on the market. Assume that in this model, the power plant can choose five different bid amounts (actions), which will affect the amount of electricity it sells on the market.

[0132] State space: To simplify the problem, factors such as market electricity prices and electricity demand are abstracted into a discrete state space. Assume that the state space of the electricity market consists of 10 discrete states, each representing a different market electricity price or demand situation.

[0133] Reward Function: The reward function is calculated using market electricity prices, demand, and bid volume. Specifically, the reward is determined based on the market electricity price (the price of electricity received by the virtual power plant) and demand (the amount of electricity sold by the power plant). In other words, the power plant's revenue is determined by the difference between the price received from selling electricity and the losses (if any) incurred due to demand.

[0134] Please see the attached Figure 2 This figure shows the cumulative rewards of the virtual power plant during the training process. As the training progresses, the Q-learning algorithm gradually optimizes the bidding strategy of the virtual power plant, and the total rewards gradually increase.

[0135] Initial Fluctuation: In the early stages of training, the initialization of the Q-table results in a random selection strategy for the virtual power plant, resulting in significant reward fluctuations. As training rounds increase, the virtual power plant gradually learns how to make better bidding decisions based on market electricity prices and demand, and the accumulated rewards stabilize. Eventually, the accumulated rewards reach a high level, indicating that the virtual power plant has learned how to maximize its profits under various market conditions.

[0136] Please see the attached Figure 3 , is the trained Q-table, which is a two-dimensional array. The rows represent different market states, and the columns represent different bidding quantities (actions) that the virtual power plant can take. Each entry represents the expected return of taking a certain action in a specific state. High Q-value area: In the Q-table, the Q-values ​​of certain state-action pairs are higher, indicating that in these states, taking the corresponding bidding quantity can bring higher expected benefits. These areas are usually associated with situations where electricity prices are higher or demand is lower in the electricity market. Low Q-value area: Some entries in the Q-table indicate that the return of a certain action in this state is lower, indicating that the virtual power plant should not choose the corresponding bidding quantity in these states. This may be because the electricity market price is low in this state, and too large a bidding quantity will result in a loss.

[0137] Example 2

[0138] Example 2 provides a virtual power plant dynamic aggregation bidding system based on reinforcement learning, which can implement the virtual power plant dynamic aggregation bidding method based on reinforcement learning described in Example 1, specifically including:

[0139] Evaluation index construction module, used to construct performance indicators and reliability indicators of the flexible resource adjustability in virtual power plants;

[0140] A model building module is used to build a virtual power plant day-ahead and real-time bidding model based on constraint-aware reinforcement learning, including a virtual power plant day-ahead bidding model, its objective function and constraints, and a virtual power plant real-time bidding model, its objective function and constraints;

[0141] The model solving module is used to solve the day-ahead and real-time bidding model of the virtual power plant using mixed integer programming-deep Q network planning.

[0142] Example 3

[0143] Example 3 provides a device for dynamic aggregate bidding for virtual power plants based on reinforcement learning. The device may include a processor and a memory. The memory stores computer instructions, and the processor is configured to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the dynamic aggregate bidding method for virtual power plants based on reinforcement learning as described in Example 1, achieving the same technical effects.

[0144] The memory may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The device may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the memory may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). A program / utility having a set (at least one) of program modules may be stored in, for example, the memory, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules typically perform the functions and / or methods of the embodiments described herein.

[0145] The processor executes various functional applications and data processing by running the programs stored in the memory, such as implementing the method provided in Example 1 of the present invention.

[0146] Example 4

[0147] Example 4 provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the dynamic aggregation bidding method for virtual power plants based on reinforcement learning as described in Example 1 are implemented, and the same technical effect can be achieved.

[0148] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by an instruction execution system, device or device or used in combination with it.

[0149] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0150] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0151] The computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0152] Of course, the computer-executable instructions of a storage medium provided by an embodiment of the present invention are not limited to the above method operations, but can also execute related operations in the method provided by any embodiment of the present invention.

[0153] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A dynamic aggregation bidding method for virtual power plants based on reinforcement learning, characterized in that: include: (1) Construct performance indicators of the flexible resource adjustable capacity in the virtual power plant, including maximum up and down adjustment capacity, response delay, response duration and adjustment rate, maximum up adjustment capacity Indicates the maximum power upward offset and maximum downward adjustment capability of the flexibility resource relative to the current operating point. Indicates the maximum power downward offset that the flexibility resource can make relative to the current operating point; response delay T R It indicates the time required for the flexibility resource to fully reach the required power after receiving the adjustment command, which corresponds to the start and stop time of the traditional unit; the response duration T D It indicates the maximum time that the flexibility resource can maintain the adjustment state after meeting the power adjustment requirements, which corresponds to the continuous start and stop time of the traditional unit; the adjustment rate R includes the upward adjustment rate R k,up and downward adjustment rate R k,down , respectively represent the maximum value of the upward and downward power adjustment of the flexibility resource per unit time, corresponding to the ramp rate of the traditional unit; (2) Construct the reliability index of the adjustable capacity of flexible resources in the virtual power plant, including the probability of response time of insufficient power, the expected value of response time of insufficient power, and the expected value of response time of insufficient power; the probability of response time of insufficient power γ LOLP represents the probability that the flexible resource cannot reach the specified adjustment range due to uncertain factors after receiving the adjustment instruction; the expected time of response to insufficient power γ LOLE It represents the expected value of the time when the adjustable capacity shortage of flexible resources is greater than or equal to the backup capacity; the expected value of the response power shortage γ EENS It indicates that the flexibility resources are affected by uncertainty factors, resulting in insufficient response, thereby reducing the expected value of power supply; (3) Construct a virtual power plant day-ahead and real-time bidding model based on constraint-aware reinforcement learning, including a virtual power plant day-ahead bidding model and a virtual power plant real-time bidding model; (4) Construct the objective function and constraints of the virtual power plant day-ahead bidding model. The objective function of the virtual power plant day-ahead bidding model is to maximize its own market revenue from participating in the day-ahead market. The constraints of the virtual power plant day-ahead bidding model include the upper and lower limits of the virtual power plant's overall power and the ramp rate constraint. (5) Construct the objective function and constraints of the virtual power plant real-time quotation model. The objective function of the virtual power plant real-time quotation model is to optimize the quotation results of the period. The constraints of the virtual power plant real-time quotation model are the same as those of the virtual power plant day-ahead quotation model, only the time scale is converted into the real-time scale; (6) Mixed integer programming-deep Q network planning is used to solve the virtual power plant day-ahead-real-time bidding model.

2. The dynamic aggregation bidding method for virtual power plants based on reinforcement learning according to claim 1 is characterized in that: Step (1) includes: in, is the power upper limit of flexibility resource k; P k,t is the power of flexibility resource k at time t; in, is the lower power limit of flexibility resource k; in, and are the response delays of upward and downward adjustment of flexibility resource k, respectively; t ds,up and t ds,down are the moments when the flexibility resources complete upward and downward adjustments, respectively; t ss The moment when the flexibility resource receives the response instruction; in, and are the response durations of upward and downward regulation of flexibility resource k, respectively; t de,up and t de,down The maximum time for the flexibility resource to maintain the adjustment state after completing the upward and downward adjustments respectively; 3. The dynamic aggregation bidding method for virtual power plants based on reinforcement learning according to claim 1 is characterized in that: Step (2) includes: Among them, l is the system operation status; p l is the probability of insufficient response power when the system is in running state l; t l The duration of the response power shortage when the system is in running state l; Among them, I t is the number of time periods a day is divided into; t is the time within the time period; T i is the number of hours included in the i-th period; C i is the theoretical maximum adjustment capacity of the flexibility resource in the i-th period; L i,t is the peak load power at time t in the i-th period; P[X≥C i -L i,t ] is the probability that the adjustable capacity shortage of flexible resources is greater than or equal to the spare capacity; Where N is the number of simulation hours; C I is the theoretical maximum adjustment capacity of the flexibility resource in the first simulation hour; L I is the load power in the first simulation hour; P I (X) is the probability of insufficient response power in the first simulation hour.

4. The dynamic aggregation bidding method for virtual power plants based on reinforcement learning according to claim 1 is characterized in that: Step (3) includes: reporting the pre-dispatching strategy to the dispatching agency; when optimizing the dispatching strategy on the cloud platform, in order to take uncertainty into account, generating several scenarios for random optimization, and predicting the clearing quantity and price of the real-time market, and selecting the strategy based on the highest combined day-ahead and real-time market returns under each scenario; After the virtual power plant cloud platform completes the day-ahead market quotation, it obtains more accurate real-time forecast data in the intraday stage, allowing it to participate in the real-time market to correct its own reported electricity volume; among them, the electricity cleared in the day-ahead stage is settled according to the day-ahead price, and the correction amount for the day-ahead stage in the real-time stage is settled according to the real-time market clearing electricity price; the real-time stage uses a rolling time window for optimization, taking the current moment and the subsequent time period as the time window, and referring to the real-time forecast information to perform optimization operation within the time period to obtain the optimal operation scheduling result for the time period, but the virtual power plant cloud platform only reports the optimization result of the current moment when reporting.

5. The dynamic aggregation bidding method for virtual power plants based on reinforcement learning according to claim 1 is characterized in that: In step (4), the optimization objective function of the virtual power plant's day-ahead quotation is as follows: Among them, π ω is the probability of the scenario ω occurring; T represents the time of a scheduling cycle; are the bidding power of the virtual power plant cloud platform at time t in the day-ahead phase, and ω is the decision variable of the day-ahead optimization scheduling problem; are the predicted values ​​of the clearing electricity prices of the day-ahead energy market and the reserve market by the virtual power plant cloud platform at time t; T RT is the number of time periods divided into each hour on the day-ahead time scale; are the winning bids in the real-time electric energy market and the reserve market under scenario ω, respectively. In the day-ahead phase, their values ​​are predicted values; They are the electricity market price and the reserve market price in the real-time stage respectively; is the overall operating cost function of the virtual power plant, which is expressed in the form of a quadratic function, and its parameters are the maximum values ​​in the flexibility resource cost function; is the total amount of electricity that the virtual power plant won in the spot market under the scenario ω, expressed as:

6. The virtual power plant dynamic aggregation bidding method based on reinforcement learning according to claim 5 is characterized in that: The upper and lower limits of the overall power of the virtual power plant are: in, The output power reported by the virtual power plant at time t when participating in the day-ahead electricity energy-reserve market; and They represent the maximum output power boundary and minimum power output boundary of the virtual power plant at time t respectively; Virtual power plant ramp rate constraints: in, and They represent the lower and upper boundaries of the virtual power plant power change rate respectively.

7. The virtual power plant dynamic aggregation bidding method based on reinforcement learning according to claim 1 is characterized in that: Step (5) includes: the objective function of the virtual power plant real-time quotation model is as follows: in, are the electricity reported by the virtual power plant cloud platform at real-time market moment n, and are the decision variables for the optimization operation in the real-time stage; Δt' is the length of the rolling optimization time window, which is an integer value; are the clearing electricity prices of the energy market and reserve market predicted by the cloud platform at the real-time market moment n; They are the adjustment of the output power by the virtual power plant cloud platform in real time at time t; T RT The number of scheduling intervals for the real-time phase; They are the cleared quantities of the day-ahead electricity market and the reserve market after the cleared quantities are expanded to the real-time time scale.

8. The virtual power plant dynamic aggregation bidding method based on reinforcement learning according to claim 1 is characterized in that: Step (6) includes: transforming the virtual power plant day-ahead-real-time bidding problem into a Markov decision process, where the basic features of the process are {s, a, P, R}, which represent the state space, action space, state transition matrix, and reward, respectively; 1) State Space When the state space corresponds to the virtual power plant day-ahead-real-time bidding model, the state set is expressed as: Among them, s represents the state set, which represents the set of multiple state information of the virtual power plant at a certain moment; P VPP,t-1 is the power of the virtual power plant at time t-1; E t-1 is the state of charge of the energy storage at time t-1; is time i at t-1 F The power of the devices in the collection; i F The collection includes photovoltaic PV, wind power W, energy storage ESS, distributed generator DG and grid AC; i b Represents the optimization cycle index, and N represents the total number of cycles; 2) Action Space The action space consists of the actions of each agent, which corresponds to the decision variables for the optimal operation of each flexibility resource in this problem; α1＝{P VPP,t} (18) 3) Rewards The agent's reward set is the objective function value in the model.

9. A dynamic aggregation bidding system for virtual power plants based on reinforcement learning, characterized in that: include: Evaluation index construction module, used to construct performance indicators and reliability indicators of the flexible resource adjustability in virtual power plants; A model building module is used to build a virtual power plant day-ahead and real-time bidding model based on constraint-aware reinforcement learning, including a virtual power plant day-ahead bidding model, its objective function and constraints, and a virtual power plant real-time bidding model, its objective function and constraints; The model solving module is used to solve the day-ahead and real-time bidding model of the virtual power plant using mixed integer programming-deep Q network planning.

10. A virtual power plant dynamic aggregation bidding device based on reinforcement learning, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the dynamic aggregation bidding method for virtual power plants based on reinforcement learning as described in any one of claims 1 to 8.

Images