Virtual power plant electric energy-standby day-ahead transaction method and system based on reinforcement learning

Abstract

The invention discloses a reinforcement learning-based virtual power plant electric energy-standby day-ahead transaction method and system, and belongs to the technical field of virtual power plants. The reinforcement learning-based virtual power plant electric energy-reserve day-ahead transaction method comprises the following steps of: starting from MT power generation cost, electricity purchasing and selling cost of a consumer, reserve purchasing and selling cost of the consumer, energy utilization efficiency of the consumer and reserve cost of the consumer, carrying out consumer electric energy-reserve transaction cost-benefit analysis; constructing a local market clearing model based on the electricity energy-reserve transaction cost benefit analysis result of the consumer, and solving the local market clearing model through an ADMM algorithm; constructing a virtual power plant decision-making model based on the cost-benefit analysis of the virtual power plant, and solving the virtual power plant decision-making model through a DDPG algorithm; and integrating solving results of the local market clearing model and the virtual power plant decision model to obtain an optimal decision result of the virtual power plant and the producer and consumer.

Description

Technical Field

[0001] The present invention belongs to the technical field of virtual power plants, and specifically relates to a method and system for day-ahead trading of electric energy and reserve energy of a virtual power plant based on reinforcement learning. Background Art

[0002] With the access of massive distributed renewable energy and the rapid growth of flexible loads such as user-side energy storage and air conditioning, end users have gradually evolved into "prosumers" with dual source and load characteristics, and the system flexibility has been significantly improved.

[0003] Aggregators usually act as agents of producers and consumers, participate in market transactions and system operations by raising demand-side resources, and realize the value connection between the supply side and the demand side while transmitting wholesale market price signals. However, the reliable implementation of the above methods often relies on a vertically integrated hosting model. Aggregators directly dispatch the distributed resources of producers and consumers while transmitting market signals. Technically, they realize a series of operations such as monitoring, forecasting, and controlling subordinate resources. Economically, they act as agents for subordinate resources to participate in market transactions and execute market transaction results. In this process, the autonomous willingness and transaction preferences of producers and consumers to explore their own regulatory potential cannot be fully taken into account, and direct load control also involves user privacy issues. How to coordinate the interactive relationship between multiple subjects to promote their mutual benefit and win-win situation still requires further research.

[0004] In addition, the access of a high proportion of distributed renewable energy has greatly increased the randomness of real-time power. Most existing studies do not require real-time power balance in the distribution network market. The backup required by the system is still provided by the adjustable units on the source side, and the system operator distributes the backup cost to the entire network. The deviation assessment costs of post-settlement cannot accurately quantify the actual value of the imbalance between supply and demand in each period, and realize the traceability and recovery of the adjustment costs. The dependence of distributed renewable energy entities on the main grid and the unequal rights and responsibilities of the penalty mechanism have led to their lack of motivation to actively improve their power supply reliability. In the future, when the penetration rate of renewable energy is high and the adjustable resources of the main grid are limited, there will be serious safety hazards. At the same time, the bottom-level prosumers have no way to provide flexibility services to the main grid, and the low utilization rate of the existing cheap and high-quality adjustable resources will lead to an increase in system operating costs. Summary of the invention

[0005] In view of the deficiencies in the prior art, the purpose of the present invention is to provide a virtual power plant electricity-reserve day-ahead trading method and system based on reinforcement learning, which solves the problems in the prior art.

[0006] The purpose of the present invention can be achieved through the following technical solutions:

[0007] The virtual power plant power-reserve day-ahead trading method based on reinforcement learning includes the following steps:

[0008] Based on the MT power generation cost, the power purchase and sale cost of the prosumer, the purchase and sale cost of the prosumer reserve, the energy efficiency of the prosumer, and the reserve cost reserved by the prosumer, a cost-benefit analysis of the prosumer power-reserve transaction is conducted;

[0009] Based on the cost-benefit analysis results of the prosumer power-reserve transaction, a local market clearing model is constructed, and the local market clearing model is solved by the ADMM algorithm;

[0010] Based on the cost-benefit analysis of virtual power plants, a virtual power plant decision model is constructed and solved using the DDPG algorithm;

[0011] By integrating the solution results of the local market clearing model and the virtual power plant decision model, the optimal decision results of the virtual power plant and the prosumer users are obtained.

[0012] Furthermore, the MT power generation cost The calculation formula is:

[0013]

[0014] In the formula, i∈[I] represents the set of producers and consumers, t∈[T] represents the optimization period, and P i MT (t) represents the active output of MT of prosumer i in time interval t, a i , b i 、c i is the cost function coefficient, Δt is the duration of a single optimization period;

[0015] The cost of electricity purchase and sale by the prosumer The calculation formula is:

[0016]

[0017] In the formula, is the electricity purchase price that prosumer i trades with the virtual power plant during time interval t, P represents the electricity price that prosumer i trades with the virtual power plant in time interval t. i net,buy (t) and P i net,sell (t) respectively represent the amount of electricity purchased and sold by prosumer i with the virtual power plant in the time interval t;

[0018] The cost of purchasing and selling spare parts by the prosumer The calculation formula is:

[0019]

[0020] In the formula, denote respectively the positive reserve quantity, positive reserve price, negative reserve quantity, and negative reserve price purchased by prosumer i from VPP in time interval t, and the positive reserve quantity, positive reserve price, negative reserve quantity, and negative reserve price sold by prosumer i from VPP in time interval t;

[0021] The prosumer energy efficiency The calculation formula is:

[0022]

[0023] Where P i FL (t) They represent the operating power of FL of prosumer i in time interval t respectively; is a non-negative utility coefficient; and represents the FL operating power limit of prosumer i;

[0024] The prosumer reserves the reserve cost The calculation formula is:

[0025]

[0026] In the formula, They represent the positive and negative spare capacities that MT and FL of prosumer i can provide in time interval t, respectively. They represent the unit costs of MT and FL of prosumer i for providing positive and negative spare capacity, respectively.

[0027] Furthermore, the objective function of the local market clearing model is:

[0028]

[0029] In the formula, They are electricity purchase and sales costs, reserve costs, gas-fired power generation costs, reserved reserve costs, energy use utility and reserved reserve utility.

[0030] Furthermore, the constraints of the local market clearing model are:

[0031]

[0032] Where P i pv (t) represents the predicted output of PV of prosumer i at time interval t, P i load (t) represents the load demand of producer / consumer i in time interval t, and They represent the net electricity purchase demand and net electricity sales demand of all prosumers respectively.

[0033] Further, the cost-benefit analysis of virtual power plants includes:

[0034] 1) ES loss cost of virtual power plants

[0035]

[0036] In the formula, and They represent the charging and discharging power of the ES of the virtual power plant in the time interval t, η ES represents the loss coefficient of ES;

[0037] 2) Virtual power plant reserve cost

[0038]

[0039] In the formula, and They represent the reserved charging and discharging capacity of the ES of the virtual power plant in time interval t, π ES It represents the unit cost of ES providing positive and negative reserve capacity;

[0040] 3) Cost of purchasing and selling electricity in the virtual power plant wholesale market

[0041]

[0042] In the formula, They represent the net power demand of the virtual power plant in time interval t, represents the predicted value of the wholesale market electricity price of the virtual power plant for the time interval t before the day;

[0043] 4) Purchase and sale of reserve costs in the virtual power plant wholesale market

[0044]

[0045] In the formula, They represent the net positive and negative reserve capacity of the virtual power plant in the time interval t, represents the predicted value of positive and negative reserve prices in the wholesale market for the time interval t before the day by the virtual power plant;

[0046] 5) Virtual power plant local market electricity sales revenue

[0047]

[0048] In the formula, represents the price of electricity sold by the virtual power plant to all prosumers in time interval t; represents the price of electricity purchased by the virtual power plant from all prosumers in time interval t;

[0049] 6) Virtual power plant local market reserve sales revenue

[0050]

[0051] In the formula, represents the price at which the virtual power plant purchases positive and negative reserves from all prosumers in time interval t; It represents the price at which the virtual power plant sells positive and negative reserves to all prosumers in time interval t.

[0052] Furthermore, the virtual power plant decision model uses the operating cost of the virtual power plant as the reward function value, and its reward function is:

[0053]

[0054] In the formula, C re is the operating cost of the virtual power plant.

[0055] Furthermore, the constraints of the virtual power plant decision model include:

[0056]

[0057]

[0058] In the formula, k∈{P,R + ,R -}, *∈{buy,sell}, It represents the price of energy and reserve purchased and sold by VPP to prosumers during period t, represents the upper and lower limits of VPP pricing during period t, w * Adjust parameters for pricing.

[0059] The virtual power plant energy-reserve day-ahead trading system based on reinforcement learning includes:

[0060] Cost-benefit analysis module: Based on MT power generation cost, prosumer power purchase and sales cost, prosumer reserve purchase and sales cost, prosumer energy efficiency and prosumer reserved reserve cost, the cost-benefit analysis of prosumer power-reserve transaction is carried out;

[0061] Local market clearing model solving module: based on the cost-benefit analysis results of the power-reserve transaction of the prosumer, a local market clearing model is constructed, and the local market clearing model is solved by the ADMM algorithm;

[0062] Virtual power plant decision model solving module: Based on the cost-benefit analysis of virtual power plants, a virtual power plant decision model is constructed and solved using the DDPG algorithm;

[0063] And, the optimal decision result output module: integrates the solution results of the local market clearing model and the virtual power plant decision model to obtain the optimal decision results of the virtual power plant and the prosumer user.

[0064] A computer storage medium stores a readable program, which can execute the above-mentioned virtual power plant electricity-reserve day-ahead trading method based on reinforcement learning when the program is running.

[0065] An electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0066] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned virtual power plant electricity-reserve day-ahead trading method based on reinforcement learning.

[0067] Beneficial effects of the present invention:

[0068] The present invention adopts energy-reserve joint trading to increase the market revenue of VPP and realize the traceability and sharing of the reserve costs of producers and consumers;

[0069] The local market of the transaction method of the present invention increases the bargaining power of prosumers and VPPs, and the abundance of power generation resources and the backup cost of adjustable resources can significantly affect the transaction benefits of prosumers. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0071] Figure 1 It is a flow chart of the virtual power plant power-reserve day-ahead trading method of the present invention;

[0072] Figure 2 It is a flow chart for solving the local market clearing model and the virtual power plant decision model in the present invention;

[0073] Figure 3 is the convergence graph of the reinforcement learning algorithm in the present invention;

[0074] Figure 4 is the power and reserve balance diagram of the prosumer in the present invention;

[0075] Figure 5 It is a VPP transaction and operation cost diagram in the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] like Figure 1 As shown, the virtual power plant power-reserve day-ahead trading method based on reinforcement learning includes the following steps:

[0078] S1, starting from MT power generation cost, prosumer power purchase and sale cost, prosumer reserve purchase and sale cost, prosumer energy use utility and prosumer reserved reserve cost, conduct cost-benefit analysis of prosumer power-reserve transaction;

[0079] 1) MT power generation cost;

[0080] MT power generation cost It can be approximated as a quadratic curve of its output, which can be specifically expressed as follows:

[0081]

[0082] In the formula, i∈[I] represents the set of producers and consumers, t∈[T] represents the optimization period, and P i MT (t) represents the active output of MT of prosumer i in time interval t, a i , b i 、c i is the cost function coefficient, and Δt is the duration of a single optimization period.

[0083] 2) The cost of electricity purchase and sale by prosumers;

[0084] If the prosumer participates in the local market and trades with the virtual power plant at the same time, the above two aspects of the prosumer's electricity purchase and sales costs are:

[0085]

[0086] In the formula, is the electricity purchase price that prosumer i trades with the virtual power plant during time interval t, P represents the electricity price that prosumer i trades with the virtual power plant in time interval t. i net,buy (t) and P inet,sell (t) respectively represent the amount of electricity purchased and sold by prosumer i with the virtual power plant in the time interval t; They represent the prices of P2P energy trading between prosumer i and prosumer j in time interval t.

[0087] 3) The cost of purchasing and selling spare parts by prosumers;

[0088] Similarly, prosumers can also conduct P2P backup transactions with upper-layer VPPs or prosumers. The above two backup purchase costs are:

[0089]

[0090] In the formula, denote respectively the positive reserve quantity, positive reserve price, negative reserve quantity, and negative reserve price purchased by prosumer i from VPP in time interval t, and the positive reserve quantity, positive reserve price, negative reserve quantity, and negative reserve price sold by prosumer i from VPP in time interval t; They represent the positive and negative reserve quantities and prices of P2P transactions between prosumer i and prosumer j in time interval t.

[0091] 4) Energy efficiency of prosumers;

[0092] The user's flexible load can provide welfare needs for production and life, and the energy efficiency of the prosumer can be expressed by a quadratic function.

[0093]

[0094] Where P i FL (t) They represent the operating power of FL of prosumer i in time interval t respectively; is a non-negative utility coefficient; and represents the FL operating power limit of prosumer i.

[0095] 5) Prosumers reserve spare costs;

[0096] The real-time power deviation caused by distributed photovoltaic and load uncertainty may cause the prosumer to face economic penalties. Therefore, each prosumer should reserve or purchase backup to avoid the potential risks caused by forecast deviation; the cost of reserve reserves for prosumers is for;

[0097]

[0098] In the formula, They represent the positive and negative spare capacities that MT and FL of prosumer i can provide in time interval t, respectively. They represent the unit costs of MT and FL of prosumer i for providing positive and negative spare capacity, respectively.

[0099] S2, based on the cost-benefit analysis results of the prosumer power-reserve transaction in S1, construct a local market clearing model and solve the local market clearing model by using the ADMM (alternating direction method of multipliers) algorithm;

[0100] The local market clearing model aims to maximize social welfare, and the objective function is:

[0101]

[0102] The clearing should satisfy the following power balance constraints:

[0103]

[0104] In the process of using the ADMM algorithm to solve the local market clearing model, an auxiliary variable y is introduced i =[Y ij ]Rewrite the coupling constraints to make them conform to the form of ADMM algorithm solution. For the sake of writing introduction, the trading period (t) is omitted. X ij ∈x i

[0105]

[0106] From an economic perspective, the Lagrange multiplier λ corresponding to the P2P transaction constraint is i,t =[λ ij ], i / j∈Ω, i≠j represents the shadow price of P2P transactions, then the augmented Lagrangian function of the optimization problem is:

[0107]

[0108] in:

[0109]

[0110] like Figure 2 As shown, let v be the number of iterations, then the ADMM solution process is as follows:

[0111] 1) Update local decision variables;

[0112] Producer i obtains the value obtained in the previous iteration Solve the local optimization decision problem:

[0113]

[0114] And will update Passed to prosumer j.

[0115] 2) Update auxiliary variables;

[0116] Solve the following global optimization problem to update the auxiliary variables:

[0117]

[0118] In order to realize decentralized transactions between producers and consumers and take into account both solution speed and privacy protection, the coupled part in the above optimization problem (15) is separated and differentiated to obtain Y ij Analytical solution of .

[0119]

[0120] Thus, prosumer i can independently update the auxiliary variable y i , without relying on a central platform for global optimization, achieving fully distributed solving.

[0121] 3) Update the Lagrange multiplier;

[0122] The original residual and dual residual calculation formula of the algorithm are as follows:

[0123]

[0124] The iterative convergence criterion of the ADMM algorithm is:

[0125] r v ≤ε 1 ,s v ≤ε 1 (twenty one)

[0126] ε 1 is the set error accuracy.

[0127] S3, based on the cost-benefit analysis of virtual power plants (VPPs), a virtual power plant (VPP) decision model is constructed, and the virtual power plant decision model is solved by the DDPG (Deep Deterministic Policy Gradient) algorithm;

[0128] In a transaction cycle, the cost-benefit analysis of a virtual power plant (VPP) includes the following:

[0129] 1) ES loss cost of virtual power plants

[0130] Considering the charging and discharging efficiency of ES and its operating losses, the relevant costs are:

[0131]

[0132] In the formula, and They represent the charge and discharge power of ES of VPP in time interval t, η ES Represents the loss coefficient of ES.

[0133] 2) Virtual power plant reserve cost

[0134] Considering that part of the reserved ES charging and discharging capacity cannot be used for electricity trading, the relevant costs are:

[0135]

[0136] In the formula, and They represent the reserved charge and discharge capacity of ES of VPP in time interval t, π ES It represents the unit cost of positive and negative reserve capacity provided by ES.

[0137] 3) Cost of purchasing and selling electricity in the virtual power plant wholesale market

[0138] VPP can participate in wholesale market transactions as an independent market player. If VPP is considered as a price acceptor in the wholesale market, the electricity purchase cost of VPP in the wholesale market is:

[0139]

[0140] In the formula, They represent the net power demand of VPP in time interval t, It represents the predicted value of wholesale market electricity price by VPP for the time interval t before the day.

[0141] 4) Purchase and sale of reserve costs in the virtual power plant wholesale market

[0142] Similarly, the purchase and reserve cost of the VPP wholesale market is:

[0143]

[0144] In the formula, They represent the net positive and negative reserve capacity required by VPP in time interval t, It represents the predicted value of positive and negative reserve prices in the wholesale market by VPP for the time interval t before the day.

[0145] 5) Virtual power plant local market electricity sales revenue

[0146] As the connecting entity between the wholesale market and the local market, VPP can set the transaction price in the local market and resell the energy purchased from the wholesale side. The income of VPP from selling electricity in the local market is:

[0147]

[0148] In the formula, represents the price of electricity sold by the virtual power plant to all prosumers in time interval t; Represents the price at which the virtual power plant purchases electricity from all prosumers in time interval t.

[0149] 6) Virtual power plant local market reserve sales revenue

[0150] Similarly, the revenue from selling reserve in the VPP local market is,

[0151]

[0152] In the formula, represents the price at which the virtual power plant purchases positive and negative reserves from all prosumers in time interval t; It represents the price at which the virtual power plant sells positive and negative reserves to all prosumers in time interval t.

[0153] Based on the cost-benefit analysis of virtual power plants, determine the reinforcement learning scheme corresponding to the VPP decision model, including state space, action space and reward function;

[0154] (1) State space set

[0155] The environmental state set mainly includes time t, transaction information of the local market, including the amount of electricity purchased and sold and the amount of reserve transactions between producer and consumer i and VPP within the time interval t, that is:

[0156]

[0157] (2) Action Space Set

[0158] For virtual power plants, as price makers, they adjust energy price strategies to maximize their own benefits. The transaction price of virtual power plant.

[0159]

[0160] At each trading moment, prosumers will optimize their trading strategies based on the virtual power plant pricing information and update the state space.

[0161] (3) Reward Function

[0162] Operating cost based on virtual power plant C re , the VPP decision reinforcement learning model is based on the operating cost C of the virtual power plant re As the reward function value:

[0163]

[0164] In order to encourage prosumers to trade with VPPs and balance the impact of traded electricity and trading benefits on total revenue, VPPs should meet the following constraints when setting subscription and sales prices:

[0165]

[0166] In the formula, k∈{P,R + ,R -}, *∈{buy,sell}, It represents the price of energy and reserve purchased and sold by VPP to prosumers during period t, represents the upper and lower limits of VPP pricing during period t, constraint (32) indicates that the price of energy sales and reserve should be higher than the purchase price, w * is the pricing adjustment parameter. When a constraint is violated, a reward function with a negative value, i.e., the degree of constraint deviation, is added to the initial reward function (30).

[0167] S4, integrates the solution results of the local market clearing model and the virtual power plant decision model to obtain the optimal decision results for the virtual power plant and the prosumer user.

[0168] Example 2

[0169] In this embodiment, the virtual power plant electricity-reserve day-ahead trading method based on reinforcement learning proposed in Embodiment 1 is experimentally verified to illustrate the superiority of the trading method proposed in the present invention (Embodiment 1).

[0170] 1. Parameter setting;

[0171] Four prosumers with intelligent energy management systems are used as examples for case analysis. Δt = 1h, air conditioners are selected as typical FL devices, and the operating power limits of a single air conditioner are set to 1.6 and 2kW respectively, and each prosumer is equipped with 60 units. The parameters of prosumers and VPPs are shown in Tables 1-3. The convergence error accuracy of ADMM is ε 0.05. The model is solved by calling Gurobi10.0.3 through MATLAB R2021b.

[0172] Set up the following scenario for case analysis:

[0173] Scenario 1 / Scenario 2: The prosumer only trades with the VPP and does not consider / considers backup;

[0174] Scenario 3 / Scenario 4: Prosumers can trade with VPPs and participate in local market P2P transactions without / without backup;

[0175] Table 1 Prosumer parameter settings

[0176]

[0177] Table 2 Virtual power plant ES parameters

[0178]

[0179] Table 3 Alternative parameters

[0180]

[0181] 2. DDPG training results;

[0182] The DDPG training convergence results are as follows Figure 3 As shown, the cumulative rewards of each round gradually converge.

[0183] 3. Analysis of the results of optimized operations of producers and consumers;

[0184] like Figure 4 As shown in (a), (b) and (c) in Figure 2, in terms of P2P transactions, the energy transactions of prosumers are all during the noon period when photovoltaic power generation is high, while standby transactions are distributed in different periods of the day. This is because the marginal cost of photovoltaic power generation is very low, and prosumers have strong bargaining power during the period when photovoltaic power generation is much greater than the load. At this time, the profit space formed by the VPP purchase and sale price is large. The existence of the local market can encourage prosumers to further tap their own regulation potential and increase the resource scheduling benefits while ensuring the comfort of electricity use.

[0185] 4. Analysis of transaction results between producers and consumers and VPPs

[0186] Depend on Figure 5 It can be seen that under the parameter configuration of this article, VPP's energy trading income accounts for a large proportion during the peak load period of the prosumer in the evening. This is because prosumers do not have energy storage resources and lack the support of photovoltaic power generation at night. The established electricity demand puts them at a disadvantage in the market supply and demand relationship, so VPP can use the price difference to expand its own profits. In terms of backup trading, VPP can use the regulation capacity of self-distributed energy storage to provide positive and negative backup for prosumers and obtain income. Due to the influence of photovoltaic uncertainty, prosumers have a large backup demand during the noon period during the day, and VPP's sales income during the corresponding period is also large.

[0187] The method of the present invention may be implemented in hardware, firmware, or as software or computer code that may be stored in a recording medium (such as a CDROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded over a network and will be stored in a local recording medium, so that the method described herein may be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, processor, or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the method shown herein.

[0188] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A virtual power plant power-reserve day-ahead trading method based on reinforcement learning, characterized in that: The following steps are involved: Based on the MT power generation cost, the power purchase and sale cost of the prosumer, the purchase and sale cost of the prosumer reserve, the energy efficiency of the prosumer, and the reserve cost reserved by the prosumer, a cost-benefit analysis of the prosumer power-reserve transaction is conducted; Based on the cost-benefit analysis results of the prosumer power-reserve transaction, a local market clearing model is constructed, and the local market clearing model is solved by the ADMM algorithm; Based on the cost-benefit analysis of virtual power plants, a virtual power plant decision model is constructed and solved using the DDPG algorithm; By integrating the solution results of the local market clearing model and the virtual power plant decision model, the optimal decision results of the virtual power plant and the prosumer users are obtained.

2. The virtual power plant power-reserve day-ahead trading method based on reinforcement learning according to claim 1 is characterized in that: The MT power generation cost The calculation formula is: In the formula, i∈[I] represents the set of producers and consumers, t∈[T] represents the optimization period, and P i MT (t) represents the active output of MT of prosumer i in time interval t, a i , b i 、c i is the cost function coefficient, Δt is the duration of a single optimization period; The cost of electricity purchase and sale by the prosumer The calculation formula is: In the formula, is the electricity purchase price that prosumer i trades with the virtual power plant during time interval t, P represents the electricity price that prosumer i trades with the virtual power plant in time interval t. i net,buy (t) and P i net,sell (t) respectively represent the amount of electricity purchased and sold by prosumer i with the virtual power plant in the time interval t; The cost of purchasing and selling reserve by the prosumer The calculation formula is: In the formula, denote respectively the positive reserve quantity, positive reserve price, negative reserve quantity, and negative reserve price purchased by prosumer i from VPP in time interval t, and the positive reserve quantity, positive reserve price, negative reserve quantity, and negative reserve price sold by prosumer i from VPP in time interval t; The prosumer energy efficiency The calculation formula is: Where P i FL (t) They represent the operating power of FL of prosumer i in time interval t respectively; is a non-negative utility coefficient; and represents the FL operating power limit of prosumer i; The prosumer reserves the reserve cost The calculation formula is: In the formula, They represent the positive and negative spare capacities that MT and FL of prosumer i can provide in time interval t, respectively. They represent the unit costs of MT and FL of prosumer i for providing positive and negative spare capacity, respectively.

3. The virtual power plant power-reserve day-ahead trading method based on reinforcement learning according to claim 2 is characterized in that: The objective function of the local market clearing model is: In the formula, They are electricity purchase and sales costs, reserve costs, gas-fired power generation costs, reserved reserve costs, energy use utility and reserved reserve utility.

4. The virtual power plant power-reserve day-ahead trading method based on reinforcement learning according to claim 3 is characterized in that: The constraints of the local market clearing model are: Where P i pv (t) represents the predicted output of PV of prosumer i at time interval t, P i load (t) represents the load demand of producer / consumer i in time interval t, and They represent the net electricity purchase demand and net electricity sales demand of all prosumers respectively.

5. The virtual power plant power-reserve day-ahead trading method based on reinforcement learning according to claim 1 is characterized in that: The cost-benefit analysis of a virtual power plant includes: 1) ES loss cost of virtual power plants In the formula, and They represent the charging and discharging power of the ES of the virtual power plant in the time interval t, η ES represents the loss coefficient of ES; 2) Virtual power plant reserve cost In the formula, and They represent the reserved charging and discharging capacity of the ES of the virtual power plant in time interval t, π ES It represents the unit cost of ES providing positive and negative reserve capacity; 3) Cost of purchasing and selling electricity in the virtual power plant wholesale market In the formula, They represent the net power demand of the virtual power plant in time interval t, represents the predicted value of the wholesale market electricity price of the virtual power plant for the time interval t before the day; 4) Purchase and sale costs of virtual power plant wholesale market In the formula, They represent the net positive and negative reserve capacity of the virtual power plant in the time interval t, represents the predicted value of positive and negative reserve prices in the wholesale market for the time interval t before the day; 5) Virtual power plant local market electricity sales revenue In the formula, represents the price of electricity sold by the virtual power plant to all prosumers in time interval t; represents the price of electricity purchased by the virtual power plant from all prosumers in time interval t; 6) Virtual power plant local market reserve sales revenue In the formula, represents the price at which the virtual power plant purchases positive and negative reserves from all prosumers in time interval t; It represents the price at which the virtual power plant sells positive and negative reserves to all prosumers in time interval t.

6. The virtual power plant power-reserve day-ahead trading method based on reinforcement learning according to claim 5 is characterized in that: The virtual power plant decision model uses the operating cost of the virtual power plant as the reward function value, and its reward function is: In the formula, C re is the operating cost of the virtual power plant.

7. The virtual power plant power-reserve day-ahead trading method based on reinforcement learning according to claim 6 is characterized in that: The constraints of the virtual power plant decision model include: In the formula, k∈{P,R + ,R - }, *∈{buy,sell}, It represents the price of energy and reserve purchased and sold by VPP to prosumers during period t, represents the upper and lower limits of VPP pricing during period t, w * Adjust parameters for pricing.

8. A virtual power plant power-reserve day-ahead trading system based on reinforcement learning, characterized in that: include: Cost-benefit analysis module: Based on MT power generation cost, prosumer power purchase and sales cost, prosumer reserve purchase and sales cost, prosumer energy efficiency and prosumer reserved reserve cost, the cost-benefit analysis of prosumer power-reserve transaction is carried out; Local market clearing model solving module: based on the cost-benefit analysis results of the power-reserve transaction of the prosumer, a local market clearing model is constructed, and the local market clearing model is solved by the ADMM algorithm; Virtual power plant decision model solving module: Based on the cost-benefit analysis of virtual power plants, a virtual power plant decision model is constructed and solved using the DDPG algorithm; And, the optimal decision result output module: integrates the solution results of the local market clearing model and the virtual power plant decision model to obtain the optimal decision results of the virtual power plant and the prosumer user.

9. A computer storage medium storing a readable program, characterized in that: When the program is running, it can execute the virtual power plant electricity-reserve day-ahead trading method based on reinforcement learning as described in any one of claims 1-7.

10. An electronic device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the virtual power plant electricity-reserve day-ahead trading method based on reinforcement learning as described in any one of claims 1-7.

Images