Virtual power plant multi-type power commodity joint transaction game method and system

Abstract

The invention relates to the technical field of power transaction, in particular to a virtual power plant multi-type power commodity joint transaction game method and system, and the method comprises the steps: constructing a multi-layer interaction framework between VPPO and VPP of multi-type power commodities; constructing a single VPP decision model by taking cost and utility minimization and market share maximization as targets; constructing a multi-virtual power plant operation optimization model based on different targets, and adopting distributed solution to obtain a local market transaction result; based on a multi-VPP operation optimization result and a local market transaction result, constructing a VPPO decision model by taking cost minimization as a target; the transaction process of the VPPO and the VPP is designed into a double-layer model, and the model is solved by adopting a data driving method of a DDPGAE algorithm to obtain a globally optimal power transaction strategy. According to the invention, the complexity problem of the virtual power plant in the joint transaction of multiple types of power commodities is effectively solved, the global optimal power transaction strategy is realized, and the market transaction efficiency and the resource configuration benefit are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power trading, and in particular to a virtual power plant multi-type power commodity joint trading game method and system. Background Art

[0002] With the rapid development of renewable energy and the widespread application of distributed energy resources (DERs), virtual power plants (VPPs) as an emerging energy management technology are playing an increasingly important role in the power market. Virtual power plants aggregate multiple DERs, such as photovoltaics, wind power, energy storage systems, and controllable loads, to form an energy supply system similar to traditional power plants, and are able to participate in various power market transactions, such as power transactions, ancillary service transactions, and demand response transactions. This aggregation can not only improve the economy and stability of DERs, but also enhance the flexibility and reliability of the power system.

[0003] However, virtual power plants face many challenges when participating in multi-type electricity commodity transactions, such as uncertainty in electricity supply and demand, complexity of market rules, and balance between economic and environmental benefits. In order to meet these challenges, an effective game theory method is needed to optimize the decision-making process of virtual power plants and ensure their efficient operation and maximum benefits in the electricity market. Summary of the invention

[0004] The present invention provides a virtual power plant multi-type electricity commodity joint trading game method and system, thereby effectively solving the problems pointed out in the background technology.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A virtual power plant multi-type power commodity joint trading game method, comprising:

[0007] Based on the principle of multi-level market price difference arbitrage, a multi-level interactive framework between VPPO and VPP for various types of electricity commodities is constructed;

[0008] According to the VPP resource allocation and market transaction optimization and based on the multi-layer interactive framework, a single VPP decision model is constructed with the goal of minimizing cost and utility and maximizing market share;

[0009] Based on the single VPP decision model, a multi-virtual power plant operation optimization model based on different objectives is constructed, and a distributed solution is used to solve the multi-VPP multi-type power commodity sharing problem to obtain local market transaction results;

[0010] Based on the optimization results of multiple VPP operations and local market transaction results, a VPPO decision model is constructed with the goal of minimizing costs;

[0011] Based on the theory of peer-to-peer non-cooperative game, the transaction process of VPPO and VPP is designed as a two-layer model. Based on the method of one party setting prices and the other party reporting quantities, the data-driven method of DDPG_AE algorithm is used to solve the model and obtain the globally optimal electricity trading strategy.

[0012] Furthermore, based on the principle of multi-level market price difference arbitrage, a multi-level interactive framework between VPPO and VPP of various types of electricity commodities is constructed, including:

[0013] According to the hierarchical structure of the electricity market, determine the types of electricity commodities in different levels of the market;

[0014] Based on the price fluctuations of multi-tier markets, potential arbitrage opportunities can be identified by calculating the price differences between different market tiers;

[0015] VPPO allocates different types of electricity commodities among various market tiers by formulating market participation strategies and pricing mechanisms;

[0016] Build an interactive framework between VPPO and VPP, and establish a two-way information transmission mechanism. VPPO transmits electricity commodity trading signals to VPP through price signals, and adjusts market strategies based on VPP's feedback.

[0017] Furthermore, the single VPP decision model is:

[0018]

[0019] In the formula, C i is the total cost of VPP, is the MT power generation cost, is the electricity purchase cost of VPP and VPPO transactions, is the electricity purchase cost for P2P transactions between VPPs, The cost of purchasing spare parts for VPP i, The purchase and standby cost of VPP i for P2P transactions, Purchase frequency modulation cost for VPP, The purchase frequency cost of VPP i for P2P transactions, Energy efficiency for VPP i, Reserve spare cost for VPP i.

[0020] Furthermore, in the early stage of market development, VPPs expand their market share and a single VPP seizes the local market model as follows:

[0021]

[0022] Where U i VPP I is the objective function for seizing market share, is the price at which VPP i trades with VPPO during period t; The power purchase and sale energy between VPP i and VPPO in period t; are the upper reserve and lower reserve prices purchased by VPP i in transactions with VPPO during period t; are the upper reserve and lower reserve prices of VPP i in transactions with VPPO during period t; is the upper reserve and lower reserve quantity purchased by VPP i in transactions with VPPO during period t; The upper reserve and lower reserve quantities sold by VPP i in transactions with VPPO during period t; is the purchase and sale frequency modulation price of VPP i in the transaction with VPPO during period t; is the purchase and sale frequency modulation volume of VPP i and VPPO in period t.

[0023] Furthermore, the augmented Lagrangian function of the optimization problem is:

[0024]

[0025] in, In the formula, C i (x i,t ) is the i-th VPP in decision x i,t The total cost under x i,t is the variable vector of the VPP decision model; i,t =Y ij,t is an auxiliary variable, Y ij,t is the electric energy, upper reserve, lower reserve and frequency regulation amount traded between VPP i and VPP j; is the Lagrange multiplier corresponding to the P2P transaction constraint; ρ t is the iteration step length; ||·||2 is the 2-norm.

[0026] Furthermore, the VPPO decision model is:

[0027]

[0028] In the formula, is the ESS loss cost of VPPO, Reserve spare cost for VPPO, is the cost of electricity purchase and sale in the VPPO wholesale market, Purchase spare costs for the VPPO wholesale market, The cost of purchasing frequency modulation for the wholesale market, The revenue from electricity sales in the local market of VPPO, Reserve income for VPPO local market sales, Sell ​​FM revenue for the local market.

[0029] Furthermore, VPPO uses reinforcement learning to continuously adjust the pricing strategy during the interaction process, which can be formalized as an MDP, including:

[0030] The state space is used to describe the current situation of the environment. The state space is represented as a vector reflecting the current state of the system.

[0031] An action space, which represents all possible actions or strategies that the agent can choose. The action space is a vector that combines the scheduling strategy and the pricing strategy to guide the specific actions taken by the agent in the decision-making process;

[0032] The reward function is used to reflect the optimization goal and calculate the immediate benefit or cost of the agent after performing a certain action in a certain state;

[0033] The state transition probability describes the probability that the environment is affected and transitions to the next state after performing a certain action in a given state.

[0034] Furthermore, the data-driven method of the DDPG_AE algorithm is used to solve the model and obtain the global optimal power trading strategy, including:

[0035] Initialize the network structure and parameters of the DDPG_AE algorithm, including the parameters of the actor network and critic network, the experience replay pool, and the adaptive exploration coefficient;

[0036] The agent obtains the current system state, selects appropriate trading actions based on the current strategy and Gaussian noise, executes actions based on market feedback, transfers to the next state and obtains corresponding rewards, and stores the quadruple in the experience replay pool;

[0037] When the number of samples in the experience replay pool reaches a preset threshold, the agent randomly samples batches of samples from the replay pool, calculates the target value and temporal difference error of the critic network, and updates the parameters of the actor network and the critic network, using a soft update mechanism to update the target network parameters;

[0038] The agent repeatedly performs sampling and training, gradually adjusts the exploration coefficient to optimize the strategy, and outputs the final actor network and critic network parameters when the stopping condition is met to obtain the globally optimal power trading strategy.

[0039] Furthermore, the DDPG_AE has a two-layer loop structure, including:

[0040] The outer loop represents one iteration of the algorithm;

[0041] The inner loop controls the parameter training at each timestamp in each iteration.

[0042] A virtual power plant multi-type power commodity joint trading game system, the system comprising:

[0043] The multi-layer interactive framework building module builds a multi-layer interactive framework between VPPO and VPP of various types of power commodities based on the principle of multi-level market price difference arbitrage;

[0044] A single decision model building module, which builds a single VPP decision model based on the resource allocation and market transaction optimization of the VPP and based on the multi-layer interactive framework, with the goal of minimizing cost and utility and maximizing market share;

[0045] The market transaction result acquisition module constructs a multi-virtual power plant operation optimization model based on different objectives based on the single VPP decision model, and adopts distributed solution to the multi-VPP multi-type power commodity sharing problem to obtain the local market transaction results;

[0046] The VPPO decision model building module builds the VPPO decision model based on the multi-VPP operation optimization results and local market transaction results with the goal of minimizing costs;

[0047] The optimal trading strategy acquisition module designs the trading process of VPPO and VPP as a two-layer model based on the theory of peer-to-peer non-cooperative game, and solves the model using the data-driven method of the DDPG_AE algorithm based on the method of one party setting prices and the other party reporting quantities, to obtain the global optimal electricity trading strategy.

[0048] The technical solution of the present invention can achieve the following technical effects:

[0049] It effectively solved the complexity problem of virtual power plants in the joint trading of multiple types of electricity commodities, realized the global optimal electricity trading strategy, and improved market transaction efficiency and resource allocation benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 A flowchart of a game method for joint trading of multiple types of power commodities in a virtual power plant;

[0052] Figure 2 A multi-layer interactive framework diagram for wholesale markets, virtual power plant aggregators, and multiple virtual power plants;

[0053] Figure 3 This is the solution framework diagram of the DDPG_AE algorithm. DETAILED DESCRIPTION

[0054] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0056] Embodiment 1

[0057] like Figure 1 As shown, the present invention provides a virtual power plant multi-type power commodity joint trading game method, the method comprising:

[0058] S1: Based on the principle of multi-level market price difference arbitrage, a multi-level interactive framework between VPPO and VPP for various types of electricity commodities is constructed;

[0059] Specifically, by analyzing the price differences between different power markets and identifying price difference opportunities in the spot market, power futures market, ancillary service market and other hierarchical markets, VPPO can arbitrage in these price differences to obtain economic benefits. The construction of a multi-layer interactive framework can ensure the decision-making coordination between VPPO and VPP, so that they can effectively transmit information and adjust strategies between different market levels to achieve optimal scheduling and trading of power resources. This framework provides a new way for VPPO to jointly trade different types of power commodities according to market conditions, thereby maximizing economic benefits.

[0060] S2: Based on the resource allocation and market transaction optimization of VPP and based on the multi-layer interaction framework, a single VPP decision model is constructed with the goal of minimizing cost and utility and maximizing market share;

[0061] Specifically, based on the resource allocation and market transaction optimization of VPP, with the support of a multi-layer interactive framework, a decision-making model is constructed for each individual VPP, with the goal of minimizing costs and utilities and maximizing market share. In this process, VPP needs to be reasonably configured and scheduled according to market demand and available resources, and optimize power trading strategies to reduce operating costs and improve resource utilization efficiency. At the same time, the decision-making model aims to help VPP improve its competitiveness in the power market and strive for a larger market share through precise resource scheduling and optimized market pricing strategies.

[0062] S3: Based on a single VPP decision model, a multi-virtual power plant operation optimization model based on different objectives is constructed, and a distributed solution is used to solve the multi-VPP multi-type power commodity sharing problem to obtain local market transaction results;

[0063] Specifically, by considering the resource allocation and market demand between different VPPs, a comprehensive optimization framework is constructed to achieve efficient operation of the entire virtual power plant system. The goals of the optimization model include cost minimization, utility maximization, and market share maximization. It aims to ensure the maximization of the overall benefits of the virtual power plant by reasonably adjusting the operation strategies of each VPP. In addition, due to the complexity of the interaction and resource sharing between multiple VPPs, a distributed solution method is adopted to efficiently handle the sharing problem of multiple VPPs and multiple types of electricity commodities. Each VPP performs local optimization calculations based on its own resources and market conditions, and jointly solves the final global optimization problem through a coordination mechanism to obtain the trading results of the local market.

[0064] S4: Based on the multi-VPP operation optimization results and local market transaction results, a VPPO decision model is constructed with the goal of minimizing costs;

[0065] Specifically, VPPO integrates the operation optimization results of multiple VPPs, and combines the trading conditions of each VPP in the local market to comprehensively evaluate the resource allocation and trading strategies of the entire virtual power plant. By analyzing these results, VPPO can optimize its resource scheduling and market strategies to ensure the best cost-effectiveness in the overall electricity market. VPPO's decision-making model will adjust the operating strategy based on this information to minimize resource scheduling costs, market transaction costs and other operating expenses, thereby improving the economic benefits of the virtual power plant.

[0066] S5: Based on the theory of peer-to-peer non-cooperative game, the transaction process of VPPO and VPP is designed as a two-layer model, and the data-driven method of DDPG_AE algorithm is used to solve the model based on the method of one party setting prices and the other party reporting quantities, so as to obtain the global optimal electricity trading strategy.

[0067] Specifically, based on the theory of peer-to-peer non-cooperative game, the transaction process between the virtual power plant operator (VPPO) and the virtual power plant unit (VPP) is designed as a two-layer game model, which describes the interaction and decision-making of both parties. This step adopts the game strategy of one party setting prices and the other party reporting quantities, and combines it with the data-driven method of the DDPG_AE algorithm (deep deterministic policy gradient algorithm combined with autoencoder) to solve it in order to achieve the global optimal electricity trading strategy. In this process, VPPO and VPP are regarded as the two parties of the game. VPPO is responsible for pricing, while VPP reports the trading volume according to the market price. Through the theory of reciprocal non-cooperative games, the model can simulate the optimal decision-making behavior of the two parties without cooperative constraints to ensure the maximization of the interests of both parties. The DDPG_AE algorithm is used to deal with the high-dimensional continuous space problems in this game model, and uses reinforcement learning to continuously adjust the decision-making strategy, and find the optimal electricity trading decision in a data-driven way; finally, based on this two-layer game model and data-driven method, a globally optimal electricity trading strategy can be obtained, so that VPPO and VPP can achieve the best economic benefits in the game process, thereby improving the operating efficiency and market competitiveness of the entire virtual power plant.

[0068] Through the present invention, the complexity problem of virtual power plants in the joint transaction of multiple types of electricity commodities is effectively solved, the global optimal electricity trading strategy is realized, and the market transaction efficiency and resource allocation benefits are improved.

[0069] As a preferred embodiment of the above, based on the principle of multi-level market price difference arbitrage, a multi-level interactive framework between VPPO and VPP of multiple types of electricity commodities is constructed, including:

[0070] S11: According to the hierarchical structure of the electricity market, determine the types of electricity commodities in different levels of the market;

[0071] S12: Based on the price fluctuations of multi-tier markets, potential arbitrage opportunities are identified by calculating the price differences between each market tier;

[0072] S13: VPPO allocates different types of electricity commodities among various market tiers by formulating market participation strategies and pricing mechanisms;

[0073] S14: Build an interactive framework between VPPO and VPP and establish a two-way information transmission mechanism. VPPO transmits power commodity trading signals to VPP through price signals and adjusts market strategies based on VPP feedback.

[0074] Specifically, virtual power plants (VPPs) are taken as the research object. These power plants are equipped with renewable energy sources such as distributed photovoltaics (PV), wind turbines (WT), and adjustable generators such as micro turbines (MT), and have a certain flexible load (FL) regulation potential. Due to capacity and policy restrictions, these VPPs cannot directly participate in wholesale market transactions. Therefore, virtual power plant aggregators (VPPOs) act as upper-level agents to participate in wholesale market transactions on behalf of these VPPs. In order to cope with the energy fluctuation risks brought about by market price fluctuations, VPPOs manage a certain capacity of energy storage systems (ESS).

[0075] In terms of market structure, VPPO does not directly control the resources of VPP, but raises demand-side resources through the local market, participates in transactions in the upper wholesale market, and uses the price difference of multi-level markets for arbitrage. Since VPPO provides VPP with a way to participate in the wholesale market, it has pricing power over VPP's electricity purchase and sales, frequency regulation, and backup. The interaction between VPPO and VPP can be summarized as a two-level optimization problem, such as Figure 2 shown.

[0076] exist Figure 2 In the figure, the upper layer is VPPO and the lower layer is VPP. Based on the forecast results of the wholesale market day-ahead prices, VPPO seeks the optimal purchase and sale prices under the existing VPP response potential. These prices are transmitted to the underlying VPPs as signals reflecting the balance of supply and demand in the system, guiding them to optimize their own power generation and consumption behaviors and reasonably allocate the proportion of transactions with VPPO in the local market to minimize the management costs of energy, frequency regulation and reserve.

[0077] As a preferred embodiment of the above, a single VPP decision model is:

[0078]

[0079] In the formula, C i is the total cost of VPP, is the MT power generation cost, is the electricity purchase cost of VPP and VPPO transactions, is the electricity purchase cost for P2P transactions between VPPs, The cost of purchasing spare parts for VPP i, The purchase and standby cost of VPP i for P2P transactions, Purchase frequency modulation cost for VPP, The purchase frequency cost of VPP i for P2P transactions, Energy efficiency for VPP i, Reserve spare cost for VPP i.

[0080] As a preferred embodiment of the above, in the initial stage of market development, VPP expands its market share, and a single VPP seizes the local market model is:

[0081]

[0082] Where U i VPP I is the objective function for seizing market share, is the price at which VPP i trades with VPPO during period t; The power purchase and sale energy between VPP i and VPPO in period t; are the upper reserve and lower reserve prices purchased by VPP i in transactions with VPPO during period t; are the upper reserve and lower reserve prices of VPP i in transactions with VPPO during period t; is the upper reserve and lower reserve quantity purchased by VPP i in transactions with VPPO during period t; The upper reserve and lower reserve quantities sold by VPP i in transactions with VPPO during period t; is the purchase and sale frequency modulation price of VPP i in the transaction with VPPO during period t; is the purchase and sale frequency modulation volume of VPP i and VPPO in period t.

[0083] Taking into account the equipment's adjustable resources, operating characteristics, costs and types of electricity purchased and sold, a single virtual power plant decision model is constructed with the goal of minimizing costs and benefits and maximizing market share.

[0084] 1) Transaction cost and utility model of a single VPP participating in the local market

[0085] ①MT power generation cost

[0086]

[0087] Where i∈I, I is the VPP set; t∈T, T is the time period set; is the active power output of MT of VPP i in period t; i , b i 、c i is the MT cost coefficient.

[0088] ②VPP i power purchase and sales costs

[0089] In order to meet the joint sharing needs of virtual power plant energy, frequency regulation and reserve under the premise of minimizing costs, this paper provides two trading channels for VPP: first, VPP can conduct peer-to-peer (P2P) transactions with other VPPs, and the transaction price is determined by the distribution network market clearing; second, VPP can trade with VPPO, and the transaction price is determined by VPPO. The income obtained by VPP from selling energy, frequency regulation or reserve to VPPO or other VPP is taken as the opposite number and included in the cost, then

[0090] The electricity purchase cost of VPP and VPPO transactions is:

[0091]

[0092] In the formula, The cost of electricity purchased from VPP and VPPO transactions; is the price at which VPP i trades with VPPO during period t; It is the electricity purchase and sale transaction between VPP i and VPPO during period t.

[0093] The cost of purchasing electricity for P2P transactions between VPPs is:

[0094]

[0095] In the formula, The cost of purchasing electricity for P2P transactions between VPPs; is the price of P2P energy trading between VPP i and VPP j during period t; is the amount of P2P energy transaction between VPP i and VPP j during period t.

[0096] ③VPP i purchase spare cost

[0097] VPP i can also conduct P2P standby transactions with the upper-layer VPPO or lower-layer VPP. The standby purchase cost of its transaction with VPPO is:

[0098]

[0099] In the formula, is the purchase and reserve cost of the transaction between VPP i and VPPO; are the upper reserve and lower reserve prices purchased by VPP i in transactions with VPPO during period t; are the upper reserve and lower reserve prices of VPP i in transactions with VPPO during period t; is the upper reserve and lower reserve quantity purchased by VPP i in transactions with VPPO during period t; The upper and lower reserve quantities sold by VPP i in transactions with VPPO during period t.

[0100] The purchase and standby cost of VPP i for P2P transactions is:

[0101]

[0102] In the formula, The purchase and standby cost of VPP i for P2P transactions; are the upper reserve and lower reserve prices of VPP i trading with VPP j in period t; It is the upper reserve and lower reserve amount of VPP i’s transaction with VPP j in period t.

[0103] ④VPP i purchase frequency modulation cost

[0104] VPP i can also conduct P2P standby transactions with the upper-layer VPPO or lower-layer VPP. The purchase and modulation cost of its transaction with VPPO is:

[0105]

[0106] In the formula, is the purchase and regulation cost of the transaction between VPP i and VPPO; is the purchase and sale frequency modulation price of VPP i in the transaction with VPPO during period t; is the purchase and sale frequency modulation volume of VPP i and VPPO in period t.

[0107] The purchase and modulation cost of VPP i for P2P transactions is:

[0108]

[0109] In the formula, The purchase frequency cost of P2P transaction for VPP i; is the frequency modulation price of VPP i trading with VPP j during period t; is the frequency modulation amount traded between VPP i and VPP j during period t.

[0110] ⑤VPP i energy efficiency

[0111] Considering that FL can provide a certain degree of flexibility by adjusting the load, this paper uses a quadratic function with increasing marginal utility to express the utility corresponding to the degree of dissatisfaction with VPP energy consumption experience. It is expressed as follows:

[0112]

[0113] In the formula, The utility corresponding to the degree of dissatisfaction of VPP users’ energy experience; is the utility coefficient; is the operating power of FL in period t of VPP i; are the upper and lower limits of the FL operating power of VPP i, respectively. The average of the two represents the most comfortable operating power set.

[0114] ⑥VPP i reserved standby cost

[0115] In order to solve the real-time power deviation problem caused by the uncertainty of distributed photovoltaic, wind turbines and loads, each VPP should reserve or purchase spare capacity to avoid the potential economic risks arising therefrom.

[0116] The reserved standby cost of VPP i is:

[0117]

[0118] In the formula, The reserve cost reserved for VPP i; are the unit costs of upper and lower backup provided by MT and FL of VPP i, respectively; are the upper and lower spare capacities that MT and FL of VPP i can provide during period t.

[0119] In summary, the decision model of each VPP is expressed as follows:

[0120]

[0121] In the formula, C i is the total cost of VPP i.

[0122] Each VPP needs to meet the following equipment operation constraints:

[0123]

[0124]

[0125] In the formula, are the active outputs of PV and WT of VPP i in period t, respectively; are the MT active output limit of VPP i and the maximum predicted output of PV and WT; A Boolean variable representing the trading identity of VPP i with VPPO in period t; Constraints (11) to (12) indicate that the reserve that can be provided by MT and FL should not exceed the surplus generation capacity of the corresponding equipment. Constraints (13) to (18) represent the operating characteristics constraints of DERs. Constraints (19) to (26) stipulate that VPP i can only participate in transactions in one identity in the same market and at the same time.

[0126] In addition, the VPP decision model should also meet the power balance constraints and P2P transaction constraints:

[0127]

[0128]

[0129] In the formula, is the predicted value of rigid load of VPP i in period t.

[0130] Therefore, the variable vector x of the VPP decision model is i It can be expressed as formula (32).

[0131]

[0132] 2) Single VPP seizes local market model

[0133] Some VPPs hope to expand their market share, especially in the early stages of market development.

[0134] The decision model of each VPP is expressed as follows:

[0135]

[0136] Where U i is the objective function of VPP i in order to seize market share.

[0137] The technical constraints are the same as above, as shown in (11)-(31).

[0138] Therefore, the variable vector x of the VPP decision model is i It can be expressed as formula (34).

[0139]

[0140] As a preferred embodiment of the above, the augmented Lagrangian function of the optimization problem is:

[0141]

[0142] in, In the formula, C i (x i,t ) is the i-th VPP in decision x i,t The total cost under x i,t is the variable vector of the VPP decision model; i,t =Y ij,t is an auxiliary variable, Y ij,t is the electric energy, upper reserve, lower reserve and frequency regulation amount traded between VPP i and VPP j; is the Lagrange multiplier corresponding to the P2P transaction constraint; ρ tis the iteration step length; ||·||2 is the 2-norm.

[0143] Specifically, a multi-virtual power plant operation optimization model based on different objectives is constructed, and a distributed solution is used to solve the multi-VPP multi-type electricity commodity sharing problem. The computing power burden is reduced through multi-center computing, and local market transaction results are obtained while protecting the privacy of VPPs.

[0144] 1) Establish an optimization model with the goal of minimizing the total cost and utility of multiple VPPs

[0145]

[0146] In the formula, C I is the total cost of multiple VPPs.

[0147] In addition, the optimization model should satisfy the following power balance constraints:

[0148]

[0149] 2) Establish an optimization model with the goal of quickly seizing market share with multiple VPPs

[0150]

[0151] Where U I is the total electricity purchase cost of multiple VPPs.

[0152] In the initial stage, VPPs select the optimization model shown in formula (39) in order to quickly seize market share. After stabilization, they select the optimization model shown in formula (35).

[0153] A distributed method is used to solve the multi-objective, multi-VPP, and multi-type electricity commodity sharing problem. The computing power burden is reduced through multi-center computing, and local market transaction results are obtained under the premise of protecting VPP privacy. It is known that among all the constraints, except for the P2P transaction constraint, which belongs to the global coupling constraint, the remaining constraints are only related to the decision variables of each VPP itself. Therefore, auxiliary variables are introduced to rewrite the coupling constraints. Let X ij,t ∈x i,t ,y i,t =Y ij,t ,have

[0154]

[0155] Where Y ij,t is the electric energy, upper reserve, lower reserve and frequency regulation traded between VPP i and VPP j.

[0156] From an economic perspective, the Lagrange multiplier corresponding to the P2P transaction constraints is represents the shadow price of P2P transactions, then the augmented Lagrangian function of the optimization problem is:

[0157]

[0158] in,

[0159]

[0160] In the formula, ρ t is the iteration step length; ||·||2 is the 2-norm.

[0161] Let k be the number of iterations, and the solution process is as follows.

[0162] ① Update local decision variables

[0163] VPP i is obtained according to the previous iteration Solve the local optimization decision problem (44) and update the Passed to VPP j.

[0164]

[0165] ②Update auxiliary variables

[0166] Solve the following global optimization problem to update the auxiliary variables:

[0167]

[0168] In order to realize decentralized transactions between VPPs while taking into account both solution speed and privacy protection, the coupling part in the above optimization problem (43) is separated to obtain Y ij,t Analytical solution of .

[0169]

[0170] VPP i can independently update its auxiliary variables without relying on a central platform for global optimization, thus achieving fully distributed solving.

[0171] ③ Update the Lagrange multiplier

[0172] The raw residual of P2P transaction volume r t (k) and the dual residual The calculation formula is as follows. In order to solve the problem of uneven residual convergence speed, the adaptive parameters δ and τ strategy is adopted to flexibly adjust the step size according to the residual state. When the original residual is large, the step size is increased to speed up the convergence speed and promote the consistency of the original variable and the auxiliary variable; when the original residual is small, the step size is reduced to reduce the oscillation of the objective function.

[0173]

[0174] Finally, VPP i independently updates the Lagrange multiplier and converts the price signal Passed to the corresponding VPP j, that is

[0175]

[0176] The iterative convergence criterion is as follows:

[0177]

[0178] Where ε1 is the set error accuracy.

[0179] As a preferred embodiment of the above, the VPPO decision model is:

[0180]

[0181] In the formula, is the ESS loss cost of VPPO, Reserve spare cost for VPPO, is the cost of electricity purchase and sale in the VPPO wholesale market, Purchase spare costs for the VPPO wholesale market, The cost of purchasing frequency modulation for the wholesale market, The revenue from electricity sales in the local market of VPPO, Reserve income for VPPO local market sales, Sell ​​FM revenue for the local market.

[0182] Specifically, a VPPO decision-making model is constructed with the goal of minimizing costs. The transaction process of VPPO and VPP is designed as a two-layer model based on the theory of peer-to-peer non-cooperative games. The model is solved using the data-driven method of the DDPG_AE algorithm based on the method in which one party sets prices and the other party reports quantities.

[0183] In order to maximize the interests of both parties, this paper designs the transaction process between VPPO and VPP as a two-layer model. VPPO has the right to decide the price of VPP's purchase and sale of energy, frequency regulation and reserve, but in order to avoid unreasonable pricing that leads to a reduction in transaction volume and affects total revenue, the transaction process uses the DDPG_AE algorithm to solve the model.

[0184] VPPO decision model:

[0185] During a trading cycle, the costs and benefits of VPPO's participation in wholesale and local market transactions can be expressed as follows.

[0186] ① ESS loss cost of VPPO

[0187]

[0188] In the formula, and are the charging and discharging power of ESS of VPPO in period t; η ESS is the loss coefficient of ESS.

[0189] ②VPPO reserved standby cost

[0190] Since part of the reserved ESS charging and discharging capacity cannot be used for electricity trading, the relevant costs are:

[0191]

[0192] In the formula, are the reserved charging and discharging capacities of ESS of VPPO in period t; ESS Unit cost of upper and lower reserve capacity provided for ESS.

[0193] ③ Cost of electricity purchase and sale in VPPO wholesale market

[0194] VPPO can participate in wholesale market transactions as an independent market participant. Assuming that VPPO is a price taker in the wholesale market, its electricity purchase cost in the wholesale market can be expressed as:

[0195]

[0196] In the formula, is the wholesale market electricity price forecast value of VPPO in period t; t VPPO-DSO is the required power of VPPO in period t.

[0197] ④VPPO wholesale market purchase cost

[0198]

[0199] In the formula, is the forecast value of the upper and lower reserve prices of VPPO in the wholesale market in period t; is the upper and lower reserve demand of VPPO in period t.

[0200] ⑤ VPPO wholesale market purchase frequency modulation cost

[0201]

[0202] ⑥ VPPO local market electricity sales revenue

[0203] As an intermediary connecting the wholesale market and the local market, VPPO can determine the transaction price in the local market and resell the energy purchased from the wholesale market. The revenue of VPPO from selling electricity in the local market can be expressed as:

[0204]

[0205] ⑦VPPO local market sale reserve income

[0206] Similarly, the profit from selling the reserve in the local market is:

[0207]

[0208] ⑧VPPO local market sales frequency revenue

[0209]

[0210] Based on the above analysis, the decision model of VPPO can be expressed as:

[0211]

[0212] In order to encourage VPP and VPPO to trade and balance the impact of trading volume and trading benefits on total revenue, VPP should meet the following constraints when formulating the order and sale price:

[0213]

[0214]

[0215] In the formula, w buy 、w sell They are purchase and sale pricing adjustment parameters respectively.

[0216] The variable vector U corresponding to the VPPO decision model is shown in formula (81):

[0217]

[0218] In the formula, SOC t is the charge state of ESS of VPPO in period t.

[0219] The VPPO decision model should also meet the following constraints:

[0220]

[0221] SOC min ≤SOC t ≤SOC max (91)

[0222]

[0223] In the formula, B ESS and They are ESS capacity and charge / discharge power limit of VPPO; SOC t , SOC min , SOC max They represent the charge state and limit value of ESS of VPPO in period t respectively; u ESS is a Boolean variable indicating the charge and discharge status of the ESS during period t. Constraints (82) to (84) represent the power and reserve balance of the VPPO, respectively. Constraints (85) to (88) indicate that the reserve that the ESS can provide should not exceed its surplus generation capacity. Constraints (89) to (92) represent the operating characteristic constraints of the ESS.

[0224] As a preferred embodiment of the above, VPPO uses reinforcement learning to continuously adjust the pricing strategy during the interaction process. The interaction process can be formalized as an MDP, which includes:

[0225] State space, used to describe the current situation of the environment. The state space is represented as a vector that reflects the current state of the system.

[0226] Action space, which represents all possible actions or strategies that the agent can choose. The action space is a vector that combines the scheduling strategy and the pricing strategy to guide the specific actions taken by the agent in the decision-making process;

[0227] The reward function is used to reflect the optimization goal and calculate the immediate benefit or cost of the agent after performing a certain action in a certain state;

[0228] The state transition probability describes the probability that the environment is affected and transitions to the next state after performing a certain action in a given state.

[0229] Specifically, the agent is defined as VPPO, and the environment is defined as the world observed by VPPO, including the upper power grid, each generator set and VPPs. Assuming that the environment can interact with VPPO multiple times, VPPO can use reinforcement learning to continuously adjust the pricing strategy during these interactions to obtain the optimal pricing solution. The interaction process can be formalized as a Markov decision process (MDP), where MDP is represented by a four-tuple (S, A, R, P), corresponding to the state space, action space, reward function and state transition probability respectively.

[0230] 1) State space: The state is used to describe the current situation of the environment in the MDP. The state space defined in this paper includes the current time stage, the current storage capacity of the energy storage device, the dispatch transaction volume of each VPPs, and the transaction price between the VPPO and the upper power grid. The state space can be expressed as an 18-dimensional vector, as shown in Equation (93).

[0231]

[0232] 2) Action space: The action space combines the scheduling strategy and pricing strategy of VPPO.

[0233] It can be expressed as a seventeen-dimensional vector, as shown in formula (94).

[0234]

[0235] 3) Reward function: The reward function should directly reflect the goal of the optimization problem. This paper defines the instantaneous reward function at time t as the negative of the cost of VPPO, as shown in formula (95):

[0236]

[0237] 4) State transition probability: The state transition probability describes the state s t Next, perform action a t After that, the environment is affected and VPPO transitions to the next state s t+1 =s' probability, as shown in formula (96):

[0238] P a (s,s')=P(s t+1 =s'|s t =s,a t =a) (96)

[0239] As a preferred embodiment of the above embodiment, the data-driven method of the DDPG_AE algorithm is used to solve the model and obtain the global optimal power trading strategy, including:

[0240] A10: Initialize the network structure and parameters of the DDPG_AE algorithm, including the parameters of the actor network and critic network, the experience replay pool, and the adaptive exploration coefficient;

[0241] A20: The agent obtains the current system state, selects appropriate trading actions based on the current strategy and Gaussian noise, executes actions based on market feedback, transfers to the next state and obtains corresponding rewards, and stores the quadruple in the experience replay pool;

[0242] A30: When the number of samples in the experience replay pool reaches the preset threshold, the agent randomly samples batches of samples from the replay pool, calculates the target value and temporal difference error of the critic network, and updates the parameters of the actor network and the critic network, using a soft update mechanism to update the target network parameters;

[0243] A40: The agent repeatedly performs sampling and training, gradually adjusts the exploration coefficient to optimize the strategy, and outputs the final actor network and critic network parameters when the stopping condition is met to obtain the globally optimal electricity trading strategy.

[0244] Specifically, the DDPG_AE algorithm solution framework is as follows Figure 3 As shown in Figure 2, the framework first constructs four neural networks, namely the actor network μ(s|θ), the critic network Actor target network μ(s|θ') and critic target network In the sampling phase, randomly select state s t Input to the actor network, the actor network according to the current state s t Adopt an adaptive action exploration mechanism to select action a t ; Then, the agent interacts with the environment according to the selected action and outputs the corresponding reward r t and the next state s t+1 , forming a four-tuple sample (s t ,a t ,r t ,s t+1 ) and stored in the experience replay pool D. This process is repeated until the number of samples in the experience replay pool reaches the predetermined size L, after which the algorithm enters the training phase. At the beginning of the training phase, first take the state s t A four-tuple sample is formed and stored in the experience replay pool D. If D has reached its maximum capacity, the first four-tuple sample in D is replaced and added to the end; then, M four-tuple samples (s k ,a k ,r k ,s k+1 ), which is input into the critic network to output the Q value function of the current state-action The Q-value function represents the expected discounted reward value that can be obtained by performing an action in a given state, as shown in formula (97); Next, the actor target network provides the next state s k+1 The estimated optimal action μ(s k+1 |θ'), the critic target network is responsible for μ(s k+1 |θ') estimates the Q-value function of the state-action at time k+1 Finally, the target value y of the state-action at the current k moment is calculated according to the Bellman equation k , as shown in formula (98).

[0245]

[0246] Where: γ represents the discount coefficient of the Bellman equation; r T Indicates the reward value corresponding to the terminal state.

[0247] When obtaining the target value y k After that, the four network parameters can be updated. In the critic network, the parameters are updated by gradient descent method. As shown in formula (99).

[0248]

[0249] Where: represents the temporal difference error of the sample; Represents the learning rate of the critic network.

[0250] The actor network is based on the current state s k Generate the optimal action μ(s k |θ), and input these actions into the critic network to generate a Q-value function and feed it back to the actor network. The actor network uses the gradient ascent method to update parameters, as shown in formula (100).

[0251]

[0252] Where: θ Represents the learning rate of the actor network.

[0253] In the parameter update module of the critic target network and the actor target network, a soft update mechanism is used to perform a weighted average of the current network parameters and the target network parameters to ensure that the parameters are updated slowly and improve the stability of training. The specific update formulas are shown in Equations (101) and (102).

[0254]

[0255] θ'←τθ+(1-τ)θ'(102)

[0256] Where: τ∈(0,1) represents the soft update factor.

[0257] In DDPG or other reinforcement learning algorithms, the action selection strategy is very important in optimization problems with continuous action spaces. It should ensure that the agent can explore the action space extensively. The action selection strategy in the original DDPG algorithm adopts the normal distribution sampling method. Specifically, the agent uses the current optimal action a'=μ(s|θ) output by the actor network as the mean of the normal distribution, and introduces the standard deviation parameter δ to construct the normal distribution. Subsequently, the agent randomly samples a new action a from the normal distribution, replaces the original action a' with the new action, and outputs it from the actor network. The implementation method is shown in Equation (103).

[0258] a=μ ε (s)=μ(s|θ)+ε, ε~N(0,σ) (103)

[0259] Where: ε represents the exploration noise, N(0,σ) is a normal distribution with mean 0 and variance σ.

[0260] In the system optimization process, the problem model is a continuous multi-step MDP. Even if the variance σ of the exploration noise ε is set to a small value, due to the effect of multi-step series connection, the variance of the resulting action (s, a) distribution may still be too large, resulting in the action selection process being biased towards exploration and lacking development, thereby affecting the convergence stability and overall effect. Therefore, this embodiment proposes an adaptive exploration action selection strategy, which introduces an adaptive exploration coefficient A before the noise in formula (103), as shown in formula (104).

[0261] a=μ Aε (s)=μ(s|θ)+A·ε, ε~N(0,σ) (104)

[0262] Where: The coefficient A combines the average loss value of the critic network in this training episode and the variance of the cumulative rewards of the previous n training episodes, as shown in Equations (105) and (106).

[0263]

[0264] Where: λ is the noise coefficient; var(·) represents the variance function; R episode-i represents the cumulative reward value generated by the interaction between the agent and the environment in the episode-i training generation, i = 1, 2, ..., n; r t episode-i represents the immediate reward at timestamp t in the episode-i training generation; It represents the critic network loss value calculated by the last sample at timestamp t of this training episode, t = 1, 2, ..., T. A is a decreasing function of the sum of the reward variance and the inverse of the critic network average loss value. The smaller the sum of the two, the larger the value of A.

[0265] In order to solve the dimensionality problem of the two terms in equation (105), we first need to normalize the cumulative reward value of the first n training generations:

[0266]

[0267] Where: R min and R max Represents the minimum and maximum cumulative rewards of the previous n training generations.

[0268] Through mathematical proof, the variance range of reward normalization is between [0, 0.25]. In order to make the loss value of the critic network also fall within this interval, we normalize the loss value of the critic network to [4, 100], as shown in formula (108).

[0269]

[0270] Where: and Indicates the minimum and maximum values ​​of the critic network loss in this training episode.

[0271] According to equations (104) and (105), when the agent is in the early stage of training, the average loss value of the critic network is large. At this time, if the change range of the reward is small (that is, the variance of the reward is small), the adaptive exploration coefficient A is increased to enhance the exploration ability of the action space. In the later stage of training, when the average loss value of the critic network is small, if the change range of the reward is large (that is, the variance of the reward is large), A is reduced to improve the utilization of the current optimal action.

[0272] As a preferred embodiment of the above, DDPG_AE has a two-layer loop structure, including:

[0273] The outer loop represents one iteration of the algorithm;

[0274] The inner loop controls the parameter training at each timestamp in each iteration.

[0275] Specifically, first, the network parameters, experience replay pool and adaptive exploration coefficient are initialized. Then, in the sampling phase, the agent obtains the initial observation state and selects the appropriate action based on Gaussian noise. After executing the action, the agent will transfer to the next state according to the state transition probability and obtain the reward, and store these four-tuple samples at the end of the experience replay pool D. If the number of samples in the experience replay pool D reaches the preset value, the sampling phase is switched to the training phase. The agent repeats the operation of the sampling phase. If the number of samples in D reaches the maximum capacity, the first sample in D is replaced and a new four-tuple sample is added. Next, a batch of samples is randomly sampled from D, and the target value, temporal difference (TD) error and each gradient of the critic network are calculated. The parameters of the critic network and the actor network are updated according to equations (99) and (100), and the soft update mechanism (such as equations (101) and (102)) is used to regularly update the critic target network and the actor target network. Finally, the adaptive exploration coefficient is calculated according to equation (105) and a new round of iteration is started. The iterative process continues until the stopping condition is met, and the final actor network parameters and critic network parameters are output.

[0276] Embodiment 2

[0277] Based on the same inventive concept as the virtual power plant multi-type power commodity joint transaction game method in the aforementioned embodiment, the present invention also provides a virtual power plant multi-type power commodity joint transaction game system, the system comprising:

[0278] The multi-layer interactive framework building module builds a multi-layer interactive framework between VPPO and VPP of various types of power commodities based on the principle of multi-level market price difference arbitrage;

[0279] A single decision model building module builds a single VPP decision model based on the resource allocation and market transaction optimization of VPP and based on a multi-layer interactive framework, with the goal of minimizing cost and utility and maximizing market share;

[0280] The market transaction result acquisition module builds a multi-virtual power plant operation optimization model based on a single VPP decision model with different objectives, and uses distributed solution to the multi-VPP multi-type power commodity sharing problem to obtain local market transaction results;

[0281] The VPPO decision model building module builds the VPPO decision model based on the multi-VPP operation optimization results and local market transaction results with the goal of minimizing costs;

[0282] The optimal trading strategy acquisition module designs the trading process of VPPO and VPP as a two-layer model based on the theory of peer-to-peer non-cooperative game, and solves the model using the data-driven method of the DDPG_AE algorithm based on the method of one party setting prices and the other party reporting quantities, to obtain the global optimal electricity trading strategy.

[0283] The above-mentioned game system in the present invention can effectively realize the game method for joint trading of multiple types of electricity commodities in a virtual power plant. The technical effects that can be achieved are as described in the above-mentioned embodiments and will not be repeated here. Although the present application has been described in conjunction with specific features and embodiments thereof, it is obvious that various modifications and combinations may be made thereto without departing from the spirit and scope of the present application. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present application as defined therein, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application is intended to include these modifications and variations.

Claims

1. A virtual power plant multi-type power commodity joint trading game method, characterized in that: include: Based on the principle of multi-level market price difference arbitrage, a multi-level interactive framework between VPPO and VPP for various types of electricity commodities is constructed; According to the VPP resource allocation and market transaction optimization and based on the multi-layer interactive framework, a single VPP decision model is constructed with the goal of minimizing cost and utility and maximizing market share; Based on the single VPP decision model, a multi-virtual power plant operation optimization model based on different objectives is constructed, and a distributed solution is used to solve the multi-VPP multi-type power commodity sharing problem to obtain local market transaction results; Based on the optimization results of multiple VPP operations and local market transaction results, a VPPO decision model is constructed with the goal of minimizing costs; Based on the theory of peer-to-peer non-cooperative game, the transaction process of VPPO and VPP is designed as a two-layer model. Based on the method of one party setting prices and the other party reporting quantities, the data-driven method of DDPG_AE algorithm is used to solve the model and obtain the globally optimal electricity trading strategy.

2. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: Based on the principle of multi-level market price difference arbitrage, a multi-level interactive framework between VPPO and VPP of various types of electricity commodities is constructed, including: According to the hierarchical structure of the electricity market, determine the types of electricity commodities in different levels of the market; Based on the price fluctuations of multi-tier markets, potential arbitrage opportunities can be identified by calculating the price differences between different market tiers; VPPO allocates different types of electricity commodities among various market tiers by formulating market participation strategies and pricing mechanisms; Build an interactive framework between VPPO and VPP, and establish a two-way information transmission mechanism. VPPO transmits electricity commodity trading signals to VPP through price signals, and adjusts market strategies based on VPP's feedback.

3. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: The single VPP decision model is: In the formula, C i is the total cost of VPP, is the MT power generation cost, is the electricity purchase cost of VPP and VPPO transactions, is the electricity purchase cost for P2P transactions between VPPs, The cost of purchasing spare parts for VPP i, The purchase and standby cost of VPP i for P2P transactions, Purchase frequency modulation cost for VPP, The purchase frequency cost of VPP i for P2P transactions, Energy efficiency for VPP i, Reserve spare cost for VPP i.

4. The virtual power plant multi-type electricity commodity joint trading game method according to claim 3 is characterized in that: In the early stage of market development, VPP expands its market share, and the model for a single VPP to seize the local market is: Where U i VPP I is the objective function for seizing market share, is the price at which VPP i trades with VPPO during period t; The power purchase and sale energy between VPP i and VPPO in period t; are the upper reserve and lower reserve prices purchased by VPP i in transactions with VPPO during period t; are the upper reserve and lower reserve prices of VPP i in transactions with VPPO during period t; is the upper reserve and lower reserve quantity purchased by VPP i in transactions with VPPO during period t; The upper reserve and lower reserve quantities sold by VPP i in transactions with VPPO during period t; is the purchase and sale frequency modulation price of VPP i in the transaction with VPPO during period t; is the purchase and sale frequency modulation volume of VPP i and VPPO in period t.

5. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: The augmented Lagrangian function of the optimization problem is: in, In the formula, C i (x i,t ) is the i-th VPP in decision x i,t The total cost under x i,t is the variable vector of the VPP decision model; i,t =Y ij,t is an auxiliary variable, Y ij,t is the electric energy, upper reserve, lower reserve and frequency regulation amount traded between VPP i and VPP j; is the Lagrange multiplier corresponding to the P2P transaction constraint; ρ t is the iteration step length; ||·||2 is the 2-norm.

6. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: The VPPO decision model is: In the formula, is the ESS loss cost of VPPO, Reserve spare cost for VPPO, is the cost of electricity purchase and sale in the VPPO wholesale market, Purchase spare costs for the VPPO wholesale market, The cost of purchasing frequency modulation for the wholesale market, The revenue from electricity sales in the local market of VPPO, Reserve income for VPPO local market sales, Sell ​​FM revenue for the local market.

7. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: VPPO uses reinforcement learning to continuously adjust pricing strategies during the interaction process, which can be formalized as an MDP, which includes: The state space is used to describe the current situation of the environment. The state space is represented as a vector reflecting the current state of the system. An action space, which represents all possible actions or strategies that the agent can choose. The action space is a vector that combines the scheduling strategy and the pricing strategy to guide the specific actions taken by the agent in the decision-making process; The reward function is used to reflect the optimization goal and calculate the immediate benefit or cost of the agent after performing a certain action in a certain state; The state transition probability describes the probability that in a given state, after performing a certain action, the environment is affected and transitions to the next state.

8. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: The data-driven method of the DDPG_AE algorithm is used to solve the model and obtain the global optimal power trading strategy, including: Initialize the network structure and parameters of the DDPG_AE algorithm, including the parameters of the actor network and critic network, the experience replay pool, and the adaptive exploration coefficient; The agent obtains the current system state, selects appropriate trading actions based on the current strategy and Gaussian noise, executes actions based on market feedback, transfers to the next state and obtains corresponding rewards, and stores the quadruple in the experience replay pool; When the number of samples in the experience replay pool reaches a preset threshold, the agent randomly samples batches of samples from the replay pool, calculates the target value and temporal difference error of the critic network, and updates the parameters of the actor network and the critic network, using a soft update mechanism to update the target network parameters; The agent repeatedly performs sampling and training, gradually adjusts the exploration coefficient to optimize the strategy, and outputs the final actor network and critic network parameters when the stopping condition is met to obtain the globally optimal power trading strategy.

9. The virtual power plant multi-type electricity commodity joint trading game method according to claim 1 is characterized in that: The DDPG_AE has a two-layer loop structure, including: The outer loop represents one iteration of the algorithm; The inner loop controls the parameter training at each timestamp in each iteration.

10. A virtual power plant multi-type power commodity joint trading game system, characterized in that: The system comprises: The multi-layer interactive framework building module builds a multi-layer interactive framework between VPPO and VPP of various types of power commodities based on the principle of multi-level market price difference arbitrage; A single decision model building module, which builds a single VPP decision model based on the resource allocation and market transaction optimization of the VPP and based on the multi-layer interactive framework, with the goal of minimizing cost and utility and maximizing market share; The market transaction result acquisition module constructs a multi-virtual power plant operation optimization model based on different objectives based on the single VPP decision model, and adopts distributed solution to the multi-VPP multi-type power commodity sharing problem to obtain the local market transaction results; The VPPO decision model building module builds the VPPO decision model based on the multi-VPP operation optimization results and local market transaction results with the goal of minimizing costs; The optimal trading strategy acquisition module designs the trading process of VPPO and VPP as a two-layer model based on the theory of peer-to-peer non-cooperative game, and solves the model using the data-driven method of the DDPG_AE algorithm based on the method of one party setting prices and the other party reporting quantities, to obtain the global optimal electricity trading strategy.

Images