A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning

Abstract

The present invention discloses a method for analyzing the equilibrium of multiple markets of electricity-carbon-green certificates based on multi-agent reinforcement learning, comprising the following steps: 1) establishing a double-layer optimization model of the electricity-carbon market and a double-layer optimization model of the electricity-green certificate market; 2) establishing an equilibrium model of the electricity-carbon-green certificate market based on the double-layer optimization model of the electricity-carbon market and the double-layer optimization model of the electricity-green certificate market. 3) Taking the power generation mechanism in the electricity-carbon-green certificate market as an intelligent agent, the equilibrium model of the electricity-carbon-green certificate market is solved by using an equilibrium solution algorithm to obtain a quotation scheme of the power generation mechanism that makes the electricity market balanced. The present invention can effectively analyze the coupling influence of the carbon market and the green certificate market on the equilibrium of the electricity market.

Description

Technical Field

[0001] The present invention relates to the field of power systems and automation thereof, and specifically to an electricity-carbon-green certificate multi-market equilibrium analysis method based on multi-agent reinforcement learning. Background Art

[0002] The design of the power market mechanism plays an important role in promoting the construction of the power market with carbon trading market participation and accelerating the construction of a high-proportion clean energy power system with green electricity as the main body. It is a key way to build a new power system and promote the "dual carbon" strategic goal. It is necessary to steadily promote the construction of the national carbon market in stages, and on the basis of ensuring the smooth and effective operation of the carbon market, continuously improve the carbon trading system, and effectively play the role of the market mechanism in reducing the carbon emission reduction costs of the whole society. At the same time, the National Energy Administration proposed to pilot green certificate trading nationwide.

[0003] The electricity market, carbon market and green certificate market have a profound impact on the decision-making of market players through the role of market mechanisms, and have interactive effects through price linkage and supply and demand relationships. The three markets are closely linked in terms of core product attributes, policies, technologies, and market positioning. Effective coordination at the mechanism level will help to form a joint force and jointly promote the development of clean energy and carbon emission reduction in the industry and society. Therefore, the study of coupling analysis between the electricity market, carbon market, green certificate market and other markets is of great significance to the construction of my country's electricity market.

[0004] The commonly used method for power market equilibrium analysis is a two-layer equilibrium model of the power market, which is composed of a bidding decision model with the goal of maximizing the profits of market entities and a market clearing model with the goal of maximizing social welfare. The solution method of this model is mainly to first transform the two-layer equilibrium model into an equilibrium optimization model with equilibrium constraints (Equilibrium Problem with Equilibrium Constraints, EPEC) based on the Karush-Kuhn-Tucher conditions (KKT conditions) of the lower model and linearization means, and then solve it after linearizing the nonlinear terms in the model. In addition, the diagonalization algorithm can also be used to solve the above EPEC model. The above model-driven method has achieved remarkable results in solving the equilibrium solution of the linear model in the power spot market.

[0005] However, in the context of new power system construction, the existing equilibrium analysis models and solution methods are difficult to meet the higher requirements of power market mechanism design. The existing equilibrium analysis methods, especially the EPEC method, are limited by the KKT conditions. The modeling process of the existing equilibrium analysis methods, especially the EPEC method, is difficult to consider the non-convexity of the cost and operating characteristics of the generator set, and the number of complementary relaxation conditions generated during the linearization of the model increases rapidly with the increase of the system scale and the complexity of the mechanism, which leads to huge challenges in solving the market equilibrium problems under large-scale systems and complex mechanisms. At the same time, the above model is essentially a complete information game problem. Each market player knows all the information about other players and market clearing, which is inconsistent with the limited information environment of the actual power market. Therefore, it is urgent to improve and perfect the equilibrium analysis method to effectively solve the complex mechanism design problems faced by the construction of the power market. Summary of the invention

[0006] The purpose of the present invention is to provide a method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, comprising the following steps:

[0007] 1) Establish a two-tier optimization model for the electricity-carbon market and a two-tier optimization model for the electricity-green certificate market;

[0008] 2) Based on the electricity-carbon market two-tier optimization model and the electricity-green certificate market two-tier optimization model, establish an electricity-carbon-green certificate market equilibrium model.

[0009] 3) Taking the power generation organizations in the electricity-carbon-green certificate market as intelligent entities, the equilibrium model of the electricity-carbon-green certificate market is solved using the equilibrium solution algorithm to obtain the quotation plan of the power generation organizations that balance the electricity market.

[0010] Furthermore, the electricity-carbon market two-layer optimization model includes an electricity-carbon market renewable energy power generation optimization model, an electricity-carbon market conventional energy power generation optimization model, and an electricity-carbon market day-ahead power market clearing module;

[0011] The electricity-carbon market renewable energy generation optimization model is as follows:

[0012]

[0013] st

[0014]

[0015] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model; is the set of renewable energy units at node n; is the output of the power generation mechanism g; a g and b g are the slope and intercept of marginal cost respectively; is the upper intercept limit; Provide quotations for power generation organizations;

[0016] The conventional energy generation optimization model of the electricity-carbon market is as follows:

[0017]

[0018] st

[0019]

[0020] In the formula, p CET is the unit carbon emission cost in the carbon emission rights market; f For free carbon emission quotas; is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n;

[0021] Among them, carbon emissions e g As shown below:

[0022]

[0023] In the formula, is the carbon emission intensity of the unit;

[0024] The day-ahead electricity market clearing model of the electricity-carbon market is as follows:

[0025]

[0026]

[0027]

[0028] -S n,m ≤B n,m (θ n -θ m )≤S n,m (9)

[0029] Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, Sn,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output. is the set of power generation mechanisms at node n.

[0030] Furthermore, the two-layer optimization model of the electricity-green certificate market includes a renewable energy power generation optimization model of the electricity-green certificate market, a conventional energy power generation optimization model of the electricity-green certificate market, and a day-ahead power market clearing model of the electricity-green certificate market;

[0031] The optimization model of renewable energy power generation in the electricity-green certificate market is as follows:

[0032]

[0033] st

[0034]

[0035] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n; p TGC is the green certificate price;

[0036] Proceeds from selling green certificates As shown below:

[0037]

[0038] The optimization model of conventional energy power generation in the electricity-green certificate market is as follows:

[0039]

[0040] st

[0041]

[0042] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n;

[0043] The day-ahead electricity market clearing of the electricity-green certificate market is as follows:

[0044]

[0045]

[0046]

[0047] -S n,m ≤B n,m (θn -θ m )≤S n,m (18)

[0048] Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output.

[0049] Furthermore, the two-layer optimization model of the electricity-carbon-green certificate market includes a conventional energy power generation optimization model of the electricity-carbon-green certificate market, a renewable energy power generation optimization model of the electricity-carbon-green certificate market, and a day-ahead power market clearing model of the electricity-carbon-green certificate market;

[0050] The optimization model of conventional energy power generation in the electricity-carbon-green certificate market is as follows:

[0051]

[0052] st

[0053]

[0054] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n;

[0055] The optimization model of renewable energy power generation in the electricity-carbon-green certificate market is as follows:

[0056]

[0057] st

[0058]

[0059] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n; p TGC is the green certificate price;

[0060] The day-ahead electricity market clearing of the electricity-carbon-green certificate market is as follows:

[0061]

[0062]

[0063]

[0064] -S n,m ≤B n,m (θ n -θ m )≤S n,m (26)

[0065] Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output.

[0066] Furthermore, the steps of solving the electricity-carbon-green certificate market equilibrium model using the equilibrium solution algorithm include:

[0067] 3.1) Set the number of training times N T , value network learning rate lr c , policy network learning rate lr a , discount rate γ, value network training frequency f, soft update frequency;

[0068] 3.2) Construct a policy network and a value network, and train them;

[0069] 3.3) Initialize the strategy network parameters θ and value network parameters ω of each agent a , value network parameter ω b ;

[0070] Initialize the target policy network θ of each agent targ and target value network parameter ω targ,a 、Target value networkω targ,b ;

[0071] 3.4) The agent observes the current electricity market clearing process and obtains the agent's local observation value that reflects the market clearing result g'=1,2,…,m;m is the number of agents;t is the number of iterations;

[0072] 3.5) The policy network generates actions that represent the agent’s offer The agent reports the price to the market environment;

[0073] 3.6) The central controller collects state observations of all agents Power market clearing will reward Feedback to each agent; store the state transfer matrix quadruple (s t ,a t ,r t ,s t+1 ); is the reward fed back to agent g'; s t+1 、s t is the state observation of the tth and t+1th iterations; a t for action;

[0074] 3.7) If tmodf = 0, a mini-batch (s) is randomly sampled from the experience replay pool t ,a t ,r t ,s t+1 ), and calculate the target policy network output and the target value network output Update value network parameters Value Network Parameters

[0075] If tmod(f×f a )=0, then update the strategy network parameter θ g ' , soft update target value network parameters Target value network parameters The soft update is to perform a weighted average of the current network parameters and the network parameters of the previous iteration;

[0076] 3.8) Repeat steps 3.4) to 3.7) until a power generation agency quotation plan that balances the power market is obtained.

[0077] Furthermore, the input of the policy network is the local observation of the agent that reflects the market clearing result. The output is the action that represents the agent's offer

[0078] Furthermore, the input of the value network is the local observations of all agents, that is, the global state s = [o 1 ,o 2 ,...,o m ], the output is the evaluation of the agent's action quality q(s,a;ω i ); a is the action; ω i is the value network parameter.

[0079] Furthermore, the policy network and the value network are connected by taking out the quadruple (s t ,a t ,r t ,s t+1 )train;

[0080] Tuple(s t ,a t ,r t ,s t+1 ) is as follows:

[0081]

[0082]

[0083] r t =[r t 1 ,r t 2 ,,,r t m ] (28)

[0084]

[0085] Further, the training strategy network μ(o i θ i ) aims to improve the value network’s evaluation of the agent’s actions q(s,a; ω i );

[0086] Policy network μ(o i θ i ) is as follows:

[0087]

[0088]

[0089]

[0090] In the formula, θ i is the decision variable; J i (θ 1 ,...,θm ) is the gradient vector; E S is the expectation of the policy network parameter q for the state s; β is the learning rate of the policy network parameter; To update the parameters.

[0091] Furthermore, the goal of training the value network is to make the value network evaluate the quality of the agent's actions q(s,a;ω i ) is closer to the action-value function

[0092] The training process of the value network is as follows:

[0093]

[0094]

[0095]

[0096] In the formula, represents the estimate of the value network output Q; Represents the error between the value network output Q and the estimated value; α represents the learning rate of the value network parameters; represents the reward; γ represents the discount rate.

[0097] The technical effect of the present invention is undoubted, and the beneficial effects of the present invention are as follows:

[0098] 1) The electricity-carbon dual-layer optimization model established in the present invention includes three parts: renewable energy profit maximization model, conventional energy power generator profit maximization model and day-ahead electricity market clearing model, which is conducive to analyzing the impact of carbon emission quotas and carbon prices on market equilibrium points;

[0099] 2) The two-layer optimization model of the electricity-green certificate market established by the present invention includes three parts: renewable energy profit maximization model, conventional energy power generator profit maximization model and day-ahead electricity market clearing model, which is conducive to analyzing the impact of green certificate prices on market equilibrium points;

[0100] 3) The two-layer optimization model of the electricity-carbon-green certificate market established in the present invention is an important basis for multi-market equilibrium analysis. It includes three parts: renewable energy profit maximization model, conventional energy generator profit maximization model and day-ahead electricity market clearing model, which is conducive to analyzing the coupling effect of carbon market and green electricity market on electricity market;

[0101] 4) The method for solving the electricity market equilibrium based on multi-agent deep reinforcement learning proposed in the present invention has shown good performance in solving the multi-agent system problem in the electricity market. In the agent system, multiple agents are in the same environment. They interact with the environment independently and use the feedback rewards from the environment to update their own strategies, and finally effectively obtain the market equilibrium solution.

[0102] The present invention can be widely applied to the multi-market equilibrium research of the power system and can effectively analyze the coupling influence of multiple markets. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 Provide a dual-tier optimization model for the electricity-carbon-green certificate market;

[0104] Figure 2 For the relationship between strategic network and value network;

[0105] Figure 3 The relationship between the six neural networks in the MATD3 algorithm;

[0106] Figure 4 The algorithm structure for solving the electricity market equilibrium based on MATD3;

[0107] Figure 5 It is a diagram of the algorithm training process;

[0108] Figure 6 Clearing electricity prices for the market under different carbon quotas and carbon prices;

[0109] Figure 7 Provide quotations for renewable energy power generators at different green certificate prices;

[0110] Figure 8 Clearing electricity prices for the market under different carbon prices and green certificate prices;

[0111] Fig. 9 Provide quotes for renewable energy units at different carbon prices and green certificate prices;

[0112] Fig.10 Provide quotes for low-emission units at different carbon prices and green certificate prices;

[0113] Fig.11 Provide quotes for high emission units at different carbon prices and green certificate prices. DETAILED DESCRIPTION

[0114] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0115] Embodiment 1:

[0116] See also Figures 1 to 11 , a method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, comprising the following steps:

[0117] 1) Establish a two-tier optimization model for the electricity-carbon market and a two-tier optimization model for the electricity-green certificate market;

[0118] 2) Based on the electricity-carbon market two-tier optimization model and the electricity-green certificate market two-tier optimization model, establish an electricity-carbon-green certificate market equilibrium model.

[0119] 3) Taking the power generation organizations in the electricity-carbon-green certificate market as intelligent entities, the equilibrium model of the electricity-carbon-green certificate market is solved using the equilibrium solution algorithm to obtain the quotation plan of the power generation organizations that balance the electricity market.

[0120] Embodiment 2:

[0121] A method for equilibrium analysis of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, the technical content of which is the same as that of Example 1, and further, the two-layer optimization model of the electricity-carbon market includes an optimization model of renewable energy power generation in the electricity-carbon market, an optimization model of conventional energy power generation in the electricity-carbon market, and a day-ahead electricity market clearing module in the electricity-carbon market;

[0122] The electricity-carbon market renewable energy generation optimization model is as follows:

[0123]

[0124] st

[0125]

[0126] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model; is the set of renewable energy units at node n; is the output of the power generation mechanism g; a g and b g are the slope and intercept of marginal cost respectively; is the upper intercept limit; Provide quotations for power generation organizations;

[0127] The conventional energy generation optimization model of the electricity-carbon market is as follows:

[0128]

[0129] st

[0130]

[0131] In the formula, p CET is the unit carbon emission cost in the carbon emission rights market; f For free carbon emission quotas; is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n;

[0132] Among them, carbon emissions e g As shown below:

[0133]

[0134] In the formula, is the carbon emission intensity of the unit;

[0135] The day-ahead electricity market clearing model of the electricity-carbon market is as follows:

[0136]

[0137]

[0138]

[0139] -S n,m ≤B n,m (θ n -θ m )≤S n,m (9)

[0140] Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output. is the set of power generation mechanisms at node n.

[0141] Embodiment 3:

[0142] A method for analyzing the equilibrium of multiple electricity-carbon-green certificate markets based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-2, and further, the two-layer optimization model of the electricity-green certificate market includes a renewable energy power generation optimization model of the electricity-green certificate market, a conventional energy power generation optimization model of the electricity-green certificate market, and a day-ahead electricity market clearing model of the electricity-green certificate market;

[0143] The optimization model of renewable energy power generation in the electricity-green certificate market is as follows:

[0144]

[0145] st

[0146]

[0147] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n; p TGC The price of green certificate.

[0148] Proceeds from selling green certificates As shown below:

[0149]

[0150] The optimization model of conventional energy power generation in the electricity-green certificate market is as follows:

[0151]

[0152] st

[0153]

[0154] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n;

[0155] The day-ahead electricity market clearing of the electricity-green certificate market is as follows:

[0156]

[0157]

[0158]

[0159] -S n,m ≤B n,m (θ n -θ m )≤S n,m (18)

[0160] Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output.

[0161] Embodiment 4:

[0162] A method for analyzing the equilibrium of multiple electricity-carbon-green certificate markets based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-3, and further, the two-layer optimization model of the electricity-carbon-green certificate market includes a conventional energy power generation optimization model of the electricity-carbon-green certificate market, a renewable energy power generation optimization model of the electricity-carbon-green certificate market, and a day-ahead electricity market clearing model of the electricity-carbon-green certificate market;

[0163] The optimization model of conventional energy power generation in the electricity-carbon-green certificate market is as follows:

[0164]

[0165] st

[0166]

[0167] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n;

[0168] The optimization model of renewable energy power generation in the electricity-carbon-green certificate market is as follows:

[0169]

[0170] st

[0171]

[0172] In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n; p TGC is the green certificate price;

[0173] The day-ahead electricity market clearing of the electricity-carbon-green certificate market is as follows:

[0174]

[0175]

[0176]

[0177] -S n,m ≤B n,m (θ n -θ m )≤S n,m (26)

[0178] Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output.

[0179] Embodiment 5:

[0180] A method for analyzing the equilibrium of multiple electricity-carbon-green certificate markets based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-4, and further, the step of solving the equilibrium model of the electricity-carbon-green certificate market by using the equilibrium solving algorithm includes:

[0181] 1) Set the number of training times N T , value network learning rate lr c , policy network learning rate lr a , discount rate γ, value network training frequency f, soft update frequency;

[0182] 2) Construct a policy network and a value network, and train them;

[0183] 3) Initialize the strategy network parameters θ and value network parameters ω of each agent a , value network parameter ω b ;

[0184] Initialize the target policy network θ of each agenttarg and target value network parameter ω targ,a 、Target value networkω targ,b ;

[0185] 4) The agent observes the current electricity market clearing process and obtains the agent's local observation value that reflects the market clearing result. g'=1,2,…,m;m is the number of agents;t is the number of iterations;

[0186] 5) The policy network generates actions that represent the agent’s offer The agent reports the price to the market environment;

[0187] 6) The central controller collects state observations of all agents Power market clearing will reward Feedback to each agent; store the state transfer matrix quadruple (s t ,a t ,r t ,s t+1 ); is the reward fed back to agent g'; s t+1 、s t is the state observation of the tth and t+1th iterations; a t for action;

[0188] 7) If tmodf = 0, a mini-batch (s) is randomly sampled from the experience replay pool t ,a t ,r t ,s t+1 ), and calculate the target policy network output and the target value network output Update value network parameters Value Network Parameters

[0189] If t mod(f×f a )=0, then update the strategy network parameter θ g' , soft update target value network parameters Target value network parameters The soft update is to perform a weighted average of the current network parameters and the network parameters of the previous iteration; a is the action;

[0190] 8) Repeat steps 4) to 7) until a power generation agency quotation plan that balances the power market is obtained.

[0191] Embodiment 6:

[0192] A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, the technical content of which is the same as any one of Examples 1-5, and further, the input of the strategy network is the local observation value of the agent reflecting the market clearing result The output is the action that represents the agent's offer

[0193] Embodiment 7:

[0194] A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-6, and further, the input of the value network is the local observation value of all agents, that is, the global state s=[o 1 ,o 2 ,...,o m ], the output is the evaluation of the agent's action quality q(s,a;ω i ); a is the action; ω i is the value network parameter.

[0195] Embodiment 8:

[0196] A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-7, and further, the strategy network and the value network are obtained by extracting a quadruple (s t ,a t ,r t ,s t+1 )train;

[0197] Tuple(s t ,a t ,r t ,s t+1 ) is as follows:

[0198]

[0199]

[0200] r t =[r t 1 ,r t 2 ,,,r t m ] (28)

[0201]

[0202] Embodiment 9:

[0203] A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-8, and further, the training strategy network μ(o i θ i ) aims to improve the value network’s evaluation of the agent’s actions q(s,a; ω i );

[0204] Policy network μ(o i θ i ) is as follows:

[0205]

[0206]

[0207]

[0208] In the formula, θ i is the decision variable; J i (θ 1 ,...,θ m ) is the gradient vector; E S is the expectation of the policy network parameter q for the state s; β is the learning rate of the policy network parameter; To update the parameters.

[0209] Embodiment 10:

[0210] A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, the technical content of which is the same as any one of Embodiments 1-9, and further, the goal of training the value network is to make the value network evaluate the quality of the agent's actions q(s,a;ω i ) is closer to the action-value function

[0211] The training process of the value network is as follows:

[0212]

[0213]

[0214]

[0215] In the formula, represents the estimate of the value network output Q; Represents the error between the value network output Q and the estimated value; α represents the learning rate of the value network parameters; represents the reward; γ represents the discount rate.

[0216] Embodiment 11:

[0217] A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning is as follows:

[0218] First, establish a two-layer optimization model of the electricity-carbon market, a two-layer optimization model of the electricity-green certificate market, and an equilibrium model of the electricity-carbon-green certificate market; then construct an equilibrium solution algorithm based on MATD3, set up a policy network, update network parameters according to environmental feedback reward information, and then optimize the agent strategy, set up a value network to judge the quality of the policy network output action, and help the policy network to update in the right direction; finally, through the simulation of the solution algorithm program, analyze the coupling effect of the participation of the carbon market and the green certificate market on the equilibrium point of the power system. The simulation results of the example on the modified IEEE 30-node system show that the electricity-carbon-green certificate multi-market equilibrium research method based on multi-agent reinforcement learning proposed by the present invention is a Nash equilibrium with high accuracy, which can effectively analyze the equilibrium of the electricity-carbon-green certificate market.

[0219] The specific steps are as follows:

[0220] 1) Establish a two-level optimization model for the electricity-carbon market

[0221] The participants in the carbon trading market include only conventional energy power generators, while renewable energy power generators do not participate. Renewable energy power generators only participate in the day-ahead electricity market to obtain income, while conventional energy power generators participate in both the day-ahead electricity market and the carbon market to obtain income.

[0222] Upper model 1: Profit maximization of renewable energy generators.

[0223] The profit maximization model for renewable energy generators is as follows:

[0224]

[0225] st

[0226]

[0227] in, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n. The present invention assumes that a power generator has only one unit. is the output of generator g. g and b g are the slope and intercept of marginal cost respectively. When making market quotations on the day-ahead market, power generators can increase their own profits by strategically reporting the slope and intercept of marginal cost. is the upper limit of the price reported by the generator. The objective function (1) is to maximize the profit of the renewable energy generator (the profit of the day-ahead market clearing minus its power generation cost).

[0228] Upper model 2: Profit maximization of conventional energy power generators.

[0229] Conventional energy units generate carbon dioxide emissions during power generation, and their carbon emissions are calculated by formula (3), where is the carbon emission intensity of the unit. The carbon emission cost of the unit is calculated by formula (4), where p CET is the carbon price in the carbon emission rights market, e f If the carbon emissions of the unit are greater than the free carbon quota, that is, the carbon emission cost is positive, the power generator needs to purchase the required carbon emission rights from the carbon emission rights market; if the carbon emissions of the unit are less than the free carbon quota, that is, the carbon emission cost is negative, the power generator can sell the remaining carbon emission rights in the carbon emission rights market.

[0230]

[0231]

[0232] The profit maximization model of conventional energy power generators is as follows:

[0233]

[0234] st (2)

[0236] in, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n. The objective function (5) is to maximize the profit of conventional energy generators (the revenue from market clearing minus their power generation costs and carbon emission costs), and constraint (2) limits the bidding range of generators.

[0237] Lower model: day-ahead electricity market clearing.

[0238]

[0239]

[0240]

[0241] -S n,m ≤B n,m (θ n -θ m )≤S n,m (9)

[0242] Where d is the load number, n and m are the node numbers, is the load set on node n, Ω nis the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch. The objective function (6) is to minimize the total power generation cost, also known as maximizing social welfare; constraint (7) is the node power balance constraint; constraint (8) is the upper and lower limit constraint of the unit output; constraint (9) is the branch flow constraint.

[0243] 2) Establish a two-tier optimization model for the electricity-green certificate market

[0244] The participants in the green certificate trading market include only renewable energy power generators, and conventional energy power generators do not participate in it. Conventional energy power generators only participate in the power day-ahead market to obtain income, while renewable energy power generators participate in both the power day-ahead market and the green certificate market to obtain income.

[0245] Upper model 1: Profit maximization of renewable energy generators.

[0246] After clearing in the electricity market, the same number of green certificates are issued to renewable energy generators based on their power generation. Renewable energy generators can sell them in the green certificate market. The income from selling green certificates is calculated by formula (10), where p TGC The price of green certificate.

[0247]

[0248] The profit maximization model for renewable energy generators is as follows:

[0249]

[0250] st (2)

[0252] in, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n. The objective function (11) is to maximize the profit of renewable energy generators (the income from clearing the day-ahead market plus the income from trading in the green certificate market minus its power generation cost), and constraint (2) limits the bidding range of generators.

[0253] Upper model 2: Profit maximization of conventional energy power generators.

[0254]

[0255] st (2)

[0257] in, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n. The objective function (12) is to maximize the profit of conventional energy generators (the revenue from day-ahead market clearing minus their generation costs), and constraint (2) limits the bidding range of generators.

[0258] Lower model: day-ahead electricity market clearing.

[0259] (6)-(9)

[0260] The lower model is the same as the lower model in the electricity-carbon market two-level optimization model.

[0261] 3) Establish a two-tier optimization model for the electricity-carbon-green certificate market

[0262] The upper layer of the two-layer optimization model of the electricity-carbon-green certificate market is the profit maximization model for power generators. The bidding strategy of power generators in the day-ahead market is solved and passed to the lower layer model; the lower layer is the day-ahead market, carbon market, and green certificate market clearing model. The market clearing results such as the node electricity price in the day-ahead market and the cleared electricity volume of each power generator, the number of green certificates issued by renewable energy power generators, and the carbon emissions of conventional energy power generators are solved and passed to the upper layer model.

[0263] The structure of the two-tier optimization model of the electricity-carbon-green certificate market is as follows: Figure 1 As shown, its components are as follows:

[0264] Upper model 1: Decision-making of conventional energy generators.

[0265] The profit maximization model of conventional energy power generators is as follows:

[0266]

[0267] st (1)

[0269] in, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n. The objective function (13) is to maximize the profit of renewable energy generators (the income from clearing the day-ahead market plus the income from trading in the green certificate market minus its power generation cost), and constraint (2) limits the quotation range of generators.

[0270] Upper model 2: Renewable energy generator decision making.

[0271] The profit maximization model for renewable energy generators is as follows:

[0272]

[0273] st (2)

[0275] in, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n. The objective function (14) is to maximize the profit of renewable energy generators (the income from clearing the day-ahead market plus the income from trading in the green certificate market minus its power generation cost), and constraint (2) limits the quotation range of generators.

[0276] Lower model: day-ahead electricity market clearing.

[0277] (6)-(9)

[0278] The lower model is the same as the lower model in the electricity-carbon market two-level optimization model.

[0279] 4) Use the multi-agent double-delay deep deterministic policy gradient algorithm to solve the above model:

[0280] In the problem of electricity market equilibrium, each market participant is modeled as an intelligent agent. Therefore, there are multiple agents in the system. The behaviors of each agent will affect each other and jointly promote the evolution of the entire system. In order to solve the problem of multi-agent system evolution in the electricity market, the multi-agent reinforcement learning (Multi-agent Reinforcement Learning) method is widely used. The Multi-agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm is a type of multi-agent reinforcement learning method. In the intelligent agent system, multiple agents are in the same environment. They interact with the environment independently and use the feedback rewards from the environment to update their own strategies. The MATD3 algorithm has shown good performance in solving the problem of multi-agent system in the electricity market. The MATD3 algorithm includes a policy network and a value network. Among them, the input of the policy network is the observation of the environment by the agent. i , output action a i =μ(o i θ i ) controls the behavior of the agent. The input of the value network is the observation of all agents, that is, the global state s = [o 1 ,o 2 ,...,o m ], output q(s,a;ωi ) is used to evaluate the quality of the agent's actions and can guide the policy network to improve its strategy. The relationship between the policy network and the value network is as follows: Figure 2 MATD3 trains the policy network and the value network separately. The training data is a quadruple (s t ,a t ,r t ,s t+1 ), as shown in (15)-(18).

[0281]

[0282]

[0283] r t =[r t 1 ,r t 2 ,,,r t m ] (17)

[0284]

[0285] Training policy network μ(o i θ i ) aims to improve the value network’s score q(s,a;ω) on the agent. i ), o i is the agent’s observation of the environment, θ i is the decision variable, as shown in (19). Monte Carlo approximation is performed on the target of the policy network and the gradient is calculated, and then gradient ascent is performed to update the parameter θ i , as shown in (20)-(21).

[0286]

[0287]

[0288]

[0289] The goal of training the value network is to make it score the agent's actions q(s,a;ω i ) is closer to the action-value function Extract a quadruple (s) from the experience replay pool t ,a t ,r t ,s t+1 ), use the strategy network of each agent to output the actions of all agents at the next moment Then calculate the target and error, as shown in (22)-(23), and finally perform gradient descent to update the parameter ωi , as shown in (24), making the scoring of the value network closer to the target.

[0290]

[0291]

[0292]

[0293] The relationship between the six neural networks in the MATD3 algorithm is as follows Figure 3 .

[0294] MATD3 algorithm element settings in the power market environment:

[0295] The present invention uses the MATD3 algorithm to solve the electricity market equilibrium, and the reinforcement learning elements in the electricity market environment are set as follows:

[0296] 1) Agent: All power generators g participating in the electricity market are set as agents. It is assumed that there are m agents in the system.

[0297] 2) Environment: The environment is defined as the clearing process of the power market. After the power generators make their bids, the power market is cleared and the clearing results are fed back to the power generators.

[0298] 3) Observation: Define the market clearing results fed back to the agent by the environment as the local observation of the agent

[0299] 4) State: Define the state is the set of local observations of all agents.

[0300] 5) Action: Define the agent’s strategic offer as an action The action set of all agents is

[0301] 6) Reward: Reward is defined as the revenue r of the generator in a power market clearing. t g , the reward set of all agents is r t =[r t 1 ,...r t g ,...r t m ].

[0302] MATD3 algorithm solution process:

[0303] Input: Number of training times N T , value network learning rate lr c , policy network learning rate lr a , discount rate γ, value network training frequency f, soft update frequency

[0304] Output: The bidding strategies of each power generator in the equilibrium state of the power market

[0305] Initialize the strategy network parameters θ and value network parameters ω of each power producer agent a ,ω b

[0306] Initialize the target strategy network θ of each power producer agent targ and target value network parameter ω targ,a ,ω targ,b

[0307] fort=1 to N T do

[0308] forg=1 to m do

[0309] The agent observes the current environment

[0310] Add noise to the policy network output to generate actions Agents report prices to the market environment

[0311] end for

[0312] The central controller collects state observations of all agents

[0313] The electricity market is cleared, and the reward is r t =[r t 1 ,...r t g ,...r t m ] Feedback to each agent

[0314] Storage state transfer matrix quaternary (s t ,a t ,r t ,s t+1 )

[0315] ift mod f=0then

[0316] Randomly sample a mini-batch (s) from the experience replay pool t ,a t ,rt ,s t+1 )

[0317] for g=1 to m do

[0318] Compute target policy network output and the target value network output

[0319] Update value network parameters

[0320] ift mod(f×f a )=0do

[0321] Update the policy network parameters θ g

[0322] Soft update target value network parameters

[0323] Soft Update Target Policy Network Parameters

[0324] end if

[0325] end for

[0326] end if

[0327] end for

[0328] Embodiment 12:

[0329] A verification experiment of a multi-market equilibrium analysis method for electricity, carbon and green certificates based on multi-agent reinforcement learning is as follows:

[0330] 1. Generator set parameter acquisition

[0331] In this embodiment, a modified IEEE 30-node test system is used for simulation analysis. The system has 30 nodes, 41 branches, 20 loads and 6 generator sets. Among them, units G1 and G2 are wind turbines owned by renewable energy power generators, units G3-G6 are conventional energy units owned by conventional energy power generators, and the parameters of the generator sets are shown in Table 1.

[0332] Table 1 Generator set parameters

[0333]

[0334] 2. Construction of a two-layer optimization model for the electricity market

[0335] 1) Construction of a two-layer optimization model for the electricity-carbon market

[0336] The electricity-carbon market two-level optimization model includes three parts: renewable energy profit maximization model, conventional energy generator profit maximization model and day-ahead electricity market clearing model. The goal of the upper model 1 is to maximize the profit of renewable energy generators, which is obtained by subtracting their power generation costs from the day-ahead market clearing income. Its calculation formula is formula (1) and the constraint condition is formula (2); the goal of the upper model 2 is to maximize the profit of conventional energy generators, which is obtained by subtracting their power generation costs and carbon emission costs from the day-ahead market clearing income. The calculation formula is formula (5) and the constraint condition is formula (2); the lower model is the day-ahead electricity market clearing model, the objective function formula (6) is to minimize the total power generation cost, the constraint formula (7) is the node power balance constraint, the constraint formula (8) is the upper and lower limit constraints of the unit output, and the constraint formula (9) is the branch flow constraint.

[0337] 2) Construction of a two-tier optimization model for the electricity-green certificate market

[0338] The two-level optimization model of the electricity-green certificate market includes three parts: the renewable energy profit maximization model, the conventional energy generator profit maximization model, and the day-ahead electricity market clearing model. The goal of the upper model 1 is to maximize the profit of renewable energy generators, which is obtained by adding the income from the day-ahead market clearing plus the income from the green certificate market minus its power generation cost. The calculation formula is (11) and the constraint condition is (2); the goal of the upper model 2 is to maximize the profit of conventional energy generators, which is obtained by subtracting its power generation cost from the income from the day-ahead market clearing. The calculation formula is (12) and the constraint condition is (2); the lower model is the day-ahead electricity market clearing model.

[0339] 3) Construction of a two-tier optimization model for the electricity-carbon-green certificate market

[0340] The two-level optimization model of the electricity-carbon-green certificate market includes three parts: the renewable energy profit maximization model, the conventional energy generator profit maximization model, and the day-ahead electricity market clearing model. The goal of the upper model 1 is to maximize the profit of renewable energy generators, which is obtained by adding the revenue of the day-ahead market clearing to the revenue of the green certificate market minus its power generation cost. The calculation formula is (13), and the constraint condition is (2); the goal of the upper model 2 is to maximize the profit of conventional energy generators, which is obtained by subtracting its power generation cost and carbon emission cost from the revenue of the day-ahead market clearing. The calculation formula is (14), and the constraint condition is (2); the lower model is the day-ahead electricity market clearing model.

[0341] 3. MATD3 algorithm model training

[0342] The aforementioned two-layer optimization model and reinforcement learning algorithm are used to solve the market equilibrium of the test system. The training process consists of 30,000 steps. The first 10,000 steps are to randomly generate actions to obtain as many observation states as possible as training data for the action network and the value network. In the next 20,000 steps, the agent generates quotation actions based on the policy network. The value network and policy network of the agent are continuously trained and updated. The policy network gradually learns the optimal quotation strategy and finally generates stable quotation actions.

[0343] 4. Simulation analysis of the multi-market equilibrium of electricity, carbon and green certificates

[0344] According to the objective function and constraints in the multi-market two-layer optimization model established above, the MATD3 algorithm model was first trained on the IEEE 30-node system. After verifying its effectiveness, it was applied to different types of market two-layer optimization models, and then the impact of different carbon quotas and carbon prices on the profits and quotations of power generation units, the impact of different green certificate prices on the quotations of renewable energy power generators, and the impact of different carbon prices and green certificate prices on the quotations of power generators and market clearing electricity prices were analyzed.

[0345] The specific simulation results are as follows:

[0346] 1) Algorithm effectiveness verification training

[0347] The MATD3 algorithm model is trained on the IEEE 30-node system. The training process is as follows: Figure 5 As shown in the figure, it can be seen that as the training progresses, the quotations of various power generators gradually decrease from the initial overly high quotations. When the number of training times reaches 20,000, the quotations gradually converge.

[0348] 2) Simulation analysis of the impact of carbon emission quotas and carbon prices on the equilibrium point of the electricity-carbon market

[0349] In order to explore the impact of carbon quota on market equilibrium point, the present invention sets e f The values ​​are 30t, 20t and 10t, which correspond to the three scenarios of loose carbon quota, moderately tightened carbon quota and severely tightened carbon quota. At the same time, in order to explore the impact of carbon price on the market equilibrium point, p is set CET From 0$ / MW to 20$ / MW in increments of 2$ / MW.

[0350] The impact of carbon quotas and carbon prices on market clearing electricity prices is as follows: Figure 6 As shown, scenarios 1, 2, and 3 correspond to scenarios of loose carbon quotas, moderately tightened carbon quotas, and severely tightened carbon quotas, respectively.

[0351] As can be seen from the figure, the clearing electricity price shows an upward trend with the increase of carbon price. The reason is that with the increase of carbon price, high-emission units need to spend higher costs to purchase carbon emission rights from the carbon market to generate electricity, and their total profit decreases. Therefore, power generators with high-emission units hope to increase the clearing electricity price in the power market on the day before by reporting prices higher than the cost of power generation, thereby increasing their profits in the power market.

[0352] Among the three carbon quota scenarios, the increase in market-clearing electricity prices is relatively small when carbon quotas are moderately tightened. In this scenario, only high-emission units have the motivation to falsify costs. When carbon quotas are severely tightened, low-emission units also need to purchase carbon emission rights. At the same time, the power generation profits of high-emission units may even be less than the cost of purchasing carbon emission rights. Therefore, in this scenario, conventional energy power generators all have the behavior of falsifying costs, and electricity prices rise significantly. In the scenario of loose carbon quotas, the carbon market has less constraints on the power generation of units. When carbon prices are low, the carbon market has little impact on the quotations of power generators. When carbon prices are high, power generators can obtain higher profits from trading in the carbon market. At this time, power generators tend to trade in the carbon market and choose to falsify costs in the electricity market, resulting in a sharp increase in clearing electricity prices.

[0353] 3) Simulation analysis of the impact of green certificate prices on the equilibrium point of the electricity-green certificate market

[0354] In order to explore the impact of green certificate price on market equilibrium point, the present invention sets p TGC From 0$ / MW to 10$.MW in increments of 1$ / MW.

[0355] What are the bids of renewable energy generators and market clearing electricity prices under different green certificate prices? Figure 7 As shown in the figure, as the price of green certificates rises, the bids of renewable energy power generators gradually decrease, and eventually tend to report the actual power generation cost. The main reason is that the income of renewable energy power generators in the green certificate market depends on their actual power generation. If they report a higher price, they may become marginal units, resulting in reduced power generation, which in turn leads to reduced income in the green certificate market.

[0356] 4) Simulation analysis of the coupling effect of carbon price and green certificate price on the equilibrium point of electricity-carbon-green certificate market

[0357] In order to explore the impact of carbon price and green certificate price on the market equilibrium point, the present invention sets the carbon price from 0$ / MW to 20$ / MW with a step size of 5$ / MW, and sets p TGC The quota will increase from 0$ / MW to 10$ / MW in steps of 2.5$ / MW, and the carbon quota will be set as a moderately tightened scenario.

[0358] The impact of carbon price and green certificate price on market equilibrium point Figure 8As shown in Figure 2, it can be seen that the market clearing electricity price increases with the increase of carbon price and decreases with the increase of green certificate price.

[0359] In order to further study the coupling effect of carbon price and green electricity price on the bidding strategy of power generators, this paper analyzes the quotations of renewable energy unit G1, low emission unit G5 and high emission unit G3 under different carbon prices and green certificate prices. Figure 9-11 As shown. In the high carbon price scenario, low-emission units tend to profit in the carbon market, and high-emission units hope to raise the clearing electricity price to increase profits, so conventional energy units have the motivation to report higher prices. Conventional energy units falsely report costs, which makes renewable energy units also tend to report higher prices. At this time, the participation of the green certificate market increases the risk aversion of renewable energy units to the reduction of their own power generation, eliminating their tendency to report high prices. It can be concluded that the participation in the carbon market increases the tendency of power generators to use market power in electricity market quotations, and the participation in the green certificate market reduces the tendency of power generators to use market power in electricity market quotations.

[0360] Comprehensive analysis of the simulation results shows that:

[0361] 1. Setting a moderately tightened carbon quota and an appropriate carbon price can increase the profits of low-emission units, effectively stimulate their willingness to generate electricity, and have a greater effect on reducing emissions from the power system; severely tightened or loose carbon quotas cannot effectively stimulate low-emission units to generate electricity, and have little effect on reducing emissions from the power system; excessively high carbon prices will cause electricity prices to soar, disrupting the operating order of the power market.

[0362] 2. Participation in the green certificate market can increase the profits of renewable energy power generators, encourage renewable energy units to generate electricity, give up their market power in the electricity market, and thus reduce the market clearing electricity price.

[0363] 3. The market-clearing electricity price is positively correlated with the carbon price and negatively correlated with the green certificate price. In the current power system environment, electricity prices are relatively greatly affected by carbon prices. Participation in the carbon market promotes power generators to use market power in the electricity market, while participation in the green certificate market restricts power generators from using market power in electricity market quotations.

[0364] From the experimental results, it can be seen that the electricity-carbon-green certificate multi-market equilibrium research method based on multi-agent reinforcement learning proposed in the present invention is a Nash equilibrium, which has high accuracy and can effectively analyze the electricity-carbon-green certificate market equilibrium.

[0365] In summary, the present invention proposes a method for studying the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning. The method first establishes a two-layer optimization model of the electricity-carbon market, a two-layer optimization model of the electricity-green certificate market and a two-layer optimization model of the electricity-carbon-green certificate market; then constructs an equilibrium solution algorithm based on MA, 3; finally, the effectiveness of the algorithm is verified on a modified IEEE30-node test system. Thus, the solution of the equilibrium model of the electricity-carbon-green certificate market and the analysis of the coupling effects of multiple markets are realized. The simulation analysis of the example shows that the electricity-carbon-green certificate multi-market equilibrium research method based on multi-agent reinforcement learning proposed in the present invention can effectively analyze the coupling effects of the carbon market and the green certificate market on the equilibrium of the electricity market.

Claims

1. A method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning, characterized in that: The following steps are involved: 1) Establish a two-tier optimization model for the electricity-carbon market and a two-tier optimization model for the electricity-green certificate market; 2) Based on the two-tier optimization model of the electricity-carbon market and the two-tier optimization model of the electricity-green certificate market, establish an electricity-carbon-green certificate market equilibrium model; 3) Taking the power generation institutions in the electricity-carbon-green certificate market as intelligent entities, the equilibrium model of the electricity-carbon-green certificate market is solved using the equilibrium solution algorithm to obtain the power generation institution quotation plan that makes the electricity market balanced; The electricity-carbon-green certificate market equilibrium model includes a conventional energy power generation optimization model for the electricity-carbon-green certificate market, a renewable energy power generation optimization model for the electricity-carbon-green certificate market, and a day-ahead power market clearing model for the electricity-carbon-green certificate market; The optimization model of conventional energy power generation in the electricity-carbon-green certificate market is as follows: In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n; is the output of the power generation mechanism g; a g and b g are the slope and intercept of marginal cost respectively; is the upper intercept limit; Provide quotation for power generation organizations; CET is the unit carbon emission cost in the carbon emission rights market; f Free carbon emission quota; g is carbon emissions; The optimization model of renewable energy power generation in the electricity-carbon-green certificate market is as follows: In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n; p TGC is the green certificate price; The day-ahead electricity market clearing of the electricity-carbon-green certificate market is as follows: -S n,m ≤B n,m (i n -θ m )≤S n,m (9) Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output; The steps of solving the electricity-carbon-green certificate market equilibrium model using the equilibrium solution algorithm include: 3.1) Set the number of training times N T , value network learning rate lr c , policy network learning rate lr a , discount rate γ, value network training frequency f, soft update frequency; 3.2) Construct a policy network and a value network, and train them; 3.3) Initialize the strategy network parameters θ and value network parameters ω of each agent a , value network parameter ω b ; Initialize the target strategy network θ of each agent targ and target value network parameter ω targ,a 、Target value networkω targ,b ; 3.4) The agent observes the current electricity market clearing process and obtains the local observation value of the agent that reflects the market clearing result g'=1,2,…,m;m is the number of agents;t is the number of iterations; 3.5) The policy network generates actions that represent the agent’s offer The agent reports the price to the market environment; 3.6) The central controller collects state observations of all agents Power market clearing will reward Feedback to each agent; store the state transfer matrix quadruple (s t ,a t ,r t ,s t+1 ); is the reward fed back to agent g'; s t+1 、s t is the state observation of the tth and t+1th iterations; a t for action; 3.7) If t mod f = 0, then randomly sample a mini-batch (s) from the experience replay pool t ,a t ,r t ,s t+1 ), and calculate the target policy network output and the target value network output Update value network parameters Value Network Parameters If t mod(f×f a )=0, then update the strategy network parameter θ g ' , soft update target value network parameters Target value network parameters The soft update is to perform a weighted average of the current network parameters and the network parameters of the previous iteration; 3.8) Repeat steps 3.4) to 3.7) until a power generation agency quotation scheme that balances the power market is obtained.

2. According to claim 1, a method for analyzing the equilibrium of electricity, carbon and green certificates in multiple markets based on multi-agent reinforcement learning is characterized in that: The electricity-carbon market two-layer optimization model includes an electricity-carbon market renewable energy power generation optimization model, an electricity-carbon market conventional energy power generation optimization model, and an electricity-carbon market day-ahead power market clearing module; The electricity-carbon market renewable energy generation optimization model is as follows: In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model; is the set of renewable energy units at node n; is the output of the power generation mechanism g; a g and b g are the slope and intercept of marginal cost respectively; is the upper intercept limit; Provide quotations for power generation organizations; The conventional energy generation optimization model of the electricity-carbon market is as follows: In the formula, p CET is the unit carbon emission cost in the carbon emission rights market; f For free carbon emission quotas; is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n; Among them, carbon emissions e g As shown below: In the formula, is the carbon emission intensity of the unit; The day-ahead electricity market clearing model of the electricity-carbon market is as follows: -S n,m ≤B n,m (i n -θ m )≤S n,m (17) Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output; is the set of power generation mechanisms at node n.

3. According to the method of claim 1, the method is characterized by: The two-layer optimization model of the electricity-green certificate market includes the electricity-green certificate market renewable energy power generation optimization model, the electricity-green certificate market conventional energy power generation optimization model, and the electricity-green certificate market day-ahead power market clearing model; The optimization model of renewable energy power generation in the electricity-green certificate market is as follows: In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of renewable energy units at node n; p TGC is the green certificate price; The optimization model of conventional energy power generation in the electricity-green certificate market is as follows: In the formula, is the node electricity price calculated by the lower-level day-ahead market clearing model, is the set of conventional energy units at node n; The day-ahead electricity market clearing of the electricity-green certificate market is as follows: -S n,m ≤B n,m (i n -θ m )≤S n,m (25) Where d is the number of the load; n and m are the numbers of the nodes; is the load set on node n, Ω n is the set of nodes connected to node n, is the power demand of load d, θ n and θ m are the phase angles of node n and node m, respectively, and B n,m is the branch susceptance connecting node n and node m, B n,m (θ n -θ m ) represents the power on the branch, S n,m is the upper capacity limit of the branch; It is the upper and lower limits of the unit output.

4. According to the method of claim 1, the method is characterized by: The input of the policy network is the local observation of the agent reflecting the market clearing result The output is the action that represents the agent's offer 5. According to the method of claim 1, the method is characterized by: The input of the value network is the local observations of all agents, that is, the global state s = [o 1 ,o 2 ,...,o m ], the output is the evaluation of the agent's action quality q(s,a;ω i ); a is the action; ω i is the value network parameter.

6. According to the method of claim 1, the method is characterized by: The policy network and the value network are connected by taking out the quadruple (s t ,a t ,r t ,s t+1 )train; Tuple(s t ,a t ,r t ,s t+1 ) is as follows:

7. The method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning according to claim 6 is characterized in that: Training policy network μ(o i θ i ) aims to improve the value network’s evaluation of the agent’s actions q(s,a; ω i ); Policy network μ(o i θ i ) is as follows: In the formula, θ i is the decision variable; J i (θ 1 ,...,θ m ) is the gradient vector; E S is the expectation of the policy network parameter q for the state s; β is the learning rate of the policy network parameter; To update the parameters.

8. The method for analyzing the equilibrium of multiple markets of electricity, carbon and green certificates based on multi-agent reinforcement learning according to claim 6 is characterized in that: The goal of training the value network is to make the value network evaluate the quality of the agent's actions q(s,a;ω i ) is closer to the action-value function The training process of the value network is as follows: In the formula, represents the estimate of the value network output Q; Represents the error between the value network output Q and the estimated value; α represents the learning rate of the value network parameters; represents the reward; γ represents the discount rate.

Images