Electricity-carbon transaction method based on multi-agent reinforcement learning

Abstract

The invention relates to the technical field of power systems, and discloses an electricity-carbon transaction method based on multi-agent reinforcement learning, and the method comprises the following steps: constructing a deep coupling P2P transaction system architecture containing electricity and carbon transaction participants, and determining the transaction variety of a carbon electricity market; providing a network fee strategy for the power distribution company, and allowing the clearing electricity price of a buyer corresponding to the power distribution company to be discounted adaptively according to the transaction volume; a decomposable reward including the overall transaction fairness of the electricity distribution and sale market and the transaction competitiveness of each market participant is constructed; and a partially observable Markov decision-making process oriented to the deep coupling transaction is provided, and a multi-agent deep deterministic strategy gradient algorithm is adopted to optimize the deep coupling property transaction of the distribution level. The method has the advantage that the competitiveness of a power distribution company can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to an electricity-carbon trading method based on multi-agent reinforcement learning. Background Art

[0002] Prior ArtAn agent-based approach to modeling interactions between emission market and electricity marketAn agent-based approach is proposed to simulate the interactions between carbon trading market and electricity market through Q-learning algorithm, in which carbon emissions are regulated by the power generation side. Prior ArtPeer-to-Peer Joint Electricity and Carbon Trading Based on Carbon-Aware Distribution Locational Marginal PricingA P2P joint carbon trading model integrating bilateral carbon trading market is proposed. Prior ArtNash Equilibrium Estimation and Analysis in JointPeer-to-Peer Electricity and Carbon Emission Auction Market With MicrogridProsumersThe market design of joint P2P electricity market and carbon emission auction market is proposed to identify uncertainties that may lead to potential financial losses. Prior ArtDistributed Energy and Carbon Emission Right Tradingin Local Energy Systems Considering the Emission Obligation on Demand SideConsidering the carbon emission obligations on the demand side, a double taxation mechanism and a trading framework combining P2P energy rights and carbon emission rights are proposed. Although the above work has achieved results, there are two issues that need more attention. First, most studies focus on minimizing operating costs or maximizing social welfare, and few pay attention to transaction fairness, which is a key factor in ensuring the efficiency of distribution market marketing, especially in extreme cases where most electricity retailers may go bankrupt due to monopoly. Second, the trading products of electricity and carbon rights are too simple. Carbon trading in these projects used to be a product separate from electricity trading. This is a short-sighted approach that may lead to inefficient and suboptimal results in electricity trading. More trading products for the electricity carbon market need to be developed.

[0003] Prior Art Cooperative game of electricity retailer in China's spot electricity market proposed a cross-regional electricity trading model for electricity retailers in the spot market, in which competitiveness is evaluated by shape value. Prior Art Investigating the effect of competitiveness power inestimating the average weighted price in electricity market took market competitiveness as a significant variable and studied the impact of market power on electricity prices in the electricity market. Prior Art A Risk-based Competitive Bi-level Framework for Operation of Active Distribution Networks with Networked Microgrids proposed a risk-based competitive framework for distribution companies, enabling them to win the competition with other competitors as sellers. However, the research mainly focused on the evaluation of competitiveness, which is far from enough for distribution companies from the perspective of competitiveness improvement strategies. In addition, the expression of competitiveness in the prior art places too much emphasis on operating profit or cost, and to some extent ignores another important indicator - the impact of trading market share on marketing competitiveness. According to the existing technology A Comprehensive Competitiveness Decision Method for Electricity Retailers Considering the Integration of Device Operation, Investment Planning, and Marketing Strategy, market share has a pulling effect on profitability, and the increase in profits can economically empower e-commerce with more marketing strategies to expand market share. In this case, market share and profitability should be used as two coordinated parts to formulate competitiveness. In addition, it can be seen from the above work that in the P2P joint construction, the competitiveness of specific distribution companies is not greatly improved. As the only market entity with distribution network assets, distribution companies should certainly make full use of this indispensable role and formulate clear pricing strategies to further improve their competitiveness in this field. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an electric carbon trading method based on multi-agent reinforcement learning.

[0005] The purpose of the present invention is achieved through the following technical solution: A method for electric carbon trading based on multi-agent reinforcement learning, comprising the following steps:

[0006] Step S1: construct a deeply coupled P2P trading system architecture including electricity and carbon trading participants, which includes smart meters, local servers, and central servers;

[0007] The smart meter is used to measure power consumption / generation and carbon emissions in real time and transmit the information to a local server;

[0008] The local server makes electric carbon trading decisions through the transaction controller TCR and encodes the transaction into a smart contract through blockchain technology;

[0009] The central server is managed by the DSO and is responsible for running the ECMS system, which includes Pub / Sub servers, authentication intelligence databases, and analysis tools;

[0010] Step S2: Determine the transaction types in the carbon power market;

[0011] Trading varieties include single carbon quota, single non-green electricity, single green electricity, carbon quota and non-green electricity coupling, carbon quota and green electricity coupling, and carbon quota sharing;

[0012] Step S3: Propose a transmission fee strategy for the distribution company, allowing the distribution company's corresponding buyer's clearing price to be adaptively discounted according to the transaction volume;

[0013] Step S4: Construct a decomposable reward that includes the overall transaction fairness of the distribution and sales electricity market and the transaction competitiveness of each market participant; and propose a partially observable Markov decision process for deep coupled transactions, which regards the transactions at the distribution level as a partially observed Markov decision process in the discrete time domain;

[0014] Step S5: Use a multi-agent deep deterministic policy gradient algorithm to optimize the deeply coupled property transactions at the distribution level.

[0015] Specifically, the transaction income of the single carbon quota is:

[0016]

[0017] In the formula, It represents the transaction price of a single carbon quota between busbar b and busbar m; It represents the trading volume of a single carbon quota between busbar b and busbar m; It represents the trading income of a single carbon quota, which is a linear function of the trading volume of a single carbon quota;

[0018] The transaction income of the single non-green electricity is:

[0019]

[0020] In the formula, in the formula, represents the power generation cost coefficient of the carbon emission unit; represents the non-green electricity transaction volume; It represents the carbon quota transaction price of busbar b; It represents the price of the transmission fee between busbar b and busbar m; Represents the transaction income of a single non-green electricity; Ω CEU is the busbar set equipped with carbon emission units in the distribution network, Ω L It is the busbar collection for power load;

[0021] The transaction income of the single green electricity is:

[0022]

[0023] In the formula, represents the green electricity trading volume; ρ av represents the average carbon emission density; △t is the unit carbon electricity trading cycle; Ω GS is the set of green power units in the distribution network; Formula (3a) represents the transaction income when bus b is equipped with green power units; Formula (3b) represents the transaction income when bus b is an electricity user; where, and For the cost of Internet access; and is the green electricity value, which is the avoided carbon emission cost of the same power generation of the benchmark thermal power unit;

[0024] The transaction income of the carbon quota coupled with non-green electricity is:

[0025]

[0026] In the formula, is the electricity trading volume in the coupled transaction of carbon quota and non-green electricity, is the carbon quota transaction price in the coupled transaction of carbon quota and non-green electricity, The trading volume of carbon quotas in the coupled transaction of carbon quotas and non-green electricity, represents the electricity generation cost of the electricity seller or the disutility of the electricity buyer; It represents the sum of NUC cost and CE cost; Indicates the carbon rights transaction amount; the transaction volume of carbon quotas in the carbon quota and non-green electricity coupling transaction is expressed as:

[0027]

[0028] In the formula is the coupling coefficient between non-green electricity and carbon permit;

[0029] The transaction income of the carbon quota coupled with green electricity is:

[0030]

[0031] In the formula, The electricity trading volume in the coupled transaction of carbon quota and green electricity; The carbon quota trading volume in the coupled transaction of carbon quota and green electricity; is the carbon quota transaction price in the coupled transaction of carbon quota and non-green electricity, It represents the cost of Internet access fee; Indicates the NUC cost; Represents the carbon quota transaction cost.

[0032] Specifically, the network fee strategy is:

[0033]

[0034] In the formula, Ω DisCo It represents the busbar set where the distributed generators invested by the distribution company are located; represents the original node electricity price sold by bus b to bus m; d bm is the discount factor; For the final settlement of electricity price, is the node electricity price of bus b, is the electricity transaction price between bus b and bus m.

[0035] Specifically, the partially observed Markov decision process described in step S4 includes the tuple<N,S,O,A,T,R,γ> , where N agents have a set of states S, a set of private observations O, a set of actions A, a set of reward functions R, a state transition function T, and a discount factor γ.

[0036] Specifically, for the state S and the observation value O, we have:

[0037] Environmental status t ={o 1,t ,o 2,t ,…,o N,t}∈S

[0038] The agents of the environment state are divided into two types:

[0039] 1. At time t, the local observation value is described as:

[0040]

[0041] Where P b,t , Q b,tis the active / reactive power, is the initial carbon permit, is carbon emissions, ρ b,t is the carbon emission intensity, NUC price, v b,t is the node voltage;

[0042] 2. Reference to the designated agent of the DSO, defined as:

[0043]

[0044] In the formula, is the active line flow; is the reactive line flow; Represents the active and reactive power purchased by distribution companies from the wholesale market and ancillary service market; P represents the total amount of carbon quotas purchased by the distribution company from the external carbon market; t / Q t Indicates the injected active / reactive power of the node; Indicates the initial carbon quota of the node; represents the actual carbon emission of the node; ρ t represents the carbon emission density of the node; represents the node electricity price; v t Represents the node voltage.

[0045] Specifically, the action set A is: system action a t ={a 1,t ,a 2,t …,a i,t ,…,a n,t}, is the action of all agents at time t;

[0046] For the first type of agent, the action at time point t is defined as a 6-dimensional vector:

[0047]

[0048] Among them, the transaction volume of 6 proposed products, product single carbon license Product single non-green power supply Product Single Green Power Supply Product coupling carbon permit and non-green electricity Product Coupling Carbon Permit & Green Electricity and product carbon permit sharing

[0049] For the second type of agent, its action is defined as a three-dimensional vector:

[0050]

[0051] Where P t in is the active power purchased from the wholesale market, is the reactive power purchased from the ancillary service market, W t ex Carbon permits purchased from external carbon markets.

[0052] Specifically, for the state transfer function T; through the state transfer function s t+1 =T(s t ,a 1:N, ) derives the environmental transition from time T to time T+1, where s t represents the environmental state at time T; a 1:N,t represents the behavior of all agents at time t.

[0053] Specifically, for the reward function R: system reward function r t = {r 1,t ,r 2,t ,…,r n,t}, describes the rewards of all agents at time t, and divides the rewards into two categories according to the two types of agents;

[0054] The first category includes electricity purchase transaction volume, electricity transaction share and carbon transaction share, as shown in the following formula:

[0055]

[0056] In the formula, Ω i is the set of buses equipped with distributed generators held by the i-th agent; and σ2 represent the standard transaction volume and market share coefficient of the ith agent, respectively; P b,t Indicates active power, W b,t represents the total carbon allowance of busbar b, and represents the electricity sales transaction volume and carbon quota sales transaction volume of bus b, represents the electricity trading share of the ith entity; represents the carbon quota trading share of the ith entity;

[0057] The second category includes power trading amount, power trading fairness and carbon trading fairness. The reward formula is:

[0058]

[0059] In the formula, represents the carbon price in the external carbon market at time t; It represents the carbon quota allocated to busbar b from the external carbon market; P represents the active / reactive power price in the wholesale market / ancillary service market; g,t It represents the active power generation of generator set g at time t; N represents the equivalent carbon quota trading volume of parent line b; g / N b Indicates the number of generator sets / busbars in the distribution network, and σ1 represent the standard transaction volume and fairness coefficient of DSO respectively; P t in is the active power purchased from the wholesale market, is the reactive power purchased from the ancillary services market, and They are to use the Jain teaching index to adjust the fairness of electricity-carbon trading; Represents the remaining carbon rights of bus bar b after carbon rights trading.

[0060] Specifically, the multi-agent deep deterministic policy gradient algorithm includes a weighted field approximation module, an embedding layer module and a cascade proxy module;

[0061] The weighted field approximation module uses the weighted information of the agents within the type to approximate the joint action of N agents into N weighted actions based on the weighted effect of the agent i's neighborhood. for:

[0062]

[0063] In the formula, w i,j Indicates the degree of influence of agent j on agent i; a j represents the action of agent j; j∈N(i) is the neighboring agent of agent i; ζ a_i,j represents a small disturbance;

[0064] Pairwise Q-value function Q θ_i (s,a i ,a j ) is expanded by Taylor's theorem and expressed as:

[0065]

[0066] In the formula, ∑ j ζ a_i,j =0, where and δ i,j ∈[0,1], transform to get the final approximation

[0067] The embedding layer module introduces an action representation space And consider a factor strategy π o , strategy π o It is parameterized by two components, one is the mapping function f:Γ→A embedded in the action, and the other is the internal strategy πS×Γ→[0,1]. At a given state point, the distribution characteristics of the action are:

[0068] E t ~π i (· |s t ) t =f(E t ) (twenty four)

[0069] In the formula, π i For E t ∈Γ for sampling, the function f deterministically represents E t Mapped to an action in set A;

[0070] The cascade proxy module is:

[0071]

[0072] in and They are the electrical factor and the carbon factor respectively.

[0073] The present invention has the following advantages:

[0074] The six differentiated electricity trading products and transmission fee trading strategies proposed in this invention can improve the competitiveness of distribution companies. The proposed WFTD3 algorithm can achieve higher rewards, lower reward deviation and medium computational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A deeply coupled market architecture at the distribution level that takes into account the direct carbon emissions of high-carbon enterprises;

[0076] Figure 2 for transactional management infrastructure;

[0077] Figure 3 The architecture of WFTD3 for solving E&C transactions in conjunction with P2P transactions;

[0078] Figure 4 It is a practical 152-node system single-line diagram;

[0079] Figure 5 The trading results of six carbon electricity products (I);

[0080] Figure 6 The trading results of six carbon electricity products (II);

[0081] Figure 7The trading results of six carbon electricity products (III);

[0082] Figure 8 The trading results of six carbon electricity products (IV);

[0083] Fig. 9 The trading results of six carbon electricity products (V);

[0084] Fig.10 The competitiveness and fairness results under different weighting coefficients;

[0085] Fig.11 Comparison of the stability and convergence of the algorithms. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0087] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0088] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0089] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.

[0090] like Figures 1 to 11 As shown, a method for electric carbon trading based on multi-agent reinforcement learning includes the following steps:

[0091] Step S1: Construct a deeply coupled P2P trading system architecture including electricity and carbon trading participants, which includes smart meters, local servers, and central servers; it also includes market participants participating in bilateral electricity trading and carbon trading, the continuous electricity and carbon trading process organized by the electricity service department, and the actual implementation of the distribution network related to electricity and carbon trading;

[0092] The smart meter is used to measure power consumption / generation and carbon emissions in real time and transmit the information to a local server; each market participant has a smart meter, which can be installed at the gateway between the market participant's load / generation and the corresponding distribution line;

[0093] The local server makes electric carbon trading decisions through the transaction controller TCR, and encodes the transaction into a smart contract through blockchain technology; the local server includes a transaction controller (TCR) and an engineering construction management system (ECMS), the transaction controller is a software engine responsible for receiving electric carbon information from the smart meters it holds, making further electric carbon trading decisions, such as electric carbon bidding or quotation, and encoding the final accepted bid or quotation into a blockchain-based smart contract program through the client application, and the engineering construction management system replaces the engineering construction quotation / bid with the registered engineering construction transaction management platform, which is operated by the central server;

[0094] The central server is managed by DSO and is responsible for running the ECMS system, which includes a Pub / Sub server, an authentication intelligent database and analysis tools; the Pub / Sub server is a Pub / Sub server built based on the publish-subscribe computer network design pattern, in which subscribers (electric carbon distribution market participants) can subscribe to transaction information (electric carbon offer / Bid) to a virtual information channel (usually called a room or subject room), and at the same time, publishers can receive transaction information from subscribers and then broadcast it to other game distribution market participants through a shared channel, all of which is transmitted in the data format of JSON via the network protocol MQTT, the authentication intelligent database is an authenticated intelligent database that contains verified data information to ensure electric carbon transactions, such as the local server serial number, the public key for verifying the block and other necessary information during the execution of the smart contract, and the analysis tool is used to store the electromechanical transaction information of the transaction for further statistics and offline analysis; through the information exchange between the above-mentioned infrastructure parts, the transaction execution between certain pairs of market entities can be determined. The information cleared in the confirmed electric carbon transaction will then be encoded into a blockchain-based smart contract, which is a program held by each local server and carries technically permitted records such as transaction time, address, code, agreement, trigger mechanism, electric carbon quantity and price.

[0095] At the circulation level, a rich array of electricity trading products is the essential feature of a sound electricity market. It can stimulate the responsiveness of different market players, encourage competition in the trading process, and ultimately promote the realization of carbon emission reduction. This paper proposes six electricity-carbon trading products;

[0096] Step S2: Determine the transaction types in the carbon power market;

[0097] The trading varieties include single carbon quota, single non-green electricity, single green electricity, carbon quota and non-green electricity coupling, carbon quota and green electricity coupling, and carbon quota sharing. As a traditional trading product, the trading of single carbon quota is completely independent of electricity trading and can be paired between any two market entities with excess or insufficient carbon rights. The trading income of the single carbon quota is:

[0098]

[0099] In the formula, in the formula, It represents the transaction price of a single carbon quota between busbar b and busbar m; It represents the trading volume of a single carbon quota between busbar b and busbar m; It represents the trading income of a single carbon quota, which is a linear function of the trading volume of a single carbon quota;

[0100] Single non-green electricity refers to electricity generated by carbon emission units, such as GT, DG, and CCS-GT units, that meets the load needs of electricity users. The transaction income of the single non-green electricity is:

[0101]

[0102] In the formula, represents the power generation cost coefficient of the carbon emission unit; represents the non-green electricity transaction volume; represents the carbon quota trading price of busbar b; It represents the price of the transmission fee between busbar b and busbar m; Represents the transaction income of a single non-green electricity; Ω CEU is the busbar set equipped with carbon emission units in the distribution network, Ω L It is the busbar collection for power load;

[0103] Single green electricity refers to green electricity distributed power generation sources, such as electricity generated by distributed photovoltaics, which promotes the environmental protection of distribution networks through green value. The transaction income of the single green electricity is:

[0104]

[0105] In the formula, represents the green electricity trading volume; ρ av represents the average carbon emission density; △t is the unit carbon electricity trading cycle; Ω GS is the set of green power units in the distribution network; Formula (3a) represents the transaction income when bus b is equipped with green power units; Formula (3b) represents the transaction income when bus b is an electricity user; where, and For the cost of Internet access; and is the value of green electricity, which is expressed as the cost of carbon emissions generated by the same power generation of the benchmark thermal power unit that can be avoided; the coupling of carbon quota and non-green electricity is different from the single carbon permit and single non-green electricity product. The carbon permit and non-green electricity coupling product deeply couples carbon permits with non-green electricity at a certain proportion of trading volume. In this product, the seller specifically refers to the carbon emission unit with residual carbon emission rights, and the buyer specifically refers to the electricity user with deficit carbon emission rights. From the seller's point of view, this product can increase the sales of electricity carbon through the deep coupling of carbon permits and non-green electricity. In addition, from the buyer's point of view, this product can reduce the frequency of transactions, thereby improving the convenience of transactions. It should be noted that the carbon quasi-non-green power coupling product does not mean that it can completely replace the single carbon quasi-non-green power product. The reason is that the core feature of the carbon permit and non-green power coupling product, that is, the proportional trading volume of carbon permits and non-green electricity, cannot absolutely guarantee that the trading volume of carbon permits and non-green electricity coupling is lower than the trading volume of single carbon permits and single non-green electricity.

[0106] The transaction income of the carbon quota coupled with non-green electricity is:

[0107]

[0108] In the formula, is the electricity trading volume in the coupled transaction of carbon quota and non-green electricity, is the carbon quota transaction price in the coupled transaction of carbon quota and non-green electricity, The trading volume of carbon quotas in the coupled transaction of carbon quotas and non-green electricity, represents the electricity generation cost of the electricity seller or the disutility of the electricity buyer; It represents the sum of NUC cost and CE cost; Indicates the carbon rights transaction amount; the transaction volume of carbon quotas in the carbon quota and non-green electricity coupling transaction is expressed as:

[0109]

[0110] In the formula is the coupling coefficient between non-green electricity and carbon permit;

[0111] Carbon quota and green electricity coupling, like carbon permit and non-green electricity coupling products, carbon emission permit and green electricity coupling products combine carbon emission permit and green electricity at a certain proportion of trading volume. In this product, the seller refers to the green power distributed generation unit with residual carbon rights, and the buyer refers to the electricity user with deficit carbon rights. The transaction income of carbon quota and green electricity coupling is:

[0112]

[0113] In the formula, The electricity trading volume in the coupled transaction of carbon quota and green electricity; The carbon quota trading volume in the coupled transaction of carbon quota and green electricity; is the carbon quota transaction price in the coupled transaction of carbon quota and non-green electricity, It represents the cost of Internet access fee; Indicates the NUC cost; represents the carbon quota transaction cost; the carbon quota transaction volume in the carbon quota and green electricity coupling transaction is expressed as:

[0114]

[0115] In the formula is the coupling coefficient between green power and carbon permit;

[0116] Carbon quota sharing, carbon emission permit is an economic means to implement physical carbon emission control on carbon emissions. In the real carbon market, these entities with excess or deficit carbon emission permits may be held by the same company. In this context, we have envisioned a carbon permit sharing product to support the circulation of carbon permits between carbon permit entities. The difference between product carbon permit sharing and product single carbon permit lies in two aspects. First, the trading counterparts of product carbon permit sharing must meet the prerequisite of belonging to the same company, such as the same ER, while single carbon permits do not need to. Second, the transaction volume of carbon permit sharing is free, which can be considered as the transfer of carbon permits within the same company, as shown in the following formula, while the transaction volume of single carbon permits is not free:

[0117]

[0118] Step S3: Propose a transmission fee strategy for the distribution company, allowing the buyer's clearing price of the distribution company to be adaptively discounted according to the transaction volume; as follows:

[0119]

[0120] In the formula, Ω DisCo It represents the busbar set where the distributed generators invested by the distribution company are located; represents the original node electricity price sold by bus b to bus m; dbm is the discount factor; For the final settlement of electricity price, is the node electricity price of bus b, is the electricity transaction price between bus b and bus m.

[0121] Step S4: Construct a decomposable reward that includes the overall transaction fairness of the distribution and sales electricity market and the transaction competitiveness of each market participant; and propose a partially observable Markov decision process for deep coupled transactions. Since each market participant manages its transaction as a sequential decision process and each market participant can only observe partial information about the transaction, the transaction at the distribution level is regarded as a partially observed Markov decision process (POMDP) ​​in the discrete time domain;

[0122] In the partially observed Markov decision process paradigm, each market participant is regarded as an agent managing the electricity-carbon strategy. The partially observed Markov decision process consists of the tuple<N,S,O,A,T,R,γ> , where N agents have a set of states S, a set of private observations O, a set of actions A, a set of reward functions R, a state transition function T and a discount factor γ. For the states S and observations O, we have:

[0123] Environmental status t ={o 1,t ,o 2,t ,…,o N,t}∈S, reflects the overall operation and electricity-carbon marketing at the distribution level. It consists of partial observations of all agents at the tth time slot. The agents of the environmental state are divided into two types. One refers to a certain market participant, such as a distribution company, an independent ER, a general electricity trader or a carbon trader. At time t, the local observation is described as:

[0124]

[0125] Where P b,t , Q b,t is the active / reactive power, is the initial carbon permit, is carbon emissions, ρ b,t is the carbon emission intensity, NUC price, v b,t is the node voltage;

[0126] The other is to cite the designated agent of DSO. In order to ensure the feasibility of operation, manage the carbon trading amount in the wholesale market and the carbon market, and ensure the fairness of carbon trading, DSO is allowed to receive more observations than other agents. It is defined as:

[0127]

[0128] In the formula, is the active line flow; is the reactive line flow; Represents the active and reactive power purchased by distribution companies from the wholesale market and ancillary service market; P represents the total amount of carbon quotas purchased by the distribution company from the external carbon market; t / Q t Indicates the injected active / reactive power of the node; Indicates the initial carbon quota of the node; represents the actual carbon emission of the node; ρ t represents the carbon emission density of the node; represents the node electricity price; v t Represents the node voltage.

[0129] Furthermore, the action set A is: system action a t ={a 1,t ,a 2,t …,a i,t ,…,a n,t}, is the action of all agents at time t;

[0130] For the first type of agent, the action at the tth time point is defined as a 6-dimensional vector: In order to better understand EDLMP, the Lagrangian function of formula (11) is further expanded:

[0131]

[0132] Among them, the transaction volume of 6 proposed products, product single carbon license Product single non-green power supply Product Single Green Power Supply Product coupling carbon permit and non-green electricity Product Coupling Carbon Permit & Green Electricity and product carbon permit sharing

[0133] For the second type of agent, its action is defined as a three-dimensional vector:

[0134]

[0135] Where P t in is the active power purchased from the wholesale market, is the reactive power purchased from the ancillary service market, W t ex Carbon permits purchased from external carbon markets.

[0136] For the state transfer function T; after the action is executed, according to the Markov decision process, through the state transfer function s t+1 =T(s t ,a 1:N, ) derives the environmental transition from time T to time T+1, where s t represents the environmental state at time T; a 1:N,t represents the behavior of all agents at time t; in particular, it is necessary to carefully study the transition of the environment state determined by the agent's behavior.

[0137] From the perspective of power trading, the action set Widely accepted “grid-friendly” properties should be followed by establishing the following physical operating limitations:

[0138]

[0139] Q b,t =P b,t κ b (14i)

[0140]

[0141] Formula (14a) represents the total power of bus b, that is, P b,t , including single non-green power supply of products Single green power Product coupling carbon certificate & electrical part of non-green power supply And product coupling carbon certificate & electrical part of green power P b,t It is positive for the seller and negative for the buyer; Equations (14b) and (14c) represent the active and reactive balance of the node; Equations (14d) and (14e) limit the bidirectional line power flow; Equation (14f) is the node voltage drop; Equation (14g) describes the relationship between line flow, current and voltage through second-order cone programming relaxation; Equation (14h) represents the node voltage range; Equation (14i) uses the power factor κ b To describe P b,t With Q b,t The relationship between the two; Formula (14j) represents the restriction Q b,t The range is within the range.

[0142] Through equations (14a) to (14j), the action set The relevant environmental state that can be set at time t, that is Transfer to the next time step t+1. Note that the transfer shown in the equations, (14a)-(14j), can be solved by a Python-based power system analysis tool in a practical sense.

[0143] At the same time, from the perspective of carbon trading, The purpose of is to adapt to the carbon-conscious characteristics of the physical operation of the distribution network. Through the following formula, and It can be directly obtained by equation (5) and equation (7) and Sure;

[0144]

[0145]

[0146] The left side of equation (15a) represents the total carbon emissions, including the carbon emissions of e-commerce, such as Carbon emissions of electricity buyers, as shown in the formula As shown in the figure, the carbon emissions generated by line power loss are as follows: As shown; and The 1 / 2 in the equation indicates that in a power transaction, the seller and the buyer should bear the carbon emissions equally. The right side of equation (15a) indicates the total carbon emissions generated by all carbon emission units in the distribution network. Equation (15b) indicates the calculation of the carbon emissions along line l along the power loss at time t. Equation (15c) indicates the calculation of the carbon emission intensity of the node, which is equal to the weighted average of all carbon emissions of the injected power in the tth time slot. Equation (15d) defines the carbon intensity of line l as the carbon intensity of the node flowing out of line l. Equation (15e) indicates the equivalent carbon emissions of bus b as a power seller, which includes: the carbon emissions of the adjacent lines around bus b, as shown in Equation As shown, the equivalent carbon emission injection of power generation or load is as follows: b,t ρ b,t △t / 2 shows the direct carbon emissions of bus b. If a high-carbon enterprise is installed on bus b, the direct carbon emissions of bus b are as follows. Formula (15f) represents the equivalent carbon emissions of bus b as an electricity buyer. Its carbon emission composition is similar to formula (15e). Formula (15g) represents the total carbon allowance of bus b, which includes the single carbon allowance of the product. Product Carbon Permit Coupling and Carbon Permit Portion in Non-Green Power Product Carbon Permit Coupling and Carbon Permit Portion in Green Power As well as product carbon permit sharing, formula (15h) represents the remaining carbon rights of bus b after carbon rights trading, formula (15i) represents the initial carbon permit of bus b in the (t+1)th time slot, which is equal to the remaining carbon permit minus the equivalent carbon emissions, and the non-negativity constraints in formulas (15j)-(15k) indicate that the carbon emissions of bus b at any time cannot exceed its carbon emission permit.

[0147] Through equations (15a)-(15k), the action set The part can set the relevant environmental state at time t, that is Moving to the next time step t+1, it should be noted that the state transfer part in equations (15a)-(15k) and the final transfer part in equations (14a)-(14j) are designed in a complementary manner and by sharing the power flow result P b,t and Mutually coupled, it is clear that equations (15a)-(15f) can be used to determine P b,t and Then directly calculate,

[0148] (14a)-(14j), and then consider the high-dimensional nature of the remaining equations. Equations (15g)-(15k), the set of related actions and environmental states may not be restricted by all these operations at a certain moment. To solve this problem, the related actions and environmental states are adjusted by the forced boundary technique, which is expressed as:

[0149]

[0150] For the reward function R: system reward function r t = {r 1,t ,r 2,t ,…,r n,t}, describes the rewards of all agents at time t, and divides the rewards into two categories according to the two agent types; as mentioned above, the first type of agents includes distribution companies, independent ERs, general power traders, and carbon traders. Considering the fierce competitive pressure and marketing survival challenges in the power purchase and distribution market, the power purchase trading competitiveness is used as the reward for this agent type,

[0151] The first category includes electricity purchase transaction volume, electricity transaction share and carbon transaction share, as shown in the following formula:

[0152]

[0153] In the formula, Ω i is the busbar set of distributed generators held by the i-th agent; and σ2 represent the standard transaction volume and market share coefficient of the ith agent, respectively; and represent electricity trading share and carbon trading share respectively; P b,t Indicates active power; W b,t It represents the total carbon quota of busbar b; and represents the electricity sales transaction volume and carbon quota sales transaction volume of bus b; represents the electricity trading share of the ith entity; represents the carbon quota trading share of the ith entity;

[0154] The second category includes power trading amount, power trading fairness and carbon trading fairness. The reward formula is:

[0155]

[0156] In the formula, represents the carbon price in the external carbon market at time t; It represents the carbon quota allocated to busbar b from the external carbon market; P represents the active / reactive power price in the wholesale market / ancillary service market; g,t It represents the active power generation of generator set g at time t; N represents the equivalent carbon quota trading volume of parent line b; g / N b Indicates the number of generator sets / busbars in the distribution network, and σ1 represent the standard transaction volume and fairness coefficient of DSO respectively; P t in is the active power purchased from the wholesale market, is the reactive power purchased from the ancillary services market, and They are to use the Jain teaching index to adjust the fairness of electricity-carbon trading; Represents the remaining carbon rights of bus bar b after carbon rights trading.

[0157] Step S5: Use a multi-agent deep deterministic policy gradient algorithm to optimize the deeply coupled property transactions at the distribution level.

[0158] MATD3 algorithm: As an extension of multi-agent learning, MATD3 algorithm solves non-stationary high-dimensional continuous problems through central criticism. Technically, each agent is equipped with an actor-critic framework, including an actor network and two critic networks. The actor network μ of agent i φi With parameter φ i , is characterized by working in a decentralized manner and only receiving local observations o i,t As input, and produces a deterministic action a i,t In addition, critics of the network Q θ_i,1,Q θ_i,2 , respectively, with θ i,1 ,θ i,2 parameterized by receiving the environment state s of all agents t and action a 1:N The connection of is taken as input, and then provides an estimate of the q-value of the action in a given state, working in a centralized manner. This centralized training and decentralized execution allows multiple agents to both compete and collaborate. Here, for simplicity, in the following context, the centralized criticism of each agent i is denoted as Q θ_i To improve stability during training, we transform the target actor network (μ φ'i ) and the target critic network (Q θ’_i,1 ,Q θ’_i,2 ) are integrated together to achieve soft update;

[0159]

[0160] Where τ<<1 is the update coefficient. Based on the target network, the critic network and the actor network are updated by time difference and deterministic policy gradient respectively:

[0161]

[0162] Proposed Multi-Agent Deep Deterministic Policy Gradient Algorithm WFTD3 Algorithm: As mentioned above, whether through central training or independent training, the performance of MATD3 in achieving the trade-off between scalability, privacy protection, and stability is somewhat weak. For this reason, a new WFTD3 algorithm is proposed, in which weighted field approximation, embedding layer, and cascade proxy technology are proposed;

[0163] The multi-agent deep deterministic policy gradient algorithm includes a weighted field approximation module, an embedding layer module and a cascade proxy module;

[0164] The weighted field approximation module, in order to mitigate the curse of dimensionality while protecting privacy from the influence of all agents' accessible states, uses the weighted information of agents within the type (called neighboring agents) to approximate the joint action of N agents as N weighted actions. Here, agents involved in the same type of market share (e.g., distribution companies & electricity retailers, electricity traders, or carbon permit traders) are adjacent agents to each other, and the weighted action a_i based on the neighborhood of agent i is:

[0165]

[0166] In the formula, w i,j Indicates the degree of influence of agent j on agent i; a j represents the action of agent j; j∈N(i) is the neighboring agent of agent i; ζa_i,j represents a small disturbance;

[0167] Pairwise Q-value function Q θ_i (s,a i ,a j ) is expanded by Taylor's theorem and expressed as:

[0168]

[0169] In the formula, ∑ j ζ a_i,j = 0, the first-order term can be removed, that is, Can be removed, including and You can θ_i (s,a i ,a j ) is proved to be close to zero under the condition that M is smooth (such as a linear function), so can be eliminated, resulting in the final approximation

[0170] Embedding layer module,In reinforcement learning, the method of generating actions in a given state is usually to directly pass through the policy network; the disadvantage of this method is that it cannot capture the structure in the underlying action space well, which weakens the learning performance in a limited action set; to make up for this gap, an action representation space is introduced And consider a factor strategy π o , strategy π o It is parameterized by two components, one is the mapping function f:Γ→A embedded in the action, and the other is the internal strategy πS×Γ→[0,1]. At a given state point, the distribution characteristics of the action are:

[0171] E t ~π i (· |s t ) t =f(E t ) (twenty four)

[0172] In the formula, π i For E t ∈Γ for sampling, the function f deterministically represents E t Mapped to an action in set A;

[0173] The behavior in the cascade agent module, Eq. (12), can be divided into two parts, namely, carbon trading volume and electricity trading volume. Their different physical properties and sizes will lead to differences in action structures and further unstable gradients during training. To solve this problem, a cascade actor technology is proposed, which is:

[0174]

[0175] in and They are electricity factor and carbon factor respectively. Specifically, during the training process, the electricity trading strategy is trained first. Get electric trading action Then, consider and i,t , Carbon trading policy Generate carbon trading behavior

[0176] The electromechanical transaction model proposed in the present invention is implemented on a real 152-bus ADN, and its single-line block diagram is shown in FIG. Figure 4 As shown. ADN includes 5 industrial zones (yellow markers), 3 commercial zones (blue markers), 3 public zones (green markers), and 3 residential zones (RZs). In addition, the real ADN contains renewable generators (i.e., PV) and traditional generators with carbon emissions (i.e., gas turbine units (GT), diesel generators (DG), and gas turbines based on carbon capture systems (CCS-GT) units). The parameters of the generators are listed in Table 1;

[0177] Table 1

[0178]

[0179] The affiliation between generators and electricity providers is shown in Table 2;

[0180] Table 2

[0181]

[0182]

[0183] To illustrate the impact of market share on the competitiveness of distribution companies, two cases are set up, including:

[0184] Case 1: Calculate the competitiveness of distribution companies taking into account trading volume, carbon market share and electricity market share.

[0185] Case 2: Calculation of the competitiveness of distribution companies considering only the trading volume.

[0186] Table 3 shows the competitiveness results of distribution companies in these two different cases. It can be seen from Table 3 that compared with Case 2, trading profits, carbon market share, and electricity market share increased by 33.19%, 7.26%, and 23%, respectively, and the competitiveness of distribution companies finally increased by 22.9%, which fully demonstrates the positive impact of market share considerations on the competitiveness of distribution companies.

[0187] Table 3

[0188]

[0189] In order to highlight the advantages of the proposed trade transaction strategy on the competitiveness of distribution companies, three comparison scenarios are set up, including:

[0190] Scenario 1: Calculate the competitiveness of distribution companies through the proposed six commercial transaction products and NUC's bottom-line strategy;

[0191] Scenario 2: Calculating the competitiveness of distribution companies through a single carbon trading product and a single electricity trading product (taken from

[14] );

[0192] Scenario 3: Without considering the NUC back-cover strategy, the competitiveness of the distribution company is calculated through the proposed six electricity carbon trading products. Table IV gives the competitiveness of the distribution company in these three scenarios. Figure 5-Figure 9 The trading results of six electricity-carbon trading products between four functional zones (i.e., IZs, cz, pz, and rz) are given.

[0193] As can be seen from Table 4, compared with Scenario 2 and Scenario 3, the profit of Scenario 1 increased by 57.29% and 22.75%, respectively, followed by the carbon market share increased by 14.09% and 3.34%, and the electricity market share increased by 23.33% and 51.30%, respectively. As a result, the competitiveness of distribution companies was finally improved by 33.92% and 24.01%, respectively. The improvement of profits, carbon market share, electricity market share and competitiveness of distribution companies fully verifies the effectiveness of the proposed power trading strategy.

[0194] Table 4

[0195]

[0196]

[0197] from Figure 5 , Figure 8 and Fig. 9 It can be seen that the trading scope of single carbon permits and coupled carbon permits is mutually exclusive. Specifically, the trading of single carbon permits is distributed within and outside the region, and the trading of coupled carbon permits is distributed within and outside the region. Figure 6 It can be seen that carbon permit sharing products occur between various power generation units, mainly from photovoltaic units to other traditional carbon emission units. It can be seen that carbon emission permit sharing can play a good role in improving the competitiveness of distribution companies by coordinating carbon emission permits between power generation units (especially photovoltaic units and other carbon emission units).

[0198] from Figure 7 , Figure 8 and Fig. 9 It can be seen that single electricity trading (i.e., single non-green electricity and single green electricity) and coupled electricity trading (i.e., carbon permit coupled with non-green electricity and carbon permit coupled with green electricity) are mainly distributed in CZs and IZs.

[0199] Due to the huge electricity-carbon demand of these two functional zones. In more detail, the transaction amount of coupled electricity trading is greater than that of single electricity trading, which indicates that coupled electricity trading between users of CZs and IZs is more desirable.

[0200] In summary, the proposed variable e-commerce transaction products can enhance the competitiveness of distribution companies by providing users with differentiated consumption choices.

[0201] In order to reveal the relationship between the fairness of the power distribution market and the competitiveness of distribution companies, a sensitivity test was conducted by increasing the weight coefficients σ1 and σ2 from 0 to 1. The competitiveness and fairness results under the relative changes of σ1 and σ2 are shown in Fig.10 As shown. Fig.10 It can be seen that there is a generally negative correlation between the competitiveness and fairness of distribution companies in the distribution market, which is typically manifested in that the maximum competitiveness and the minimum fairness of distribution companies appear at the same point {σ1,σ2}={0.5,0.5}. Fig.10 It can be seen that: 1) As σ2 increases in the range of 0 to 0.5 (i.e., the importance of market share to the competitiveness of distribution companies increases), the distribution company's carbon market share also increases, resulting in an increase in the competitiveness of distribution companies; ii) σ2 increases in the range of 0.5 to 1. The increase in market share cannot further increase the profits of distribution companies, which forces distribution companies to reduce their market share and ultimately reduce their competitiveness. In addition, we can see that changes in the competitiveness of distribution companies will cause a sharp drop in fairness, indicating that the impact of distribution company competitiveness on the fairness of the distribution market is significant. In summary, this sensitivity test can coordinate the relationship between market share and profit for distributed power grid operators and provide data support for distributed power grid operators to make fair decisions in the distribution market.

[0202] In order to prove that the proposed WFTD3 has the performance of convergence, stability and computational efficiency, we introduce three multi-agent reinforcement learning methods: MAPPO, COMA and MATD3 for comparison. The learning curves of these four methods are shown in Figure 2. Fig.11 As shown, the shaded area represents half the standard deviation of the average evaluation of 10 experiments. Table 5 gives the computational performance results of these four methods within 500 scenarios. Fig.11 It can be seen that the proposed WFTD3 converges to the highest scenario reward of 538.67, while MAPPO, COMA, and MATD3 converge to lower scenario rewards of 320.44, 174.00, and 332.71, respectively.

[0203] Table 5

[0204]

[0205] As can be seen from Table V, the computation time of the WFTD3 algorithm is 10.47% lower than that of the MATD3 algorithm, and 8.55% and 9.59% higher than that of the MAPPO algorithm and the COMA algorithm, respectively. This is because i) the use of weighted q-value functions enables WFTD3 to only connect local observations and actions, rather than global observations and actions in MATD3; ii) in WFTD3, the requirements for explicit agent communication and decentralized review network operation increase the computational burden compared to the shared review network and zero information interaction in MAPPO and COMA. In summary, the WFTD3 algorithm proposed in this paper achieves a good balance between convergence, stability, and computational efficiency;

[0206] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above-mentioned technical content without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any changes, modifications, equivalent changes and modifications made to the above embodiments based on the technology of the present invention without departing from the content of the technical solution of the present invention belong to the protection scope of the present technical solution.

Claims

1. A method for electric carbon trading based on multi-agent reinforcement learning, characterized in that: The following steps are involved: Step S1: construct a deeply coupled P2P trading system architecture including electricity and carbon trading participants, which includes smart meters, local servers, and central servers; The smart meter is used to measure power consumption / generation and carbon emissions in real time and transmit the information to a local server; The local server makes electric carbon trading decisions through the transaction controller TCR and encodes the transaction into a smart contract through blockchain technology; The central server is managed by the DSO and is responsible for running the ECMS system, which includes Pub / Sub servers, authentication intelligence databases, and analysis tools; Step S2: Determine the transaction types in the carbon power market; Trading varieties include single carbon quota, single non-green electricity, single green electricity, carbon quota and non-green electricity coupling, carbon quota and green electricity coupling, and carbon quota sharing; Step S3: Propose a transmission fee strategy for the distribution company, allowing the distribution company's corresponding buyer's clearing price to be adaptively discounted according to the transaction volume; Step S4: construct a decomposable reward that includes the overall transaction fairness of the distribution and sales power market and the transaction competitiveness of each market participant; and A partially observable Markov decision process for deep coupled trading is proposed, which regards the trading at the distribution level as a partially observable Markov decision process in the discrete time domain. Step S5: Use a multi-agent deep deterministic policy gradient algorithm to optimize the deeply coupled property transactions at the distribution level.

2. According to claim 1, a method for electric carbon trading based on multi-agent reinforcement learning is characterized in that: The trading income of the single carbon quota is: In the formula, It represents the transaction price of a single carbon quota between busbar b and busbar m; It represents the trading volume of a single carbon quota between busbar b and busbar m; It represents the trading income of a single carbon quota, which is a linear function of the trading volume of a single carbon quota; The transaction income of the single non-green electricity is: In the formula, in the formula, represents the power generation cost coefficient of the carbon emission unit; represents the non-green electricity transaction volume; represents the carbon quota trading price of busbar b; It represents the price of the transmission fee between busbar b and busbar m; Represents the transaction income of a single non-green electricity; Ω CEU is the busbar set equipped with carbon emission units in the distribution network, Ω L It is the busbar collection for power load; The transaction income of the single green electricity is: In the formula, represents the green electricity trading volume; ρ av represents the average carbon emission density; △t is the unit carbon electricity trading cycle; Ω GS is the set of green power units in the distribution network; Formula (3a) represents the transaction income when bus b is equipped with green power units; Formula (3b) represents the transaction income when bus b is an electricity user; where, and For the cost of Internet access; and is the green electricity value, which is the avoided carbon emission cost of the same power generation of the benchmark thermal power unit; The transaction income of the carbon quota coupled with non-green electricity is: In the formula, is the electricity trading volume in the coupled transaction of carbon quota and non-green electricity, is the carbon quota transaction price in the coupled transaction of carbon quota and non-green electricity, The trading volume of carbon quotas in the coupled transaction of carbon quotas and non-green electricity, represents the electricity generation cost of the electricity seller or the disutility of the electricity buyer; It represents the sum of NUC cost and CE cost; Indicates the carbon rights transaction amount; the transaction volume of carbon quotas in the carbon quota and non-green electricity coupling transaction is expressed as: In the formula is the coupling coefficient between non-green electricity and carbon permit; The transaction income of the carbon quota coupled with green electricity is: In the formula, The electricity trading volume in the coupled transaction of carbon quota and green electricity; The carbon quota trading volume in the coupled transaction of carbon quota and green electricity; is the carbon quota transaction price in the coupled transaction of carbon quota and non-green electricity, It represents the cost of Internet access fee; Indicates the NUC cost; Represents the carbon quota transaction cost.

3. The electric carbon trading method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The network fee strategy is: In the formula, Ω DisCo It represents the busbar set where the distributed generators invested by the distribution company are located; represents the original node electricity price sold by bus b to bus m; d bm is the discount factor; For the final settlement of electricity price, is the node electricity price of bus b, is the electricity transaction price between bus b and bus m.

4. The electric carbon trading method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The partially observed Markov decision process in step S4 includes the tuple<N,S,O,A,T,R,γ> , where N agents have a set of states S, a set of private observations O, a set of actions A, a set of reward functions R, a state transition function T, and a discount factor γ.

5. The electric carbon trading method based on multi-agent reinforcement learning according to claim 4 is characterized in that: For the state S and observation value O, we have: Environmental status t ={o 1,t ,o 2,t ,…,o N,t }∈S The agents of the environment state are divided into two types:

1. At time t, the local observation value is described as: Where P b,t , Q b,t is the active / reactive power, is the initial carbon permit, is carbon emissions, ρ b,t is the carbon emission intensity, NUC price, v b,t is the node voltage; 2. Reference to the designated agent of the DSO, defined as: In the formula, is the active line flow; is the reactive line flow; Represents the active and reactive power purchased by distribution companies from the wholesale market and ancillary service market; represents the total amount of carbon allowances purchased by distribution companies from the external carbon market; P t / Q t Indicates the injected active / reactive power of the node; Indicates the initial carbon quota of the node; represents the actual carbon emission of the node; ρ t represents the carbon emission density of the node; represents the node electricity price; v t Represents the node voltage.

6. The electric carbon trading method based on multi-agent reinforcement learning according to claim 5 is characterized by: The action set A is: system action a t ={a 1,t ,a 2,t …,a i,t ,…,a n,t }, is the action of all agents at time t; For the first type of agent, the action at time point t is defined as a 6-dimensional vector: Among them, the transaction volume of 6 proposed products, product single carbon license Product single non-green power supply Product Single Green Power Supply Product coupling carbon permit and non-green electricity Product Coupling Carbon Permit & Green Electricity and product carbon permit sharing For the second type of agent, its action is defined as a three-dimensional vector: Where P t in is the active power purchased from the wholesale market, is the reactive power purchased from the ancillary service market, W t ex Carbon permits purchased from external carbon markets.

7. The electric carbon trading method based on multi-agent reinforcement learning according to claim 5 is characterized by: For the state transfer function T; through the state transfer function s t+1 =T(s t ,a 1:N ,) derive the environmental transition from time T to time T+1, where s t represents the environmental state at time T; a 1:N,t represents the behavior of all agents at time t.

8. The electric carbon trading method based on multi-agent reinforcement learning according to claim 5 is characterized by: For the reward function R: system reward function r t = {r 1,t ,r 2,t ,…,r n,t }, describes the rewards of all agents at time t, and divides the rewards into two categories according to the two types of agents; The first category includes electricity purchase transaction volume, electricity transaction share and carbon transaction share, as shown in the following formula: In the formula, Ω i is the set of buses equipped with distributed generators held by the i-th agent; and σ2 represent the standard transaction volume and market share coefficient of the ith agent, respectively; P b,t Indicates active power, W b,t represents the total carbon allowance of busbar b, and represents the electricity sales transaction volume and carbon quota sales transaction volume of bus b, represents the electricity trading share of the ith entity; represents the carbon quota trading share of the ith entity; The second category includes power trading amount, power trading fairness and carbon trading fairness. The reward formula is: In the formula, represents the carbon price in the external carbon market at time t; It represents the carbon quota allocated to busbar b from the external carbon market; P represents the active / reactive power price in the wholesale market / ancillary service market; g,t It represents the active power generation of generator set g at time t; N represents the equivalent carbon quota trading volume of busbar b; g / N b Indicates the number of generator sets / busbars in the distribution network, and σ1 represent the standard transaction volume and fairness coefficient of DSO respectively; P t in is the active power purchased from the wholesale market, is the reactive power purchased from the ancillary services market, and They are to use the Jain teaching index to adjust the fairness of electricity-carbon trading; Represents the remaining carbon rights of bus bar b after carbon rights trading.

9. The electric carbon trading method based on multi-agent reinforcement learning according to claim 5 is characterized by: The multi-agent deep deterministic policy gradient algorithm includes a weighted field approximation module, an embedding layer module and a cascade proxy module; The weighted field approximation module uses the weighted information of the agents within the type to approximate the joint action of N agents into N weighted actions based on the weighted effect of the agent i's neighborhood. for: In the formula, w i,j Indicates the degree of influence of agent j on agent i; a j represents the action of agent j; j∈N(i) is the neighboring agent of agent i; ζ a_i,j represents a small disturbance; Pairwise Q-value function Q θ_i (s,a i ,a j ) is expanded by Taylor's theorem and expressed as: In the formula, ∑ j ζ a_i,j =0, where and δ i,j ∈[0,1], transform to get the final approximation The embedding layer module introduces an action representation space And consider a factor strategy π o , strategy π o It is parameterized by two components, one is the mapping function f:Γ→A embedded in the action, and the other is the internal strategy πS×Γ→[0,1]. At a given state point, the distribution characteristics of the action are: AND t ~π i (·|s t ) to t =f(E t ) (24) In the formula, π i For E t ∈Γ for sampling, the function f deterministically represents E t Mapped to an action in set A; The cascade proxy module is: in and They are the electrical factor and the carbon factor respectively.

Images