Electricity-carbon coupling market user side autonomous decision-making method

Abstract

The invention relates to the technical field of power systems, and particularly discloses an electricity-carbon coupling market user-side autonomous decision-making method, which comprises the following steps: constructing an electricity-carbon market transaction model, enabling the model to coordinate an electricity-carbon sensor through an electricity-carbon coordinator, distributing electricity-carbon data to different distributed renewable energy sources DER, and establishing an electricity-carbon market transaction model; the DER obtains a shared model from the electricity-carbon market coordinator, carries out local training, and then combines the local training model to generate a global model; establishing a partial observable Markov decision-making process of discrete time step length to manage the power carbon market bid / price of the DER as a sequential decision-making process; and based on a Markov decision process, each DER executes local training in the federal reinforcement learning model, and each DER obtains an own electricity-carbon market transaction decision through a deep Q network. The advantages of the invention are that the provided carbon electricity market framework can realize better trade-off between the user side transaction cost and the total social welfare of the carbon electricity organization.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a user-side autonomous decision-making method in an electricity-carbon coupling market. Background Art

[0002] Excessive carbon emissions (CE) pose a serious threat to the global climate, such as severe weather events, sea level rise, and destruction of ecosystems and biodiversity. Nearly 48.2% of China's carbon emissions come from fossil fuel power generation. To address these issues, decarbonization has been listed as a key priority by the power sector, and a number of environmentally-based policies have been formulated to reduce CE, such as carbon caps, cap-and-trade, and carbon taxes. As the penetration rate of distributed energy resources (DERs) in distribution networks has increased significantly, carbon trading is seen as an important way to promote local decarbonization and the adaptation of renewable energy at the distribution level. How to fully incentivize users to participate in coupled electricity-carbon trading at the distribution level and how to achieve information privacy and differentiation goals for distributed energy trading are issues that require urgent attention. Existing technologies use smart contract-based trading platforms to achieve energy and carbon quota exchanges between prosumers and microgrids, but ignore the limitations of distribution networks. The prior art proposes an operating mechanism for electricity-carbon trading between distributed photovoltaics based on cross-chain trading technology. The prior art proposes a blockchain application for trading energy and carbon licenses between networked microgrids. However, this solution based on power generation may impose higher CE costs on users, further leading to incentive imbalance and weakening social welfare. As the fundamental driving force for the supply side to produce consumer electronics, the user should also bear internal economic responsibilities. To this end, the prior art proposes the concept of bilateral carbon trading to allocate CE costs to the user side and the supply side.

[0003] Although the CE obligation on the user side has been generally recognized, it is still in its preliminary stage for practical implementation due to the short-sighted consideration of CE allocated from the supply side of the power sector. So far, there has been little research in the field of carbon-to-electricity market, and carbon-to-electricity trading is limited to a small range of typical homogeneous end users, which deviates from the development trend of the carbon-to-electricity market on the user side. Driven by demand-side electrification, more heterogeneous distribution networks from different industries, located in different geographical locations, and with different consumption patterns should be encouraged to participate in the power distribution market. Peer-to-peer (P2P) trading paradigm has become a feasible coordination architecture. Existing technologies take into account the heterogeneous tendencies of DER while achieving transaction privacy. The P2P model can be roughly divided into system-centric mechanism, user-centric mechanism, and hybrid mechanism. In the system-centric mechanism, the central coordinator (usually the distribution network operator (DSO)) basically participates in collecting DER's carbon-to-electricity market bids and optimizing peer carbon-to-electricity market transactions to maximize social welfare. This mechanism relies heavily on the central coordinator and may prove to be unfeasible due to scalability issues and high operation and maintenance costs [7]. As an alternative, nodes in user-centric mechanisms make decisions and communicate directly without a central coordinator, where decentralized models such as primal-dual gradients, consensus algorithms, and alternating direction multipliers are often used. However, the lack of a central coordinator leads to market efficiency and global social welfare failing to achieve the desired results. In this context, hybrid designs have been further developed, such as supply-demand ratios, intermediate market rates, and bill sharing in P2P transactions. In this mechanism, trading nodes are forced to share the same clearing price, which contradicts the basic idea of ​​the P2P market, which is to obtain specific pricing signals for self-interested electricity-carbon market traders.

[0004] In addition, among the above three mechanisms, the electric carbon market relies heavily on accurate parameters, reliable uncertainty estimates and complete economic models, which are usually impractical in reality. To solve this problem, reinforcement learning (RL), as the next generation decision-making method, can play a huge advantage in capturing system uncertainty and adapting to various state dynamics. However, the disadvantage of this centralized learning algorithm lies in the scalability and privacy issues caused by the mandatory acquisition of local observations and actions of all agents. In order to popularize this problem, federated learning (FL) technology began to be integrated into reinforcement learning to form a federated reinforcement learning (FRL) algorithm. Specifically, the natural distribution characteristics of FRL can fully realize local model training without sharing information. So far, the application of FRL in power trading has attracted more and more attention, such as integrated energy systems, building communities, microgrids, etc. It should be pointed out that from the existing technology, it can be seen that the research on FRL in power trading still lacks sufficient consideration of heterogeneous DER with differentiated goals on the user side, which limits the advantage of FRL in enabling DER to make adaptive decisions in a heterogeneous market environment. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for autonomous decision-making on the user side of an electricity-carbon coupling market. The purpose of the present invention is achieved through the following technical solutions: a method for autonomous decision-making on the user side of an electricity-carbon coupling market, the method comprising:

[0006] Construct an electricity-carbon market trading model, which coordinates carbon-electricity sensors through a carbon-electricity coordinator to distribute carbon-electricity data to different DERs. DERs obtain a shared model from the electricity-carbon market coordinator and perform local training, and then merge the local training models to generate a global model; the DERs include a subway model, an aggregated electric vehicle model, a PV photovoltaic model, a carbon emission unit model, and a load aggregator model;

[0007] Establish a partially observable Markov decision process with discrete time steps to manage DER’s electricity carbon market bid / ask as a sequential decision process;

[0008] Based on the Markov decision process, each DER performs local training in the federated reinforcement learning model, and each DER makes its own electricity-carbon market trading decisions through local self-training.

[0009] Specifically, the subway model is:

[0010] K=uk,u∈{1,2,...,U}(1a)

[0011]

[0012] 0≤ISOE i ≤100%(1d)

[0013]

[0014] Among them, formula (1a) represents the total number of subway stations in operation; formula (1b) represents the total operating time from the first station to the terminal station; formula (1c) limits the operating time between each station; formula (1d) limits the operating ISOE of the OESD for each inter-station journey; formulas (1e)-(1g) are the coordination of operating time and electricity carbon market trading cycle; T metro is the unit electricity carbon market transaction cycle; and is the running time between the first and last stations in the tth electric regulation transaction cycle; is the operation time covering the entire interval in the tth electricity transaction cycle, according to the The sum of the running time of the interval; the first station interval of the t-th trading cycle It is also the last station interval of the t-1th trading cycle The last station interval of the tth trading cycle It is the first station interval of the t+1th trading cycle Two adjacent trading cycles and The sum is equal to I S / I E The running time of the inter-station section; Formula (1h) represents the net electricity consumption of the subway during the t-th electricity carbon market trading period, which is represented by the function P(i,T i ,ISOE i )calculate:

[0015]

[0016] In the formula, a 1,i ,a 2,i ,a 3,i ,a 4,i , and a 5,i are pre-calibrated constant coefficients.

[0017] The aggregate electric vehicle model is:

[0018]

[0019] Where, formula (2a) represents the charging power of the aggregated EV Subject to pre-calibration interval [l EV,agg ,μ EV,agg], through this pre-calibrated interval, the aggregated EV obtains the desired charging energy; the hyperparameter matrices H and J are used to describe the sequential charging space:

[0020]

[0021] In the formula, H i is the set of all permutations of rows containing i 1s and (Ti)0s Matrix; (1) i and (0) i The lengths of the values ​​0 and 1 are In addition, and Minimum / maximum charging power And every electric car also, and The minimum / maximum charging power of each individual electric vehicle and Calculation, the calculation is as follows:

[0022]

[0023] Where: μ EV,agg is the upper bound of charging power of aggregated electric vehicles; l EV,agg A lower bound for the charging power of aggregated electric vehicles; The upper limit of charging power for the nth electric vehicle; The lower limit of charging power for the nth electric vehicle;

[0024] The PV photovoltaic model is:

[0025]

[0026] Where: P t PV is the photovoltaic power generation at time t; P t PV,min is the lower limit of photovoltaic power generation at time t; P t PV,max is the upper limit; min is the power factor; κ min Need to be maintained at [κ min Hysteresis,κ min ahead] range.

[0027] Specifically, the carbon emission unit model is:

[0028]

[0029] W t al,dir =W t burn +W t m +W t PFC (7a)

[0030] W t burn =∑ f 44NCV f FC f CC f OF f / 12 (7b)

[0031] W t m =44NC c (1-S c -A c )P al,t △t / 12k (7c)

[0032] W t PFC =(6500ρCF4+9200ρC2F6)∑ t ∑ l P lt / 1000k (7d)

[0033]

[0034] W t of,dir =3.667G t C f η+∑ i E i ,tρi (9)

[0035] W t pep,dir =W t pep,burn +W t pep,pro +W t pep,cbn (10a)

[0036] W t pp,burn =44∑ i AD i,t NCV i EF i OF i / 12 (10b)

[0037] Wt pp,pro =∑ r AD r,t CC r -AD ca,t CC ca -AD pa,t CC pa (10c)

[0038] W t pep,cbn =AD ls,t EF ca PUR ca (10d)

[0039]

[0040] Where: μ g,t is the unit commitment variable of generator unit g at time t (1 for on, 0 for off); is the output of distributed generation g at time t; and are the lower and upper bounds of the output of distributed generation g at time t; and are the lower and upper bounds of the active climbing capacity of generator set g at time t. and is the minimum online and offline time of generator set g at time t. j is the carbon emission of the jth carbon-containing material; CC r is the carbon content in the carbon raw material.

[0041] Formula (4a) represents the active power generation limit; Formula (4b) represents the active power ramp limit; Formulas (4c)-(4d) represent the online time and minimum offline time constraints; is the base load of the dth load aggregator at time t; and is the minimum load and maximum load of the dth load aggregator at time t; and is the maximum and minimum demand response of the dth load aggregator at time t. Formula (5a) limits the load of the dth load aggregator after the demand response; Formula (5b) limits the range of the demand response; Formula (5c) represents the load factor requirement. is the direct carbon emission of the cement plant, ρ cec represents the carbon emission factor; Indicates the output during combustion and clinkering; is the power consumption; a cec 、b cec and c cec is a constant coefficient; Direct carbon emissions from aluminum smelters; Carbon emissions from burning fuel, Carbon emissions for raw material consumption; Carbon emissions for electrolytic aluminum production; f represents the fuel index; NCV f / FC f / OF f Represents the average lower heating value, net consumption and carbon oxidation rate of renewable fuels; CC f It represents the carbon content of renewable fuel per unit calorific value; P al,t Indicates the power consumption of the aluminum smelter; k, NC c , S c , and A c is a constant parameter used to describe the raw material consumption in the production of electrolytic aluminum; ρ CF4 and ρ C2F6 represents the constant CE coefficient; is the direct cost benefit of the steel plant; co / ρ gas / ρ st Indicates the CE coefficient of coal, natural gas and other materials; It indicates the coal and gas needed for steel production; represents steel production and gas production; is the direct carbon emissions of the refinery; G t Indicates the crude oil consumed by the refinery; C f represents the mass fraction of coke in the refined product; η represents the utilization efficiency; E i,t / ρ i represents the i-th energy and the corresponding CE coefficient; Direct carbon emissions from petrochemical plants; Carbon emissions from burning fossil fuels, Carbon emissions from consumption of carbon-containing raw materials, is the carbon emission from calcium carbonate decomposition; i∈{diesel, coke} represents the fossil fuel index, AD i,t / NCV i / EF i / OF i is a constant coefficient used to describe the consumption of the i-th fuel, r∈{coke, blue carbon, electrode paste} represents the index of carbon-containing raw materials; AD r,t Indicates the consumption of carbon-containing raw materials; AD ca,t / AD pa,t Indicates the output of carbide and purified ash; CC ca,t / CC pa,t Indicates the carbon content in carbides and purified ash; AD ls,t represents carbonate consumption; EF ca / PUR caCE coefficient indicating carbonate and carbonate purity; is the direct carbon emission of the commercial center, A is the surface area of ​​the commercial building; ρ j is the CE coefficient of the jth carbonaceous material; d k / m k / ρ k is the average distance, weight of carbon-containing materials and carbon emission coefficient of the kth mode of transportation; ρ h / e h is the CE coefficient of the h-th energy source and the energy consumption of the demolition process; Y CM / T is the service life and daily operating hours.

[0042] Specifically, it also includes constructing a load aggregator model, and the load aggregator model is:

[0043]

[0044] Among them, formula (5a) limits the load of the dth load aggregator after demand response; formula (5b) limits the scope of demand response; formula (5c) represents the load factor requirement.

[0045] Specifically, the electricity carbon market transaction model is:

[0046]

[0047] Q b,t =P b,t κ b (12h)

[0048]

[0049]

[0050] Where: B b is the dielectric constant of node b; G b is the conductance of node b; P b,t is the active power of node b at time t; Q b,t is the reactive power of node b at time t; S l,max is the maximum power flow of distribution line l; and is the active / reactive power flow of distribution line l at time t; a l,t and v b,t is the square current of distribution line l at time t and the square node voltage amplitude of node b at time t; ρ b,t is the carbon emission intensity of node b at time t; ρ l,t is the carbon emission intensity of distribution line l at time t; is the carbon emission associated with the power loss along the distribution line l at time t; is the indirect carbon emission of node b at time t; is the direct carbon emission of node b at time t; is the total carbon emissions of node b at time t; is the amount of carbon emission permits purchased from the external carbon emission market; W b,t The amount of carbon permits traded at node b at time t; where formulas (12a)-(12b) represent the active and reactive power balance of the node; formulas (12c)-(12d) constrain the bidirectional branch power flow; formula (12e) represents the branch voltage drop; formula (12f) represents the power flow through second-order cone relaxation; formula (12g) represents the node voltage range; formula (12h) converts P b,t and Q b,t and power factor κ b Coupling; Formula (12i) limits Q b,t range; formula (12j) indicates CE balance, where the right hand indicates the total CE from the CE unit; and the left hand divides the total CE into three parts: the CE of the electricity seller in formula (12j-1), the CE of the electricity buyer in formula (12j-2), and the CE associated with the branch power loss in formula (12j-3); formula (12k) calculates the CE associated with the branch power loss; formula (12l) calculates the node CE intensity by the weighted average CE of all injected powers; formula (12m) defines the carbon intensity of branch l as the carbon intensity of the node flowing out of branch l; formulas (12n)-(12o) calculate the indirect CE of electricity sellers and buyers; formula (12p) adds the indirect CE and direct CE to obtain the total CE; formula (12q) indicates that the CE of bus B should always be lower than its carbon permit.

[0051] Specifically, the discrete time step partially observable Markov decision process is obtained by<N,S,O,A,T,R,γ> ,Define, N agents, state set S, private observation set O, action set A, reward function set R, state transition function T and discount factor γ;

[0052] State Sets t ={o 1,t ,o 2,t ,…,o N,t}∈S includes partial observations of all agents at time t. The local observation is described as:

[0053] subway:

[0054]

[0055] Where: v b,t is the voltage amplitude of node b at time t; ρl,t is the carbon emission intensity of distribution line l at time t; The initial carbon permit of node b at time t; [v b,t ,ρ b,t ] is the operating characteristic of the distribution network; For transaction information; are the operating parameters of the subway, here, express Up / down time; Indicates that at time t To ISOE set up between stations;

[0056] Other electrocarbon players:

[0057]

[0058] Action Set A t ={a 1,t ,a 2,t …,a i,t ,…,a N,t} is the action of all agents at time t, defined as:

[0059] subway:

[0060]

[0061] Where: P b,t is the electricity trading volume of node b at time t; W b,t The trading carbon allowance of node b at time t; P b,t =P b1,t +P b2,t +…+P bN,t , carbon permit trading volume W b,t =W b1,t +W b2,t +…+W bN,t , and the running time and

[0062] Other electrocarbon players:

[0063] a i,t =[P b,t , W b,t ]∈Α (16)

[0064] State transition T: through function s t+1 =T(s t ,a 1:N,t, ) to achieve the environment state s t With action a1:N,t associated;

[0065] Reward function set R: The personalized target of DER calculated according to the obtained actions. The reward function of all DERs has the same composition and is expressed as follows:

[0066]

[0067] Where: represents the carbon tax price, which is determined by the carbon intensity ρ b,t and the product of carbon tax κ; represents the reward function of DER i, transactional utility that reflects personalized goals; for the cost of carbon emissions; Cost of network usage; is the cost of bilateral carbon trading.

[0068] Specifically, the transaction utility Modeled as a quadratic function:

[0069]

[0070] Where, for subway and aggregated electric vehicles: Indicates energy saving, and are all normal numbers, indicating (P i,t ) and P i,t There is a clear monotonic relationship between them;

[0071] For PV: represents PV consumption, where and are all negative constants;

[0072] For carbon emission units: represents the cost of electricity generation, where and They are all normal numbers;

[0073] For load aggregators: represents the negative utility of expected failure in electricity consumption, where and All are normal numbers;

[0074] Bilateral carbon trading costs Modeled as a linear function of carbon permits:

[0075]

[0076] Where: is the bilateral carbon trading cost; Wim,t is the carbon emission trading volume between two carbon trading entities i and m at time t. Specifically, the training process of the federated reinforcement learning model includes global aggregation of updated local models and an actor-critic network used for local model training;

[0077] Denote I as the set of DERs, and D i Represented as a dataset, each DER trains its own local model The model is then passed to the electricity carbon market coordinator to determine the global model It is represented by w a =f(w1,w2,…,w I ), and then the global model w a Broadcast to all DERs for local updates, achieving the differentiated goals of heterogeneous DERs in a self-decision-making and privacy-preserving manner.

[0078] Specifically, WFTD3 is used as a reinforcement learning algorithm for each DER to train its local model in the electricity carbon market transaction. In WFTD 3, each DER has an actor-critic framework, including an actor network and two critic networks with parameters μ θi The DER i actor network Acting in a decentralized manner, the local observations o i,t As input, and produces a deterministic action a i,t , which is then passed to the critic network Q θi,1 ,Q θ_i,2 , to output Q-value estimates for action evaluation; incorporated into the target actor network in WFTD 3 and the target critic network (Q θ’i,1 ,Q θ’i,2 ), its soft update formula is:

[0079]

[0080] Where τ<<1 is the target update factor, the critic network is updated via temporal difference learning, and the actor network is updated via deterministic policy gradient:

[0081]

[0082] The weighted mean field technique is used to describe the influence of neighbor agents on the behavior of the i-th DER:

[0083]

[0084] Where: θ i is the network parameter; E is the sample size; L(θ i) is the loss function; is the target Q value at time t; γ is the discount factor; s t is the state at time t; For actor networks in s t Actions generated under the state; is the Q value predicted by the Critic network; r i t The reward obtained by the agent from the environment after performing an action; It is the optimization goal of the Actor network; is the gradient of the parameter; is the state at time t; represents the weighted action of the i-th DER, j∈j∈N(i) represents the neighbor agent of the i-th DER, ζ a_i,j represents a slight disturbance, and then Taylor's theorem is used to derive the pairwise Q value function Q θ_i (s,a i ,a j ):

[0085]

[0086] In the formula, due to ∑ j ζ a_i,j = 0, the first-order term can be eliminated; the second term, as the remainder of the Taylor polynomial

[0087] +δ i,j ζ a_i,j andδ i,j ∈[0,1], when Q θ_i (s,a i ,a j ) is M-smooth, we can get Specifically, meta-functions and time-varying control variants are used to coordinate the local models trained in WFTD 3. Global Model of Hedian Carbon Market Coordinator It includes metafunctions and time-varying control variants;

[0088] The metafunction is formulated as:

[0089]

[0090] Where: w is the network parameter; β represents the step size; is the gradient of the metafunction; I is the sample size; the metafunction F(w) is composed of the metafunctions F1(w), F2(w), ..., F I The average value of (w) is expressed as follows, where the function of the i-th DER is defined as:

[0091]

[0092] To update the local model, the gradient of formula (20) is calculated The calculation is as follows:

[0093]

[0094] Further development of unbiased estimation:

[0095]

[0096] Where: is an unbiased estimate; |D i | indicates D i The size of l i (w; x, y) is the error of the evaluation model w when predicting the true label L based on the input data d; Through its unbiased estimate To calculate, through independent batches and Further solution:

[0097]

[0098] For each DER, a random control variable c is proposed i , where the average control variable c = ∑c is initialized for the electricity carbon market coordinator i / I, using the obtained Local model w i The update can be defined as:

[0099]

[0100] Then, the random control variable c is also updated via the calculated gradient i ,

[0101]

[0102] Using equations (24) and (25), local updates are then aggregated to update the parameters of the electricity carbon market coordinator,

[0103]

[0104] Through formulas (23)-(30), a round of communication and update between DER and the electricity carbon market coordinator is completed.

[0105] The present invention has the following advantages:

[0106] 1. This invention proposes a new E&C market framework, which enables heterogeneous DER to achieve its differentiation goals at the user end, and builds an interactive operation model for typical urban rail transit in joint E&C transactions, especially for the rarely developed subway rail transit.

[0107] 2. In order to improve and adapt to the actual carbon market, this paper newly establishes a direct CE model for typical high-carbon enterprises and incorporates it into bilateral DERs transactions at the distribution level.

[0108] 3. This paper proposes a pFedScv framework combined with the WFTD 3 algorithm to solve the carbon-electricity trading problem. The framework allows users to make autonomous decisions and protect privacy during the carbon-electricity trading process.

[0109] 4. The proposed carbon power market framework can achieve a better balance between user-side transaction costs and the total social welfare of carbon power organizations. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Figure 1 This is a schematic diagram of the user autonomous electricity-carbon coupling market mechanism based on pFedScv of the present invention;

[0111] Figure 2 This is a schematic diagram of the coordination between subway operation time and electricity service handling cycle of the present invention;

[0112] Figure 3 This is a schematic diagram of a subway measurement point of the present invention;

[0113] Figure 4 This is a schematic diagram of the pFedScv framework of the present invention;

[0114] Figure 5 For the real electric carbon distribution market;

[0115] Figure 6 Schematic diagram of personalized target costs for five representative DERs under three electricity-carbon market frameworks;

[0116] Figure 7 Schematic diagram of CE costs of five representative DERs under three electricity-carbon market frameworks;

[0117] Figure 8 Schematic diagram of network usage costs for five representative DERs under three electricity-carbon market frameworks;

[0118] Fig. 9 Schematic diagram of bilateral carbon trading costs for five representative DERs under three electricity carbon market frameworks;

[0119] Fig.10 This is a schematic diagram of the first type of electricity trading;

[0120] Fig.11This is a schematic diagram of the second type of electricity trading;

[0121] Fig.12 This is a schematic diagram of the third type of electricity trading;

[0122] Fig.13 This is a schematic diagram of the first type of carbon emission permit trading;

[0123] Fig.14 This is a schematic diagram of the second type of carbon emission permit trading;

[0124] Fig.15 This is a schematic diagram of the third type of carbon emission permit trading;

[0125] Fig.16 This is a schematic diagram of the power trading results in case 1;

[0126] Fig.17 This is a schematic diagram of the power trading results in case 2;

[0127] Fig.18 This is a schematic diagram of the carbon trading results of Case 1;

[0128] Fig.19 This is a schematic diagram of the carbon trading results of Case 2;

[0129] Fig. 20 Convergence curves and stability diagrams of four FL algorithms for load aggregators;

[0130] Fig.21 Convergence curves and stability diagrams of four FL algorithms for aggregated electric vehicles;

[0131] Fig. 22 Convergence curves and stability diagrams of four FL algorithms for distributed energy using CE;

[0132] Fig.23 The convergence curves and stability diagrams of the four FL algorithms for subways;

[0133] Fig.24 Convergence curves and stability diagrams of four FL algorithms for photovoltaics;

[0134] Fig.25 Box plots of four FL algorithms. DETAILED DESCRIPTION

[0135] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0136] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0137] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0138] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description. Figures 1 to 25 As shown, a method for autonomous decision-making on the user side of an electricity-carbon coupling market, the method comprising:

[0139] A power-carbon market trading model is constructed. The model coordinates carbon-electric sensors through a carbon-electric coordinator to distribute carbon-electric data to different DERs. DERs obtain shared models from the power-carbon market coordinator and perform local training, and perform local training to adjust their local policies for personalized goals. Then the power-carbon market coordinator merges the local training models and generates a global model. The DERs include a subway model, an aggregated electric vehicle model, a PV photovoltaic model, a carbon emission unit model, and a load aggregator model. Heterogeneous DERs with differentiated goals: In order to improve trading efficiency and release trading potential, the present invention considers the heterogeneity of 5 representative DERs with different geographical locations, trading modes, and trading goals. In addition to the power-carbon market trading In addition to the common minimization of cost, the differentiated goals of this DER involve subways and aggregated EVs: energy saving; PV: maximizing local consumption; CE units: minimizing power generation costs; load aggregators: minimizing negative utilities. All of these differentiated goals are formulated as reward conditions for effective solutions. In order to more effectively manage the carbon electricity market, some small-scale users are aggregated to form load aggregators. Load aggregators are divided into 5 types according to geographical areas, including IZ, CZ, RZ, PZ and AZ; direct CE units include cement plants, aluminum smelters, steel mills, petrochemical plants, refineries and commercial centers. Direct CE refers to CE generated on the user side due to a series of production and operation activities, such as burning, decomposing and consuming fossil fuels;

[0140] The subway is a relatively undeveloped area, but it should be an important DER in the electricity-carbon market distribution market. From the perspective of electricity, it can both obtain power from the traction system and inject power into the distribution network through regenerative braking. From the perspective of carbon, the subway is undoubtedly one of the most representative transportation sectors, contributing about 20% of CE among all users. For this reason, energy conservation and decarbonization of the subway are regarded as important goals. The traditional subway model mainly achieves energy conservation through operation, route management and renewable energy utilization based on on-board energy storage without updating the existing infrastructure. However, these models only discuss net electricity consumption from the perspective of a single section, rather than from the perspective of unit electricity and electricity trading cycle, which makes it difficult for the subway to participate in trading decisions. In order to overcome this challenge, the present invention proposes a trading operation model for the subway, which enables the coordination between the operating time in multiple station sections and unit electricity-carbon market trading cycles, specifically:

[0141] K=uk,u∈{1,2,...,U}(1a)

[0142]

[0143] 0≤ISOE i ≤100%(1d)

[0144]

[0145]

[0146] For ease of understanding, Figure 2 The coordination between the operation time of multiple inter-station sections and the trading time of the electricity-carbon market is shown, where u is the trip index of the subway line; k is the total number of subway stations within a single trip of the subway line; i is the index of the inter-station section; T i is the running time of the subway in the i-th section; ISOE i is the initial energy state of the on-board energy storage device in the i-th section; formula (1a) represents the total number of subway stations in operation; formula (1b) represents the total operating time from the first station to the terminal station; formula (1c) limits the operating time between each station; formula (1d) limits the operating ISOE of the OESD for each inter-station journey; formulas (1e)-(1g) are the coordinated operating time and the electricity-carbon market trading cycle; T metro is the unit electricity carbon market transaction cycle; and is the running time between the first and last stations in the tth electric regulation transaction cycle; is the operation time covering the entire interval in the tth electricity transaction cycle, according to the The sum of the running time of the interval is calculated; the first station interval I of the tth trading cycle S t It is also the last station interval of the t-1th trading cycle The last station interval of the tth trading cycle It is the first station interval of the t+1th trading cycle Two adjacent trading cycles and The sum is equal to I S / I E The running time of the inter-station section; Formula (1h) represents the net electricity consumption of the subway during the t-th electricity carbon market trading period, which is represented by the function P(i,T i ,ISOE i )calculate:

[0147]

[0148] In the formula, a 1,i ,a 2,i ,a 3,i ,a 4,i , and a 5,i are pre-calibrated constant coefficients.

[0149] like Figure 3As shown, the measurement point of the subway is the intermediate bus between the 110 / 35kV converter and the 35 / 1.5kV traction converter. In this way, the E&C transaction volume of the subway and other DERs can be attributed to the bus at the same voltage level, which is beneficial for calculation and analysis.

[0150] In practical applications, it is difficult for a single electric vehicle to participate in the electricity market due to its own capacity limitations. In order to effectively manage electric vehicle transactions on a large scale, the aggregation of electric vehicles is necessary. Aggregated electric vehicles, as participants in the electricity-carbon market, are charged through smart charging. Usually, the charging requirements of aggregated EVs are determined by six parameters: arrival and departure time The upper and lower limits of charging rate (p EV,U ,p EV,L ) and the maximum and minimum charge energies (e EV,U ,e EV,L ), these six parameters are sent from the EV driver to the charging station and are assumed to be internally consistent, on which the operating charge constraints are derived in the form of a hyperparameter space; the aggregated EV model is:

[0151]

[0152] Where, formula (2a) represents the charging power of the aggregated EV Subject to pre-calibration interval [l EV,agg ,μ EV,agg ], through this pre-calibrated interval, the aggregated EV obtains the desired charging energy; the hyperparameter matrices H and J are used to describe the sequential charging space:

[0153]

[0154] In the formula, H i is the set of all permutations of rows containing i 1s and (Ti)0s Matrix; (1) i and (0) i The lengths of the values ​​0 and 1 are In addition, and Minimum / maximum charging power And every electric car also, and The minimum / maximum charging power of each individual electric vehicle and Calculation, the calculation is as follows:

[0155]

[0156] Photovoltaic integration is usually required to have a minimum power factor, which is achieved by adjusting the power factor through the intelligent inverter, which not only provides active power but also absorbs or generates reactive power. The PV photovoltaic model is:

[0157]

[0158] Where: P t PV is the photovoltaic power generation at time t; P t PV,min is the lower limit of photovoltaic power generation at time t; P t PV,max is the upper limit; min is the power factor; κ min Need to be maintained at [κ min Hysteresis,κ min ahead] range.

[0159] The present invention considers three types of distributed power generation: diesel generators, gas turbines, and gas turbines based on carbon capture systems. The carbon emission unit model is:

[0160]

[0161] Among them, μ g,t is the unit commitment variable of generator unit g at time t (1 for on, 0 for off); is the output of distributed generation g at time t; and are the lower and upper bounds of the output of distributed generation g at time t; and are the lower and upper bounds of the active ramp capacity of generator set g at time t; and is the minimum online time and offline time of generator set g at time t, formula (4a) represents the active power generation limit; formula (4b) represents the active power ramp limit; formulas (4c)-(4d) represent the online time and minimum offline time constraints;

[0162] Direct cement plant Mainly from the combustion and clinkering process, in which the operating conditions are generally constant, so It can be considered as linear with the production of cement plant:

[0163]

[0164] In the formula, is the direct carbon emission of the cement plant, ρ cecIndicates the carbon emission coefficient; D t cec Indicates the output of the combustion and clinker process, expressed by the power consumption Quadratic function calculation of

[0165]

[0166] In the formula, is the power consumption; a cec 、b cec and c cec is a constant coefficient;

[0167] Direct CE for Aluminum Smelters Including fuel combustion CE Raw material consumption CE and electrolytic aluminum production CE

[0168] W t al,dir =W t burn +W t m +W t PFC (7a)

[0169] Determined by the CE properties of the fuel used to produce electrolytic aluminum and calculated by the following formula:

[0170] W t burn =∑ f 44NCV f FC f CC f OF f / 12 (7b)

[0171] Where, f represents the fuel index; NCV f / FC f / OF f Represents the average lower heating value, net consumption and carbon oxidation rate of renewable fuels; CC f Represents the carbon per unit of calorific value of renewable fuels;

[0172] Depends on the consumption level of the raw material, i.e., carbon anode, expressed as:

[0173] W t m =44NC c (1-S c -A c ) al,t△t / 12k (7c)

[0174] Where P al,t Indicates the power consumption of the aluminum smelter; k, NC c , S c , and A c It is a constant parameter used to describe the consumption of raw materials in the production of electrolytic aluminum;

[0175] From the anode effect in the electrolytic aluminum production, two carbon gases CF4 and C2F6 are emitted, which are:

[0176] W t PFC =(6500ρ CF4 +9200ρ C2F6 )∑ t ∑ l P lt / 1000k (7d)

[0177] In the formula, ρ CF4 and ρ C2F6 represents the constant CE coefficient, the direct cost-effectiveness of the steel plant It includes four aspects, including the cost benefits from coal combustion, the cost benefits from gas combustion, the cost benefits from other steel production processes not related to coal and gas, and the cost benefits from steel production. is defined as:

[0178]

[0179] In the formula, ρ co / ρ gas / ρ st Indicates the CE coefficient of materials such as coal and gas; It indicates the coal and gas needed for steel production; represents steel production and gas production;

[0180] Direct CE for refineries Consumption comes from coke combustion and related raw material processing and transportation:

[0181]

[0182] Where G t Indicates the crude oil consumed by the refinery; C f represents the mass fraction of coke in the refined product; η represents the utilization efficiency; E i,t / ρ i represents the i-th energy and the corresponding CE coefficient;

[0183] Direct CE for petrochemical plants Caused by three projects, including fossil fuel burning Carbon-containing raw material consumption Decomposition with calcium carbonate CE:

[0184] W t pep,dir =W t pep,burn +W t pep,pro +W t pep,cbn (10a)

[0185] Determined by diesel and coke consumed:

[0186] W t pp,burn =44∑ i AD i,t NCV i EF i OF i / 12 (10b)

[0187] Where i∈{diesel, coke} represents the fossil fuel index; AD i,t / NCV i / EF i / OF i is the constant coefficient used to describe the i-th fuel consumption;

[0188] Depends on the consumption of carbon-containing raw materials:

[0189] W t pp,pro =∑ r AD r,t CC r -AD ca,t CC ca -AD pa,t CC pa (10c)

[0190] Where r∈{coke, blue carbon, electrode paste} represents the index of carbon-containing raw materials; CC r is the carbon content in the carbon raw material; AD r,t Indicates the consumption of carbon-containing raw materials; AD ca,t / AD pa,t Indicates the production of carbides (carbonaceous products) and purified ash (carbonaceous waste); CC ca,t / CC pa,t Indicates the carbon content in carbides and purified ash;

[0191] also, The formula is defined as

[0192] W t pep,cbn =AD ls,t EF ca PUR ca (10d)

[0193] Among them, AD ls,t represents carbonate consumption; EF ca / PUR ca CE coefficient indicating carbonate and carbonate purity;

[0194]

[0195] Where A is the surface area of ​​the commercial building; W j is the carbon emission of the jth carbon-containing material; ρ j is the CE coefficient of the jth carbonaceous material; d k / m k / ρ k is the weight of carbonaceous materials over the average distance, the CE coefficient of the kth mode of transportation; ρ h / e h is the CE coefficient of the h-th energy source and the energy consumption of the demolition process; Y CM / T is the service life and daily operating hours.

[0196] Specifically, it also includes constructing a load aggregator model, and the load aggregator model is:

[0197]

[0198] in, is the base load of the dth load aggregator at time t; and is the minimum load and maximum load of the dth load aggregator at time t; and are the maximum and minimum demand responses of the dth load aggregator at time t. Formula (5a) limits the load of the dth load aggregator after the demand response; formula (5b) limits the range of the demand response; formula (5c) represents the load factor requirement.

[0199] Specifically, the electricity carbon market transaction model is:

[0200]

[0201] Q b,t =P b,t κ b (12h)

[0202]

[0203]

[0204] Where: B b is the dielectric constant of node b; G b is the conductance of node b; P b,t is the active power of node b at time t; Q b,t is the reactive power of node b at time t; S l,max is the maximum power flow of distribution line l; and is the active / reactive power flow of distribution line l at time t; a l,t and v b,t is the square current of distribution line l at time t and the square node voltage amplitude of node b at time t; ρ b,t is the carbon emission intensity of node b at time t; ρ l,t is the carbon emission intensity of distribution line l at time t; is the carbon emission associated with the power loss along the distribution line l at time t; is the indirect carbon emission of node b at time t; is the direct carbon emission of node b at time t; is the total carbon emissions of node b at time t; is the amount of carbon emission permits purchased from the external carbon emission market; W b,t The amount of carbon permits traded at node b at time t; where formulas (12a)-(12b) represent the active and reactive power balance of the node; formulas (12c)-(12d) constrain the bidirectional branch power flow; formula (12e) represents the branch voltage drop; formula (12f) represents the power flow through second-order cone relaxation; formula (12g) represents the node voltage range; formula (12h) converts P b,t and Q b,t and power factor κ b Coupling; Formula (12i) limits Q b,trange; formula (12j) indicates CE balance, where the right hand indicates the total CE from the CE unit; and the left hand divides the total CE into three parts: the CE of the electricity seller in formula (12j-1), the CE of the electricity buyer in formula (12j-2), and the CE associated with the branch power loss in formula (12j-3); formula (12k) calculates the CE associated with the branch power loss; formula (12l) calculates the node CE intensity by the weighted average CE of all injected powers; formula (12m) defines the carbon intensity of branch l as the carbon intensity of the node flowing out of branch l; formulas (12n)-(12o) calculate the indirect CE of electricity sellers and buyers; formula (12p) adds the indirect CE and direct CE to obtain the total CE; formula (12q) indicates that the CE of bus B should always be lower than its carbon permit.

[0205] Establish a partially observable Markov decision process with discrete time steps to manage DER’s electricity carbon market bid / ask as a sequential decision process;

[0206] Based on the Markov decision process, each DER performs local training in the federated reinforcement learning model, and each DER makes its own electricity-carbon market trading decisions through a deep Q-network.

[0207] According to the types of electricity-carbon market participants, 11 agents are set up, including subways, aggregated EVs, PVs, three CE units, and five load aggregators, namely IZ, CZ, RZ, PZ, and AZ. The partially observable Markov decision process of the discrete time step is implemented by<N,S,O,A,T,R,γ> ,Define, N agents, state set S, private observation set O, action set A, reward function set R, state transition function T and discount factor γ;

[0208] State Sets t ={o 1,t ,o 2,t ,…,o N,t}∈S includes partial observations of all agents at time t. According to the DER type, the local observation is described as:

[0209] subway:

[0210]

[0211] Where: v b,t is the voltage amplitude of node b at time t; ρ l,t is the carbon emission intensity of distribution line l at time t; The initial carbon permit of node b at time t; [v b,t ,ρ b,t ] is the operating characteristic of the distribution network; For transaction information; are the operating parameters of the subway, here, express Up / down time; Indicates that at time t To ISOE set up between stations;

[0212] Other electrocarbon players:

[0213]

[0214] Action Set A t ={a 1,t ,a 2,t …,a i,t ,…,a N,t} is the action of all agents at time t, defined as:

[0215] subway:

[0216]

[0217] Where: P b,t is the electricity trading volume of node b at time t; W b,t The trading carbon allowance of node b at time t; P b,t =P b1,t +P b2,t +…+P bN,t , carbon permit trading volume W b,t =W b1,t +W b2,t +…+W bN,t , and the running time and

[0218] Other electrocarbon players:

[0219] a i,t =[P b,t ,W b,t ]∈Α (16)

[0220] State transition T: through function s t+1 =T(s t ,a 1:N,t, ) to achieve the environment state s t With action a 1:N,t associated; specifically, the state The conversion is performed by the action [P b,t ,W b,t ] is determined as solved in the electricity-carbon market interaction model in (12). , the price By using marginal theory (12) to solve. In addition, the state of the subway By constraints (1c)-(1d) and (1f) the action For the state ISOEt, it is done through an internal optimization model that aims to minimize the ISOE fluctuations from the first To The inter-station section, which is implemented by a portable commercial software;

[0221] Reward function set R: The personalized target of DER calculated according to the obtained actions. The reward function of all DERs has the same composition, but is designed to have specific connotations to suit practical applications, which is expressed as follows:

[0222]

[0223] In the formula, represents the carbon tax price, which is determined by the carbon intensity ρ b,t and the product of carbon tax κ; represents the reward function of DER i, transactional utility that reflects personalized goals; for the cost of carbon emissions; Cost of network usage; is the cost of bilateral carbon trading.

[0224] Specifically, the transaction utility Modeled as a quadratic function:

[0225]

[0226] Where, for subway and aggregated electric vehicles: Indicates energy saving, and are all normal numbers, indicating (P i,t ) and P i,t There is a clear monotonic relationship between them;

[0227] For PV: represents PV consumption, where and are all negative constants;

[0228] For carbon emission units: represents the cost of electricity generation, where and They are all normal numbers;

[0229] For load aggregators: represents the negative utility of expected failure in electricity consumption, where and All are normal numbers;

[0230] Bilateral carbon trading costs Modeled as a linear function of carbon permits, it is used to reflect transaction reputation, transaction scale, etc.:

[0231]

[0232] Where: is the bilateral carbon trading cost; W im,t is the carbon emission trading volume between two carbon trading entities i and m at time t. Based on the Markov decision process, each DER can perform local training in FRL to achieve personalized goals during electricity-carbon market transactions. Here, the motivation behind FRL in electricity-carbon market transactions is mainly twofold. First, compared with the communication overhead and scalability issues of existing multi-agent reinforcement learning algorithms, FRL has higher performance due to the requirement of only periodic updates and the distribution of the training process across multiple agents. Second, unlike the centralized training integrated in multi-agent RL that relies on collecting all local information from DERs, FRL allows data privacy to be protected by retaining their local data and only sharing model updates. Therefore, FRL is applied to the coupled electricity-carbon market considering heterogeneous DERs with differentiated goals. The training process of the federated reinforcement learning model includes global aggregation of updated local models and local model training using an actor-critic network;

[0233] Denote I as the set of DERs and Di as the dataset, where each DER trains its own local model. The model is then passed to the electricity carbon market coordinator to determine the global model It is represented by w a =f(w1,w2,…,w I ), and then the global model w a Broadcast to all DERs for local updates, achieving the differentiated goals of heterogeneous DERs in a self-decision-making and privacy-preserving manner.

[0234] Specifically, WFTD3 is used as a reinforcement learning algorithm for each DER to train its local model in the electricity carbon market transaction. The algorithm introduces a weighted mean field to improve the overall performance of the traditional version TD 3. In WFTD 3, each DER has an actor-critic framework, including an actor network and two critic networks with parameters μ θi The DER i actor network Acting in a decentralized manner, the local observations o i,t As input, and produces a deterministic action a i,t , which is then passed to the critic network Qθi,1 ,Q θ_i,2 , to output Q-value estimates for action evaluation; incorporated into the target actor network in WFTD 3 and the target critic network (Q θ’i,1 ,Q θ’i,2 ), its soft update formula is:

[0235]

[0236] Where τ<<1 is the target update factor, the critic network is updated via temporal difference learning, and the actor network is updated via deterministic policy gradient:

[0237]

[0238] In order to further improve the learning performance, the concept of neighbor agents is introduced. In the electric carbon market, internal type agents, such as different CE units, are considered to be neighboring agents. These neighbor agents with similar electric carbon market trading patterns can interact with each other on trading information to make more informed electric carbon market trading decisions. The weighted mean field technique is used to describe the impact of neighbor agents on the behavior of the i-th DER:

[0239]

[0240] Where: θ i is the network parameter; E is the sample size; L(θ i ) is the loss function; is the target Q value at time t; γ is the discount factor; s t is the state at time t; For actor networks in s t Actions generated under the state; Q value predicted by the Critic network; The reward obtained by the agent from the environment after performing an action; It is the optimization goal of the Actor network; is the gradient of the parameter; is the state at time t; represents the weighted action of the i-th DER, j∈j∈N(i) represents the neighbor agent of the i-th DER, ζ a_i,j represents a slight disturbance, and then Taylor's theorem is used to derive the pairwise Q value function Q θ_i (s,a i ,a j ):

[0241]

[0242] In the formula, due to ∑ j ζa_i,j = 0, the first-order term can be eliminated; the second term, as the remainder of the Taylor polynomial

[0243] +δ i,j ζ a_i,j andδ i,j ∈[0,1], when Q θ_i (s,a i ,a j ) is M-smooth, we can get Specifically, meta-functions and time-varying control variants are used to coordinate the local models trained in WFTD 3. Global Model of Hedian Carbon Market Coordinator Its main components are meta-functions and time-varying control variants;

[0244] Unlike traditional FL which seeks to find a model that works for all scenarios, the meta-function aims to find an initialization that has good performance after updating wrt. new scenarios, possibly through local gradient updates of only one or a few steps, and the function is formulated as:

[0245]

[0246] In the formula, w is the network parameter; β represents the step size; is the gradient of the meta-function; I is the sample size. The advantage of formula (23) is that it not only maintains the framework of traditional FL, but also captures the differences between differentiated DERs, even the differences between DERs within the same type, because each DER can use the solution of (23) as the initial point and slightly update it to perform well for its own dataset;

[0247] The metafunction F(w) is composed of the metafunctions F1(w), F2(w), ..., F I The average value of (w) is expressed as follows, where the function of the i-th DER is defined as:

[0248]

[0249] To update the local model, the gradient of formula (20) is calculated That is, the loss function f i The second order of (w) is calculated as follows:

[0250]

[0251] Since solving for the gradient can be computationally expensive, we further develop unbiased estimates:

[0252]

[0253] Where: is an unbiased estimate; |D i | indicates D i The size of l i (w; x, y) is the error of the evaluation model w when predicting the true label L based on the input data d; Through its unbiased estimate To calculate, through independent batches and Further solution:

[0254]

[0255] In addition, in order to correct the “client drift” problem in traditional FL, a random control variable c is proposed for each DER. i , where the average control variable c = ∑c is initialized for the electricity carbon market coordinator i / I, using the obtained Local model w i The update can be defined as:

[0256]

[0257] Then, the random control variable c is also updated via the calculated gradient i ,

[0258] c i '←(c i -c)+(ww i ) / Kη l (29) Using formula (24) and formula (25), the local updates are then aggregated to update the parameters of the electricity carbon market coordinator,

[0259]

[0260] Through formulas (23)-(30), a round of communication and update between DER and the electricity carbon market coordinator is completed.

[0261] The proposed electricity-carbon market mechanism is tested on a real 53-node distribution network in a city in China. Figure 5 As shown in Figure 1, the DERs integrated in the distribution network include subways, aggregated EVs, PVs, CE units (i.e., GT, DG, and CCS-GT units), and load aggregators (IZ, CZ, RZ, PZ, SZ). The specific parameters of these DERs are shown in Table I.

[0262] Table 1

[0263]

[0264]

[0265] The minimum / maximum charging rates for electric vehicles are 0kW and 8kW respectively, and the subway operates from 6am to 20pm. The line length is 3000m, the OESD capacity is 8.3kWh, the operating time window is 110-210s, and the stop time between each station is 40s.

[0266] In order to demonstrate the advantages of the proposed electricity-carbon market mechanism, the traditional system-centric mechanism and peer-centric mechanism are compared. The differences between the three markets are as follows:

[0267] 1) Type 1: System-centered mechanism: DSOs jointly clear the electricity carbon market in a centralized manner, aiming to minimize the total cost of all DERs and the operating costs of DSOs. DERs need to share private electricity carbon market transaction data with DSOs without considering differentiated goals such as pooled markets.

[0268] 2) Type 2: Peer-to-peer mechanism: The carbon market is cleared in a decentralized manner, allowing DERs to have differentiated goals to make their own decisions. Each DER needs to share private carbon market transaction data with the DSO to obtain the carbon market price, similar to the work in

[23] .

[0269] 3) Type 3: Hybrid mechanism (proposed): The newly added carbon market coordinator (in actual application, the carbon market industry association or trading center) coordinates carbon market transactions by aggregating implicit carbon market parameters, and then each DER obtains the shared model from the carbon market coordinator and makes local trading decisions based on its personalized goals. Each DER needs to share private carbon market transaction data with the DSO (profit neutral and independent of the carbon market) to obtain carbon market prices.

[0270] Figure 6-Figure 9 Four transaction cost items of five representative DERs under three electricity-carbon market frameworks are compared. Table I gives the clearing results. Figure 6-Figure 9 It can be seen that the system-centric framework has the highest transaction cost of the electricity-carbon market due to the myopia of the user side of the electricity-carbon market. In the peer-centric framework, the overall level of transaction costs is significantly reduced compared to the system-centric framework. In addition, in the hybrid electricity-carbon market framework, since each DER is indirectly affected by the E&C trading decisions of other DERs through the shared model of the electricity-carbon market coordinator, the transaction cost is higher than that of the peer-centric framework, but still lower than that of the system-centric framework.

[0271] In order to further verify the effectiveness of the proposed electricity-carbon market framework, the E&C transaction results under the three electricity-carbon market mechanisms mentioned above are as follows: Figure 10-15 As shown. Figure 10-12It can be seen that the electricity trading results of the three electricity markets have similar compositions. Specifically, internal power generation units, i.e., PV and CE units, play a leading role as electricity sellers. The significant difference lies in the reliance on electricity purchased from the external electricity market, where type 1 shows the lowest degree of reliance, type 2 the highest, and type 3 occupies an intermediate position. In addition, from Figure 13-Figure 15 It can be seen that the carbon trading results of the three electricity-carbon markets have similar spatial distributions. Carbon emission rights buyers are mainly distributed in IZ and some buses equipped with CE devices, while carbon permit sellers are mainly distributed in RZ, PZ and SZ. In particular, what distinguishes the three electricity-carbon markets is the purchase of carbon permits from external carbon markets. Similar to electricity trading, carbon trading in type 1 shows the lightest level of carbon permit purchases, type 2 the heaviest, and type 3 falls in between. Therefore, the results of carbon-electricity trading further prove this point.

[0272] In order to demonstrate the necessity and advantages of combining direct carbon emissions with carbon emissions trading, two cases are designed:

[0273] Case 1: Direct carbon emissions are included in the carbon emission responsibility of carbon reduction trading at the distribution level;

[0274] Case 2: Direct carbon emissions are cleared by the external carbon market and excluded from the carbon emission obligations of emission reduction carbon trading at the distribution level.

[0275] Figure 16-Figure 19 The results of electricity carbon market transactions in these two cases are shown. Table 2 lists the economic benefits of the two cases.

[0276] Table 2

[0277]

[0278] from Fig.16 and Fig.17 It can be observed that the PV consumption in Case 1 is greatly increased compared to Case 2, while the generation of CE units is largely suppressed. Fig.18 and Fig.19 It can be seen that the important seller of bilateral carbon trading shifts from RA in case 2 to PV in case 1, which greatly increases the profitability of PV. In addition, Fig.18 , Fig.19From Table II, it can be inferred that the introduction of direct CE in Case 1 significantly increases the carbon trading volume of DER peers at the distribution level by 17.59tCO2, but reduces the total excess CE from the external carbon market by 40.75tCO2, compared with Case 2. This further leads to a reduction of excess CE costs by $343.44 and an increase of $336.05 in total social welfare. Therefore, considering direct CE is conducive to the decarbonization of the power market at the distribution level and is crucial to the economics of operation.

[0279] In order to highlight the training performance of the proposed pFedScv algorithm, three other personalized FL algorithms are used for comparison, namely,

[0280] 1) FedProx: adds a neighbor term to the local subproblem to handle data heterogeneity and enables merging a variable number of local works.

[0281] 2) FedBN: Batch normalization technology is applied to the client, where a client BN layer and an update layer are set up to address different degrees of data heterogeneity.

[0282] 3) FedAla: This algorithm adopts an adaptive local aggregation module to determine the initial point of the local agent by aggregating the global model and the local model into the local target.

[0283] Figure 20-24 The evolution of the episode rewards of the five aggregators over 500 episodes for the four algorithms is shown, where the solid line and the shaded area indicate the moving average over 10 episodes and the oscillation of the reward during the training process, respectively. It can be clearly seen that the proposed pFedScv algorithm outperforms the other three algorithms by converging to the highest reward level faster. Taking RZ as an example, the proposed pFedScv algorithm converges to an episode reward of 110.83, which is 25.14%, 66.55% and 43.86% higher than FedProx, FedBN and FedAla, respectively. Fig.25The interquartile range of the standard deviations of the four algorithms is given, where the median of the standard deviation of the proposed pFedScv algorithm is 74.07%, 87.24%, and 52.65% lower than that of the FedProx, FedBN, and FedAla algorithms, respectively. This fully verifies the advantage of the stable performance of the proposed pFedScv algorithm. Table III shows the computational time results of the four algorithms on 1000 sets, where the total running time of the proposed pFedScv algorithm is 32.93%, 29.03%, and 48.56% lower than that of the FedProx, FedBN, and FedAla algorithms, due to i) the neighbor terms in FedProx add additional computational complexity; ii) batch normalization of different data from different DERs makes it difficult for FedBN to converge; iii) aggregating global and local models in FedAla sacrifices computational time to improve its learning performance. In summary, compared with the FedProx, FedBN and FedAla algorithms, the pFedScv algorithm has overwhelming advantages in terms of fast convergence speed, high reward level, small reward deviation and short calculation time.

[0284] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the art can make many possible changes and modifications to the technical solution of the present invention by using the above-mentioned technical content without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Therefore, any changes, modifications, equivalent changes and modifications made to the above embodiments based on the technology of the present invention without departing from the content of the technical solution of the present invention belong to the protection scope of the present technical solution.

Claims

1. A user-side autonomous decision-making method for an electricity-carbon coupling market, characterized by: The method includes: Construct an electricity-carbon market trading model, which distributes carbon-electricity data to different DERs by coordinating carbon-electricity sensors through a carbon-electricity coordinator. DERs obtain shared models from the electricity-carbon market coordinator and perform local training, and then merge local training models to generate a global model; the DERs include a subway model, an aggregated electric vehicle model, a PV photovoltaic model, and a carbon emission unit model; establish a partially observable Markov decision process with discrete time steps to manage DERs' electricity-carbon market bids / asks as a sequential decision process; Based on the Markov decision process, each DER performs local training in the federated reinforcement learning model, and each DER makes its own electricity-carbon market trading decisions through local self-training.

2. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 2 is characterized by: The subway model is: K=uk,u∈{1,2,...,U} (1a) T i Low ≤T i ≤T i Up (1c) 0≤ISOE i ≤100% (1d) Where: u is the trip index of the subway line; k is the total number of subway stations within a single trip of the subway line; i is the index of the subway station section; T i is the running time of the subway in the i-th section; ISOE i is the initial energy state of the on-board energy storage device in the i-th segment; Formula (1a) represents the total number of subway stations in operation; Formula (1b) represents the total operating time from the first station to the terminal station; Formula (1c) limits the operating time between each station; Formula (1d) limits the operating ISOE of the OESD for each inter-station journey; Formulas (1e)-(1g) are the coordination of operating time and electricity carbon market trading cycle; T metro is the unit electricity carbon market transaction cycle; and is the running time between the first and last stations in the tth electric regulation transaction cycle; is the operation time covering the entire interval in the tth electricity transaction cycle, according to the The sum of the running time of the interval; the first station interval of the t-th trading cycle It is also the last station interval of the t-1th trading cycle The last station interval of the tth trading cycle It is the first station interval of the t+1th trading cycle Two adjacent trading cycles and The sum is equal to I S / I E The running time of the inter-station section; Formula (1h) represents the net electricity consumption of the subway during the t-th electricity carbon market trading period, which is represented by the function P(i,T i ,ISOE i )calculate: In the formula, a 1,i ,a 2,i ,a 3,i ,a 4,i , and a 5,i is a pre-calibrated constant coefficient; The aggregate electric vehicle model is: Where, formula (2a) represents the charging power of the aggregated EV Subject to pre-calibration interval [l EV,agg ,μ EV,agg ], through this pre-calibrated interval, the aggregated EV obtains the desired charging energy; the hyperparameter matrices H and J are used to describe the sequential charging space: In the formula, H i is the set of all permutations of rows containing i 1s and (Ti)0s Matrix; (1) i and (0) i The lengths of the values ​​0 and 1 are In addition, and Minimum / maximum charging power And every electric car also, and The minimum / maximum charging power of each individual electric vehicle and Calculation, the calculation is as follows: Where: μ EV,agg is the upper bound of charging power of aggregated electric vehicles; l EV,agg A lower bound for the charging power of aggregated electric vehicles; The upper limit of charging power for the nth electric vehicle; The lower limit of charging power for the nth electric vehicle; The PV photovoltaic model is: Where: P t PV is the photovoltaic power generation at time t; P t PV,min is the lower limit of photovoltaic power generation at time t; P t PV,max is the upper limit; min is the power factor; κ min Need to be maintained at [κ min Hysteresis, κ min ahead] range.

3. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 1 is characterized by: The carbon emission unit model is: W t al,dir =W t burn +W t m +W t PFC (7a) W t burn =∑ f 44NCV f FC f CC f OF f / 12 (7b) W t m =44NC c (1-S c -A c )P al,t △t / 12k (7c) W t PFC =(6500r CF4 +9200r C2F6 )∑ t ∑ l P lt / 1000k (7d) W t of,dir =3.667G t C f n+∑ i E i,t r i (9) IN t pep,dir =In t pep,burn +W t pep,pro +W t pep,cbn (10a) W t pp,burn =44∑ i AD i,t NCV i EF i OF i / 12 (10b) W t pp,pro =∑ r AD r,t CC r -AD ca,t CC ca -AD pa,t CC pa (10c) W t pep,cbn =AD ls,t EF ca PUR ca (10d) Where: μ g,t is the unit commitment variable of generator unit g at time t (1 for on, 0 for off); is the output of distributed generation g at time t; and are the lower and upper bounds of the output of distributed generation g at time t; and are the lower and upper bounds of the active ramp capacity of generator set g at time t; and is the minimum online time and offline time of generator set g at time t; W j is the carbon emission of the jth carbon-containing material; CC r is the carbon content in the carbon raw material; formula (4a) represents the active power generation limit; formula (4b) represents the active power ramp limit; formulas (4c)-(4d) represent the online time and minimum offline time constraints; is the base load of the dth load aggregator at time t; and is the minimum load and maximum load of the dth load aggregator at time t; and are the maximum and minimum demand responses of the dth load aggregator at time t; Formula (5a) limits the load of the dth load aggregator after the demand response; Formula (5b) limits the range of the demand response; Formula (5c) represents the load factor requirement; is the direct carbon emission of the cement plant, ρ cec represents the carbon emission factor; Indicates the output during combustion and clinkering; is the power consumption; a cec , b cec and c cec is a constant coefficient; Direct carbon emissions from aluminum smelters; Carbon emissions from burning fuel, Carbon emissions for raw material consumption; Carbon emissions for electrolytic aluminum production; f represents the fuel index; NCV f / FC f / OF f Represents the average lower heating value, net consumption and carbon oxidation rate of renewable fuels; CC f It represents the carbon content of renewable fuel per unit calorific value; P al,t Indicates the power consumption of the aluminum smelter; k, NC c , S c and A c is a constant parameter used to describe the raw material consumption in the production of electrolytic aluminum; ρ CF4 and ρ C2F6 represents the constant CE coefficient; is the direct cost benefit of the steel plant; co / ρ gas / ρ st Indicates the CE coefficient of coal, natural gas and other materials; It indicates the coal and gas needed for steel production; represents the output of steel and gas; is the direct carbon emissions of the refinery; G t Indicates the crude oil consumed by the refinery; C f represents the mass fraction of coke in the refined product; η represents the utilization efficiency; E i,t / ρ i represents the i-th energy and the corresponding CE coefficient; Direct carbon emissions from petrochemical plants; Carbon emissions from burning fossil fuels, Carbon emissions from consumption of carbon-containing raw materials, is the carbon emission from calcium carbonate decomposition; i∈{diesel, coke} represents the fossil fuel index, AD i,t / NCV i / EF i / OF i is a constant coefficient used to describe the consumption of the i-th fuel, r∈{coke, blue carbon, electrode paste} represents the index of carbon-containing raw materials; AD r,t Indicates the consumption of carbon-containing raw materials; AD ca,t / AD pa,t Indicates the production of carbide and purified ash; CC ca,t / CC pa,t Indicates the carbon content in carbides and purified ash; AD ls,t represents carbonate consumption; EF ca / PUR ca CE coefficient indicating carbonate and carbonate purity; is the direct carbon emission of the commercial center, A is the surface area of ​​the commercial building; ρ j is the CE coefficient of the jth carbonaceous material; d k / m k / ρ k is the average distance, weight of carbon-containing materials and carbon emission coefficient of the kth mode of transportation; ρ h / e h is the CE coefficient of the h-th energy source and the energy consumption of the demolition process; Y CM / T is the service life and daily operating hours.

4. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 1 is characterized by: The electricity carbon market transaction model is: Q b,t JP b,t κ b (12h) Where: B b is the dielectric constant of node b; G b is the conductance of node b; P b,t is the active power of node b at time t; Q b,t is the reactive power of node b at time t; S l,max is the maximum power flow of distribution line l; and is the active / reactive power flow of distribution line l at time t; a l,t and v b,t is the square current of distribution line l at time t and the square node voltage amplitude of node b at time t; ρ b,t is the carbon emission intensity of node b at time t; ρ l,t is the carbon emission intensity of distribution line l at time t; is the carbon emission associated with the power loss along the distribution line l at time t; is the indirect carbon emission of node b at time t; is the direct carbon emission of node b at time t; is the total carbon emissions of node b at time t; is the amount of carbon emission permits purchased from the external carbon emission market; W b,t The amount of carbon permits traded at node b at time t; where formulas (12a)-(12b) represent the active and reactive power balance of the node; formulas (12c)-(12d) constrain the bidirectional branch power flow; formula (12e) represents the branch voltage drop; formula (12f) represents the power flow through second-order cone relaxation; formula (12g) represents the node voltage range; formula (12h) converts P b,t and Q b,t and power factor κ b Coupling; Formula (12i) limits Q b,t range; formula (12j) indicates CE balance, where the right hand indicates the total CE from the CE unit; and the left hand divides the total CE into three parts: the CE of the electricity seller in formula (12j-1), the CE of the electricity buyer in formula (12j-2), and the CE associated with the branch power loss in formula (12j-3); formula (12k) calculates the CE related to the branch power loss; formula (12l) calculates the node CE intensity by the weighted average CE of all injected powers; formula (12m) defines the carbon intensity of branch l as the carbon intensity of the node flowing out of branch l; formulas (12n)-(12o) calculate the indirect CE of electricity sellers and buyers; formula (12p) adds the indirect CE and direct CE to obtain the total CE; formula (12q) indicates that the CE of bus B should always be lower than its carbon permit.

5. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 1 is characterized by: The partially observable Markov decision process of the discrete time step is given by<N,S,O,A,T,R,γ> ,Define, N agents, state set S, private observation set O, action set A, reward function set R, state transition function T and discount factor γ; State Sets t ={o 1,t ,o 2,t ,…,o N,t }∈S includes partial observations of all agents at time t. The local observation is described as: subway: Where: v b,t is the voltage amplitude of node b at time t; ρ l,t is the carbon emission intensity of distribution line l at time t; The initial carbon permit of node b at time t; [v b,t ,ρ b,t ] is the operating characteristic of the distribution network; For transaction information; are the operating parameters of the subway, here, express Up / down time; Indicates that at time t, To ISOE set up between stations; Other electrocarbon players: Action Set A t ={a 1,t ,a 2,t …,a i,t ,…,a N,t } is the action of all agents at time t, defined as: subway: Where: P b,t is the electricity trading volume of node b at time t; W b,t The trading carbon allowance of node b at time t; P b,t =P b1,t +P b2,t +…+P bN,t , carbon permit trading volume W b,t =W b1,t +W b2,t +…+W bN,t , and the running time and Other electrocarbon players: a i,t =[P b,t ,W b,t ]∈Α (16) State transition T: through function s t+1 =T(s t ,a 1:N,t ,) to achieve, the environment state s t With action a 1:N,t associated; Reward function set R: The personalized target of DER calculated according to the obtained actions. The reward function of all DERs has the same composition and is expressed as follows: Where: represents the carbon tax price, which is determined by the carbon intensity ρ b,t and the product of carbon tax κ; represents the reward function of DER i, transactional utility that reflects personalized goals; for the cost of carbon emissions; Cost of network usage; is the cost of bilateral carbon trading.

6. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 5 is characterized by: The transaction utility Modeled as a quadratic function: Where, for subway and aggregated electric vehicles: Indicates energy saving, and are all normal numbers, indicating (P i,t ) and P i,t There is a clear monotonic relationship between them; For PV: represents PV consumption, where and are all negative constants; For carbon emission units: represents the cost of electricity generation, where and They are all normal numbers; For load aggregators: represents the negative utility of expected failure in electricity consumption, where and All are normal numbers; Bilateral carbon trading costs Modeled as a linear function of carbon permits: Where: is the bilateral carbon trading cost; W im,t is the carbon emission trading volume between the two carbon trading entities i and m at time t.

7. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 6 is characterized by: The training process of the federated reinforcement learning model includes global aggregation of updated local models and an actor-critic network used for local model training; Denote I as the set of DERs, and D i Represented as a dataset, each DER trains its own local model The model is then passed to the electricity carbon market coordinator to determine the global model It is represented by w a =f(w1,w2,…,w I ), and then the global model w a Broadcast to all DERs for local updates, achieving the differentiated goals of heterogeneous DERs in a self-decision-making and privacy-preserving manner.

8. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 6 is characterized by: WFTD3 is used as a reinforcement learning algorithm for each DER to train its local model in the electricity carbon market transaction. In WFTD 3, each DER has an actor-critic framework, including an actor network and two critic networks with parameters μ θi DERi's actor network Acting in a decentralized manner, the local observations o i,t As input, and produces a deterministic action a i,t , which is then passed to the critic network Q θi,1 ,Q θ_i,2 , to output Q-value estimates for action evaluation; incorporated into the target actor network in WFTD 3 and the target critic network (Q θ’i,1 ,Q θ’i,2 ), its soft update formula is: Where τ<<1 is the target update factor, the critic network is updated via temporal difference learning, and the actor network is updated via deterministic policy gradient: The weighted mean field technique is used to describe the influence of neighbor agents on the behavior of the i-th DER: Where: θ i is the network parameter; E is the sample size; L(θ i ) is the loss function; is the target Q value at time t; γ is the discount factor; s t is the state at time t; For actor networks in s t Actions generated under the state; Q value predicted by the Critic network; The reward obtained by the agent from the environment after performing an action; It is the optimization goal of the Actor network; is the gradient of the parameter; is the state at time t; represents the weighted action of the i-th DER, j∈j∈N(i) represents the neighbor agent of the i-th DER, ζ a_i,j represents a slight disturbance, and then Taylor's theorem is used to derive the pairwise Q value function Q θ_i (s,a i ,a j ): In the formula, due to ∑ j ζ a_i,j = 0, the first-order term can be eliminated; the second term, as the remainder of the Taylor polynomial When Q θ_i (s,a i ,a j ) is M-smooth, we can get The final approximation of .

9. The method for autonomous decision-making on the user side of the electricity-carbon coupling market according to claim 6 is characterized by: Adopting meta-functions and time-varying control variants to coordinate local models trained in WFTD 3 Global Model of Hedian Carbon Market Coordinator It includes metafunctions and time-varying control variants; The metafunction is formulated as: Where: w is the network parameter; β represents the step size; is the gradient of the metafunction; I is the sample size; the metafunction F(w) is composed of the metafunctions F1(w), F2(w), ..., F I The average value of (w) is expressed as follows, where the function of the i-th DER is defined as: To update the local model, the gradient of formula (20) is calculated The calculation is as follows: Further development of unbiased estimation: Where: is an unbiased estimate; |D i | indicates D i The size of l i (w; x, y) is the error of the evaluation model w when predicting the true label L based on the input data d; Through its unbiased estimation To calculate, through independent batches and Further solution: For each DER, a random control variable c is proposed i , where the average control variable c = ∑c is initialized for the electricity carbon market coordinator i / I, using the obtained Local model w i The update can be defined as: Then, the random control variable c is also updated via the calculated gradient i , c′ i ←(c i -c)+(ww i ) / Kη l (29) Using equations (24) and (25), local updates are then aggregated to update the parameters of the electricity carbon market coordinator, Through formulas (23)-(30), a round of communication and update between DER and the electricity carbon market coordinator is completed.

Images