Multi-microgrid security collaboration and electricity-carbon transaction method and system driven by online measurement data

Abstract

The invention relates to an on-line measurement data driven multi-microgrid security collaboration and electric carbon transaction method and system. The method comprises the following steps: step 1, establishing an MMG security collaboration training framework based on on-line measurement data driven multi-agent federal deep reinforcement learning (FMADRL); step 2, establishing a dynamic pricing and trading method oriented to point-to-point electricity-carbon trading demand response between MMGs; 3, training a local double-delay depth deterministic strategy gradient TD3 electricity-carbon transaction model, and calculating the transaction demand quantity at each moment; and step 4, aggregating and generating a global collaboration model, thereby realizing experience sharing and security collaboration of MMG. According to the invention, security privacy protection in the collaboration process of each MG main body can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-microgrid safety collaboration, and relates to a method and system for multi-microgrid safety collaboration and electricity-carbon trading based on federated reinforcement learning and demand response, and in particular to a method and system for multi-microgrid safety collaboration and electricity-carbon trading based on federated reinforcement learning and demand response driven by online measurement data. Background Art

[0002] As the need to actively deepen the environmentally friendly development of the energy system and effectively respond to the global climate crisis and energy shortage becomes more urgent, the collaboration between multi-microgrids (MMGs) can achieve autonomous energy sharing and surplus and shortage, which is an important means to promote win-win cooperation among multiple entities, large-scale new energy consumption, and energy supply and demand balance. However, the existing technology still has many deficiencies in terms of security assurance in MMG collaboration, and cannot effectively respond to complex collaboration needs, especially in the scenario of electricity and carbon quota trading.

[0003] The existing security collaboration and trading methods lack effective protection for MMG user measurement data. The data transmission and storage in the collaboration process are vulnerable to attacks, and there is a risk of user privacy leakage. With the gradual digitization and onlineization of energy trading, the online measurement sensitive data collected by power users' electricity meters, mutual inductors, high-precision carbon meters and other equipment are widely used in the MMG collaboration and trading process, but the existing trading mechanism cannot provide sufficient privacy protection measures, making these data easy to be maliciously stolen or tampered with, posing a potential threat to users and trading systems.

[0004] Therefore, the present invention proposes a method and system for multi-microgrid security collaboration and electricity-carbon trading based on federated reinforcement learning and demand response driven by online measurement data.

[0005] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a method for multi-microgrid security collaboration and electricity-carbon trading based on federated reinforcement learning and demand response driven by online measurement data, which can achieve security and privacy protection in the collaboration process of various MG entities.

[0007] The present invention solves the practical problem by adopting the following technical solutions:

[0008] A multi-microgrid safety coordination and electricity-carbon trading method driven by online measurement data includes the following steps:

[0009] Step 1: Establish a MMG safety collaborative training framework based on multi-agent federated deep reinforcement learning FMADRL driven by online measurement data for multiple microgrids MMG in the power grid domain and user side;

[0010] Step 2: Establish a dynamic pricing and trading method for point-to-point electricity-carbon trading demand response between MMGs within the framework of Step 1;

[0011] Step 3: According to the dynamic pricing and trading method in step 2, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment;

[0012] Step 4: Train a multi-agent security collaboration model based on federated learning to ensure the privacy of electricity-carbon trading while aggregating and generating a global collaboration model to achieve experience sharing and security collaboration of MMG.

[0013] Moreover, the MMG safety collaborative training framework based on FMADRL established in step 1 includes three levels: MG physical layer, TD3 scheduling agent layer and central server layer, where:

[0014] (1) MG physical layer: including external upper grid energy supply, as well as multiple interconnected MG functional areas with discrete spatial distribution serving different multi-energy users such as residents, businesses and industries. They are respectively equipped with energy supply side consisting of photovoltaic and wind turbines, energy conversion side consisting of fuel cells, micro gas turbines and gas boilers, and energy storage devices consisting of electric energy storage (EES) and thermal energy storage (TES);

[0015] (2) TD3 dispatching agent layer: including multiple "one-to-one" agents that independently supervise and dispatch the operation of each MG. Based on the operating status of the responsible MG in each time slot, the response behavior of each adjustable unit is converted into a time-series Markov decision process, and the TD3 electricity-carbon trading model is trained locally.

[0016] (3) Central server layer: used to aggregate and update model parameters, perform dynamic pricing of P2P electricity-carbon trading, and perform collaborative training of multi-agent electricity-carbon trading models for all MG scheduling agents in the region after each communication training round.

[0017] Moreover, the specific steps of step 2 include:

[0018] (1) Calculate the hourly surplus of MG electricity production in each residential, commercial and industrial area and the balance of carbon quotas

[0019]

[0020]

[0021] In the formula, Indicates the real-time electrical load of MG; are the electric powers output by EES, FC, MT, PV and WG at time t respectively; They represent the power consumption of combined heat and power (CHP) and GB at time t. ( or ) and gas consumption The thermal power output; the marginal carbon emission factor c of the power distribution system and the gas grid at the distribution network node is used e,t 、c g,t Calculate the tradable carbon quota at this scheduling time Indicates the amount of electricity each MG purchases from the upper-level power grid; It is the annual carbon quota allocated free of charge to each MG according to the baseline method and is averaged on a daily basis.

[0022] (2) Calculate the net energy demand (including electricity and carbon quota) of the P2P electricity-carbon trading market at time t Energy buying and selling prices

[0023]

[0024]

[0025]

[0026] Among them, the electricity transaction price and the trading price of carbon quotas All through the P2P electricity carbon trading market The calculation is done. Transaction requirements Sales volume Exact match The local bid price and Ask Price Equal to the median price If the market energy deficit It is necessary to report to the higher-level electricity carbon market Buy to balance, local bid price Adjust upwards according to market demand, otherwise there will be excess energy The market surplus is expressed as Sell ​​to the upper-level carbon market and lower the local selling price

[0027] (3) During the transaction process, each MG agent conducts transactions through the secure collaborative framework constructed in step 1. Each MG agent performs homomorphic encryption on the transaction demand and surplus results calculated in step (1) and only uploads the model parameters instead of directly transmitting sensitive transaction data.

[0028] The central server adopts the pricing method (2) to perform dynamic transaction pricing based on the transaction demand model parameters of each MG without accessing the actual measurement data of each MG;

[0029] Moreover, the specific steps of step 3 include:

[0030] (1) Construct a Markov decision process;

[0031] The energy load of each MG functional area, the price signal of electricity-carbon trading, the output of renewable energy, the energy storage status and other measurement data factors are used as the learning environment state s i,t , by performing specific energy purchase, conversion and storage actions a i,t , and then switch to a new system energy storage and supply and demand state s i,t+1 , returns a corresponding action reward r i,t .

[0032] The TD3 agent solves the optimal electricity-carbon security trading demand through interactive trial and error and strategy evolution with the MMG real-time environment;

[0033] State space s i,t It reflects the spatiotemporal perception of the MG’s operating state at time t, and plays a key role in guiding the optimization strategy learning. The three MG functional areas contain the following state information:

[0034]

[0035] In the formula, the price signal It is the electricity-carbon price established based on step 2, including P2P electricity, carbon quota trading price and gas purchase price; They represent the real-time heat load of MG respectively; Represent the energy storage and discharge power of EES and TES respectively.

[0036] The MG agents in the three functional areas dynamically dispatch the operating power of energy equipment at time t, and their action space a i,t Defined as:

[0037]

[0038] Respectively represent the ratio of the power output of FC, MT and GB to their power upper limit. A value of 0 means no output. Represents EES / TES energy storage action, It stands for energy releasing action.

[0039] Based on the status i,t Execute action a i,t The reward function value r obtained after i,t Including its electricity carbon trading and gas costs Equipment operating costs And the thermal energy supply and demand balance constraint penalty If the MG agent fails to meet the heat supply and demand balance under the constraints of each component, it will be affected by the formula economic penalties.

[0040]

[0041]

[0042] In the formula, Indicates the amount of gas purchased by CHP and GB from the external gas network; [·] + / - =max / min{·,0}; are the operation and maintenance cost coefficients of CHP, GB and electric thermal energy storage equipment respectively; Indicates the gas consumption of CHP at time t Output power or Respectively represent electric thermal energy storage devices or The state at time t and t-1; is the energy loss rate; based on whether its heat demand can be met, whether there is energy waste, and whether the equipment exceeds the limit of ramping, etc., the heat supply and demand balance and operation constraint penalty coefficient are used to calculate the energy loss rate. Quantify the economic penalty for scheduling actions performed by the MG agent.

[0043] (2) TD3 Intelligent Electricity-Carbon Trading Network Training

[0044] Each MG agent uses an optimized scheduling model trained by the improved TD3 algorithm based on the Actor-Critic architecture, including a policy network actor and a pair of Q network critics to compensate for the Q value overestimation problem that is common in traditional DDPG algorithms. For the i-th MG at each time step t, the critic online network Batch sample B experience tuples (s) from the experience replay pool i,t ,a i,t ,r i,t ,s i,t+1), using two target critic networks to calculate the temporal difference error respectively and through The two target critic network TD gradients are propagated in the direction of , guiding the update of the corresponding online critic network. At the same time, the actor network performs policy enhancement based on this Q value and updates it through stochastic gradient descent:

[0045]

[0046]

[0047]

[0048] In the formula, is the online actor network φ i,t Based on the state-oriented actions selected by the deterministic behavior strategy μ, a small and finite random noise ε is added by using the target strategy smoothing regularization method to solve the problem of overestimation of the traditional critic network. is the online critic network θ i,t The target Q value calculated based on the state and action; The meaning of is the same as that of ; γ is the target Q value discount factor, J(·) represents the discounted cumulative reward function, α θ and α φ are the gradient descent learning rates of each network. Online critic network θ i,t With actor network φ i,t Perform a soft update in the following ways:

[0049]

[0050] In addition, double With φ i ' ,t+1 The network performs delayed soft updates in the following way to improve the optimal action search capability and operation stability of the TD3 scheduling agent, where τ is the soft update coefficient.

[0051]

[0052] (3) Solve the transaction demand;

[0053] In the TD3 model, the Actor network selects an optimal electricity-carbon trading action based on the current state, including the purchase and sale of electricity from the P2P electricity market or the external power grid, and the purchase and sale of carbon quotas from the carbon trading market. The Critic network evaluates the Q value of the action and provides feedback on the strategy. If the Q value is high, it means that the action is reasonable, and the agent will continue to choose similar actions in similar states in the future; if the Q value is low, the agent will try different actions to increase future rewards. The agent selects the appropriate electricity-carbon trading and energy scheduling strategy according to the state at each moment, evaluates the value of the action through the Critic network, and continuously optimizes its decision-making process. Ultimately, the agent can dynamically adjust the energy equipment and trading strategies of the microgrid to ensure that electricity-carbon trading is carried out in the best way while meeting the demand for heat and electricity.

[0054] Moreover, the specific steps of step 4 include:

[0055] (1) Total communication rounds N during initialization training e , learning rate η, number of MG agents i∈Ω;

[0056] (2) The local model of the i-th MG agent is defined as f i e , which performs TD3 local training to obtain model parameters The iterative update is performed as follows. Under the premise of ensuring privacy, each MG agent will not share its actual online measurement data, but only upload the trained model parameters. to the central server;

[0057]

[0058] (3) The central server aggregates the model parameters of all MG agents The weighted average training parameters are calculated by To build a global model This process avoids the leakage of private data and realizes experience sharing and secure collaborative optimization among different MG agents through parameter aggregation. Subsequently, the central server broadcasts the aggregated global model parameters to all MG agents participating in the training;

[0059]

[0060] (4) Global parameters that the MG agent will receive Replace the local parameters, i.e. In this safe collaborative process, each MG agent jointly improves its scheduling strategy through a shared global model. When the collaborative training of each MG agent reaches convergence, several global models f are generated.G , and repeat the above steps, ending the training phase after reaching the maximum number of rounds.

[0061] (5) After multiple rounds of parameter aggregation and federated learning training, a globally optimal collaborative model is output, which aggregates the local model parameters of all MG agents, including the knowledge and experience of each agent, in a secure manner without leaking privacy.

[0062] An online measurement data-driven multi-microgrid safety coordination and electricity carbon trading system, comprising:

[0063] A safety collaborative training framework establishment module is built for the grid domain and user-side multi-microgrid MMGs, and an MMG safety collaborative training framework based on multi-agent federated deep reinforcement learning FMADRL driven by online measurement data is established;

[0064] A module for establishing dynamic pricing and trading methods, which establishes dynamic pricing and trading methods for point-to-point electricity-carbon trading demand response between MMGs in the framework;

[0065] The local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model construction module, based on the dynamic pricing and trading method, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment;

[0066] The global collaborative model generation module trains a multi-agent security collaborative model based on federated learning. While ensuring the privacy of electricity-carbon trading, it aggregates and generates a global collaborative model to achieve MMG's experience sharing and security collaboration.

[0067] An online measurement data-driven multi-microgrid safety coordination and electricity carbon trading system, comprising:

[0068] A safety collaborative training framework establishment module is built for the grid domain and user-side multi-microgrid MMGs, and an MMG safety collaborative training framework based on multi-agent federated deep reinforcement learning FMADRL driven by online measurement data is established;

[0069] A module for establishing dynamic pricing and trading methods, which establishes dynamic pricing and trading methods for point-to-point electricity-carbon trading demand response between MMGs in the framework;

[0070] The local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model construction module, based on the dynamic pricing and trading method, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment;

[0071] The global collaborative model generation module trains a multi-agent security collaborative model based on federated learning. While ensuring the privacy of electricity-carbon trading, it aggregates and generates a global collaborative model to achieve MMG's experience sharing and security collaboration.

[0072] Advantages and beneficial effects of the present invention:

[0073] 1. The present invention proposes a method and system for secure collaboration and electricity-carbon trading of multi-microgrids based on federated reinforcement learning and demand response driven by online measurement data, so as to achieve security and privacy protection in the collaborative process of each microgrid (MG) subject. First, a heterogeneous MMG collaborative training framework based on multi-agent federated deep reinforcement learning privacy protection is established, which provides a basis for the subsequent collaborative interaction of electricity-carbon trading and online measurement data. Secondly, a P2P electricity-carbon trading market between MMGs is established in the framework, and dynamic pricing is driven by online measurement data to formulate attractive transaction prices that can respond to market demand, so as to achieve higher profit margins, encourage independent MG subjects to trade electricity and carbon quotas, and realize energy sharing and complementary surpluses and shortages. Subsequently, based on the online measurement data collected by the electric energy meters, mutual inductors, and high-precision carbon meters of local energy users of MMG, the local twin delayed deep deterministic policy gradient (TD3) electricity-carbon trading model is trained. With its excellent self-learning ability and environmental adaptability, the trading information contained in the massive online measurement data of heterogeneous energy load curves, time-of-use prices, weather conditions, etc. included in the multi-agents of MMG is fully mined. Finally, the multi-agent electricity-carbon trading model under the privacy protection of federated learning is trained. Each MG intelligent agent only needs to share the parameters of the locally trained TD3 electricity-carbon trading model without transmitting real data. By continuously updating and iterating the global aggregation model to optimize the strategy, the privacy protection effect of "data collaboration without leaving the local area" is achieved, and the secure collaboration of the measurement data in the MG user domain is realized.

[0074] 2. Aiming at the privacy and security issues existing in the multi-agent collaboration under the cross-domain interaction between the power grid and MG users, the present invention proposes a method and system for the safe collaboration and electricity-carbon trading of multi-microgrids driven by online measurement data and demand response, so as to realize the safety and privacy protection in the collaboration process of each MG subject. The proposed MMG collaborative training framework under the privacy protection of multi-agent federated deep reinforcement learning (FMADRL) enables each MG agent to only share the parameters of the TD3 electricity-carbon trading model trained locally without transmitting the real measurement data, and optimizes the strategy by continuously updating and iterating the global aggregation model to achieve the privacy protection effect of "data collaboration without leaving the local area", and realize the safe collaboration of the online measurement data in the MG user domain. In addition, the proposed electricity-carbon safety trading method breaks through the limitations of the traditional single expensive and high-carbon emission upper power grid power purchase, and can provide a highly attractive transaction price with a higher profit margin according to the dynamic pricing method of market demand response, and encourage independent MG subjects to trade electricity and carbon quotas, realize energy sharing and complementarity of surplus and shortage, and improve the low-carbon nature and safety win-win nature of the transaction economy in the collaborative process. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 A multi-agent federated deep reinforcement learning multi-microgrid safety collaborative training framework diagram established by the present invention based on online measurement data drive;

[0076] Figure 2 It is a schematic diagram of a single MG electricity-carbon trading model based on the TD3 algorithm of the present invention;

[0077] FIG3( a ) is a diagram showing the calculation result of the P2P electricity-carbon trading price based on demand response of the present invention - P2P trading electricity price;

[0078] FIG3( b ) is a diagram showing the calculation results of the P2P electricity-carbon trading price based on demand response of the present invention - P2P trading carbon price. DETAILED DESCRIPTION

[0079] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0080] A multi-microgrid safety coordination and electricity-carbon trading method driven by online measurement data includes the following steps:

[0081] Step 1: Establish a multi-agent federated deep reinforcement learning (FMADRL) MMG safety collaborative training framework based on online measurement data driven for the grid domain and user-side multi-microgrid (MMG);

[0082] The MMG safety collaborative training framework based on FMADRL established in step 1 is as follows: Figure 1 As shown, it includes three layers: MG physical layer, TD3 scheduling intelligent agent layer and central server layer, among which:

[0083] (1) MG physical layer: including external upper grid power supply and serving residents (Ω R ), Business (Ω C ) and industrial (Ω I ) and multiple interconnected MG functional areas with discrete spatial distribution, respectively configured with photovoltaic (PV) and wind power generation (WG) energy supply side, fuel cell (FC), micro gas turbine (MT) and gas boiler (GB) energy conversion side, and electric energy storage (EES) and thermal energy storage (TES) energy storage devices;

[0084] (2) TD3 dispatching agent layer: including multiple "one-to-one" agents that independently supervise and dispatch the operation of each MG. Based on the operating status of the responsible MG in each time slot, the response behavior of each adjustable unit is converted into a time-series Markov decision process, and the TD3 electricity-carbon trading model is trained locally.

[0085] (3) Central server layer: used to aggregate and update model parameters, perform P2P electricity-carbon trading dynamic pricing, and conduct collaborative training of multi-agent electricity-carbon trading models for all MG scheduling agents in the region after each communication training round, so as to achieve multi-agent collaboration and privacy data “not leaving the local area, available but not visible”;

[0086] Step 2: Establish a dynamic pricing and trading method for peer-to-peer (P2P) electricity-carbon trading demand response between MMGs within the framework of Step 1;

[0087] The specific method of step 2 is:

[0088] At the central server level in the MMG security collaboration framework of step 1, a dynamic pricing and trading method for P2P electricity-carbon trading demand response is established to encourage independent MG entities to trade electricity and carbon quotas with extremely attractive trading prices, and participate in energy sharing and surplus and shortage complementarity.

[0089] The specific steps of step 2 include:

[0090] (1) Calculate the hourly surplus of MG electricity production in each residential, commercial and industrial area and the balance of carbon quotas

[0091]

[0092]

[0093] In the formula, Indicates the real-time electrical load of MG; are the electric powers output by EES, FC, MT, PV and WG at time t respectively; They represent the power consumption of combined heat and power (CHP) and GB at time t. ( or ) and gas consumption The thermal power output; the marginal carbon emission factor c of the power distribution system and the gas grid at the distribution network node is used e,t 、c g,t Calculate the tradable carbon quota at this scheduling time Indicates the amount of electricity each MG purchases from the upper-level power grid; It is the annual carbon quota allocated free of charge to each MG according to the baseline method and is averaged on a daily basis.

[0094] (2) Calculate the net energy demand (including electricity and carbon quota) of the P2P electricity-carbon trading market at time t Energy buying and selling prices

[0095]

[0096]

[0097]

[0098] Among them, the electricity transaction price and the trading price of carbon quotas All through the P2P electricity carbon trading market The calculation is done. Transaction requirements Sales volume Exact match The local bid price and Ask Price Equal to the median price If the market energy deficit It is necessary to report to the higher-level electricity carbon market Buy to balance, local bid price Adjust upwards according to market demand, otherwise there will be excess energy The market surplus is expressed as Sell ​​to the upper-level carbon market and lower the local selling price

[0099] (3) Privacy protection and secure transactions

[0100] During the transaction process, each MG agent conducts transactions through the secure collaborative framework constructed in step 1; each MG agent performs homomorphic encryption on the transaction demand and surplus results calculated in (1), and only uploads the model parameters instead of directly transmitting sensitive transaction data.

[0101] This is achieved through the federated learning security collaboration mechanism in step 4. The central server adopts the pricing method (2) to perform dynamic transaction pricing based on the transaction demand model parameters of each MG without accessing the actual measurement data of each MG, thus ensuring transaction privacy.

[0102] Step 3: According to the dynamic pricing and trading method in step 2, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment;

[0103] Step 3 is to construct the TD3 dispatching agent layer in the MMG security collaboration framework of step 1. Based on the online measurement data collected by the electric energy meters, mutual inductors, and high-precision carbon meters of each local power user in the physical layer of step 1, MMG attributes its response behavior at time t to a time-series Markov decision process, trains the local TD3 electricity-carbon trading model, and calculates the transaction demand in the security collaboration process;

[0104] The specific steps of step 3 include:

[0105] (1) Construct a Markov decision process.

[0106] The energy load of each MG functional area, the price signal of electricity-carbon trading, the output of renewable energy, the energy storage status and other measurement data factors are used as the learning environment state s i,t , by performing specific energy purchase, conversion and storage actions a i,t, and then switch to a new system energy storage and supply and demand state s i,t+1 , returns a corresponding action reward r i,t .

[0107] The TD3 agent solves the optimal electricity-carbon security trading demand through interactive trial and error and strategy evolution with the MMG real-time environment;

[0108] State space s i,t It reflects the spatiotemporal perception of the MG’s operating state at time t, and plays a key role in guiding the optimization strategy learning. The three MG functional areas contain the following state information:

[0109]

[0110] In the formula, the price signal It is the electricity-carbon price established based on step 2, including P2P electricity, carbon quota trading price and gas purchase price; They represent the real-time heat load of MG respectively; Represent the energy storage and discharge power of EES and TES respectively.

[0111] The MG agents in the three functional areas dynamically dispatch the operating power of energy equipment at time t, and their action space a i,t Defined as:

[0112]

[0113] Respectively represent the ratio of the power output of FC, MT and GB to their power upper limit. A value of 0 means no output. Represents EES / TES energy storage action, It stands for energy releasing action.

[0114] The optimization goal of each MG intelligent body is to reduce the economic cost and CO2 emissions caused by scheduling and carbon quota trading as much as possible within a certain time scale, and formulate a reasonable electricity-carbon transaction; therefore, based on the state s i,t Execute action a i,t The reward function value r obtained after i,t Including its electricity carbon trading and gas costs Equipment operating costs And the thermal energy supply and demand balance constraint penalty Different from the net demand for electricity Due to the trading constraints, thermal energy cannot be bought and sold through the P2P electricity-carbon trading market. Therefore, if the MG intelligent body cannot meet the thermal energy supply and demand balance under the constraints of each component, it will be subject to the formula. economic penalties.

[0115]

[0116]

[0117] In the formula, Indicates the amount of gas purchased by CHP and GB from the external gas network; [·] + / - =max / min{·,0}; are the operation and maintenance cost coefficients of CHP, GB and electric thermal energy storage equipment respectively; Indicates the gas consumption of CHP at time t The output power ( or ); Respectively represent the electric thermal energy storage device ( or ) at time t and t-1; is the energy loss rate; based on whether its heat demand can be met, whether there is energy waste, and whether the equipment exceeds the limit of ramping, etc., the heat supply and demand balance and operation constraint penalty coefficient are used to calculate the energy loss rate. Quantify the economic penalty for scheduling actions performed by the MG agent.

[0118] (2) TD3 Intelligent Electricity-Carbon Trading Network Training

[0119] Each MG agent uses an optimized scheduling model trained by the improved TD3 algorithm based on the Actor-Critic architecture, such as Figure 2 The figure shows a policy network (actor) and a pair of Q networks (critic) to compensate for the Q value overestimation problem that is common in traditional DDPG algorithms. For the i-th MG at each time step t, the critic online network Batch sample B experience tuples (s) from the experience replay pool i,t ,a i,t ,r i,t ,s i,t+1 ), using two target critic networks to calculate the temporal difference error respectively and through The two target critic network TD gradients are propagated in the direction of , guiding the update of the corresponding online critic network. At the same time, the actor network performs policy enhancement based on this Q value and updates it through stochastic gradient descent:

[0120]

[0121]

[0122]

[0123] In the formula, is the online actor network φ i,t Based on the state-oriented actions selected by the deterministic behavior strategy μ, a small and finite random noise ε is added by using the target strategy smoothing regularization method to solve the problem of overestimation of the traditional critic network. is the online critic network θ i,t The target Q value calculated based on the state and action; The meaning of is the same as that of ; γ is the target Q value discount factor, J(·) represents the discounted cumulative reward function, α θ and α φ are the gradient descent learning rates of each network. Online critic network θ i,t With actor network φ i,t Perform a soft update in the following ways:

[0124]

[0125] In addition, double With φ i ' ,t+1 The network performs delayed soft updates in the following way to improve the optimal action search capability and operation stability of the TD3 scheduling agent, where τ is the soft update coefficient.

[0126]

[0127] (3) Solving the transaction demand

[0128] In the TD3 model, the Actor network selects an optimal electricity-carbon trading action based on the current state, which includes purchasing and selling electricity from the P2P electricity market or the external power grid, and purchasing and selling carbon quotas from the carbon trading market. The Critic network evaluates the Q value of the action and provides feedback on the strategy. If the Q value is high, it means that the action is reasonable, and the agent will continue to choose similar actions in similar states in the future; if the Q value is low, the agent will try different actions to increase future rewards. The agent selects the appropriate electricity-carbon trading and energy scheduling strategy according to the state at each moment, evaluates the value of the action through the Critic network, and continuously optimizes its decision-making process. Ultimately, the agent can dynamically adjust the energy equipment and trading strategies of the microgrid to ensure that electricity-carbon trading is carried out in the best way while meeting the demand for heat and electricity.

[0129] Step 4: Train a multi-agent security collaboration model based on federated learning to ensure the privacy of electricity-carbon trading while aggregating and generating a global collaboration model to achieve experience sharing and security collaboration of MMG;

[0130] The step 4 proposes a MMG security collaboration method based on the FMADRL algorithm, which is built on the security collaboration framework of step 1. Through the participation of the central server layer, it coordinates the TD3 scheduling layer constructed in step 3 and the independent MGs in the physical layer. It can replace the transmission of privacy data with the aggregation of TD3 electricity-carbon trading model parameters, and realizes the MMG online measurement data security collaboration, experience sharing and multi-subject secure electricity-carbon trading in the region under privacy protection.

[0131] The specific steps of step 4 include:

[0132] (1) Total communication rounds N during initialization training e , learning rate η, number of MG agents i∈Ω;

[0133] (2) The local model of the i-th MG agent is defined as f i e , which performs TD3 local training to obtain model parameters The iterative update is performed as follows. Under the premise of ensuring privacy, each MG agent will not share its actual online measurement data, but only upload the trained model parameters. to the central server;

[0134]

[0135] (3) The central server aggregates the model parameters of all MG agents The weighted average training parameters are calculated by To build a global model This process avoids the leakage of private data and realizes experience sharing and secure collaborative optimization among different MG agents through parameter aggregation. Subsequently, the central server broadcasts the aggregated global model parameters to all MG agents participating in the training;

[0136]

[0137] (4) Global parameters that the MG agent will receive Replace the local parameters, i.e. In this safe collaborative process, each MG agent jointly improves its scheduling strategy through a shared global model. When the collaborative training of each MG agent reaches convergence, several global models f are generated. G , and repeat the above steps, ending the training phase after reaching the maximum number of rounds.

[0138] (5) After multiple rounds of parameter aggregation and federated learning training, a globally optimal collaborative model is output, which aggregates the local model parameters of all MG agents in a secure way without leaking privacy, including the knowledge and experience of each agent, and achieves more accurate electricity-carbon trading and efficient and safe collaboration. In the future, the output global model can be adapted to advanced MMG collaborative applications such as optimized scheduling.

[0139] Through the above implementation steps, the transaction results of MMG multi-agent secure collaboration are shown in Figure 3(a) and Figure 3(b). Based on the demand electricity, time-of-use electricity price and carbon price reported by MMG in real time, the P2P trading market can formulate dynamic electricity-carbon trading prices according to market demand response. Under the federated learning framework of each MG, the local TD3 model parameters are transmitted to the central server layer in an encrypted manner to ensure that the private measurement data is not directly exchanged, avoiding the potential privacy leakage risk caused by data sharing. Secondly, the central server only aggregates and updates the local model parameters, and the training results of the global model are returned to each MG. This process does not involve the upload and sharing of the original data, effectively realizing the privacy protection of "data does not leave the local area". Finally, through multiple rounds of collaborative training, it is ensured that each microgrid can achieve secure model update and optimization under the premise of privacy protection, thereby improving the operational safety and reliability of the entire MMG system and effectively ensuring the collaborative security of MMG.

[0140] An online measurement data-driven multi-microgrid safety coordination and electricity carbon trading system, comprising:

[0141] A safety collaborative training framework establishment module is built for the grid domain and user-side multi-microgrid MMGs, and an MMG safety collaborative training framework based on multi-agent federated deep reinforcement learning FMADRL driven by online measurement data is established;

[0142] A module for establishing dynamic pricing and trading methods, which establishes dynamic pricing and trading methods for point-to-point electricity-carbon trading demand response between MMGs in the framework;

[0143] The local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model construction module, based on the dynamic pricing and trading method, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment;

[0144] The global collaborative model generation module trains a multi-agent security collaborative model based on federated learning. While ensuring the privacy of electricity-carbon trading, it aggregates and generates a global collaborative model to achieve MMG's experience sharing and security collaboration.

[0145] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A multi-microgrid safety coordination and electricity-carbon trading method driven by online measurement data, characterized by: The steps include: Step 1: Establish a MMG safety collaborative training framework based on multi-agent federated deep reinforcement learning FMADRL driven by online measurement data for multiple microgrids MMG in the power grid domain and user side; Step 2: Establish a dynamic pricing and trading method for point-to-point electricity-carbon trading demand response between MMGs within the framework of Step 1; Step 3: According to the dynamic pricing and trading method in step 2, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment; Step 4: Train a multi-agent security collaboration model based on federated learning to ensure the privacy of electricity-carbon trading while aggregating and generating a global collaboration model to achieve experience sharing and security collaboration of MMG.

2. According to the online measurement data driven multi-microgrid safety coordination and electricity-carbon trading method of claim 1, it is characterized by: The FMADRL-based MMG safety collaborative training framework established in step 1 includes three layers: MG physical layer, TD3 scheduling agent layer and central server layer, where: (1) MG physical layer: including external upper grid energy supply, as well as multiple interconnected MG functional areas with discrete spatial distribution serving different multi-energy users such as residents, businesses and industries. They are respectively equipped with energy supply side consisting of photovoltaic and wind turbines, energy conversion side consisting of fuel cells, micro gas turbines and gas boilers, and energy storage devices consisting of electric energy storage EES and thermal energy storage TES. (2) TD3 dispatching agent layer: including multiple "one-to-one" agents that independently supervise and dispatch the operation of each MG. Based on the operating status of the responsible MG in each time slot, the response behavior of each adjustable unit is converted into a time-series Markov decision process, and the TD3 electricity-carbon trading model is trained locally. (3) Central server layer: used to aggregate and update model parameters, perform dynamic pricing of P2P electricity-carbon trading, and perform collaborative training of multi-agent electricity-carbon trading models for all MG scheduling agents in the region after each communication training round.

3. According to the online measurement data driven multi-microgrid safety coordination and electricity-carbon trading method of claim 1, it is characterized by: The specific steps of step 2 include: (1) Calculate the hourly surplus of MG electricity production in each residential, commercial and industrial area and the balance of carbon quotas In the formula, Indicates the real-time electrical load of MG; are the electric powers output by EES, FC, MT, PV and WG at time t respectively; They represent the power consumption of CHP and GB of the combined heat and power unit at time t. Gas consumption The thermal power output; the marginal carbon emission factor c of the power distribution system and the gas grid at the distribution network node is used e,t 、c g,t Calculate the tradable carbon quota at this scheduling time Indicates the amount of electricity each MG purchases from the upper-level power grid; The annual carbon quota allocated free of charge to each MG according to the baseline method and averaged on a daily basis; (2) Calculate the net energy demand of the P2P electricity-carbon trading market at time t Energy buying and selling prices Among them, the electricity transaction price and the trading price of carbon quotas All through the P2P electricity carbon trading market The calculation is made; if Transaction requirements Sales volume Exact match The local bid price and Ask Price Equal to the median price If the market energy deficit It is necessary to report to the higher-level electricity carbon market Buy to balance, local bid price Adjust upwards according to market demand, otherwise there will be excess energy The market surplus is expressed as Sell ​​to the upper-level carbon market and lower the local selling price (3) During the transaction process, each MG agent conducts transactions through the secure collaborative framework constructed in step 1. Each MG agent performs homomorphic encryption on the transaction demand and surplus results calculated in step (1) and only uploads the model parameters instead of directly transmitting sensitive transaction data. The central server adopts the pricing method (2) to perform dynamic transaction pricing based on the transaction demand model parameters of each MG without accessing the actual measurement data of each MG.

4. According to the online measurement data driven multi-microgrid safety coordination and electricity-carbon trading method of claim 1, it is characterized by: The specific steps of step 3 include: (1) Construct a Markov decision process; The energy load of each MG functional area, the price signal of electricity-carbon trading, the output of renewable energy, the energy storage status and other measurement data factors are used as the learning environment state s i,t , by performing specific energy purchase, conversion and storage actions a i,t , and then switch to a new system energy storage and supply and demand state s i,t+1 , returns a corresponding action reward r i,t ; The TD3 agent solves the optimal electricity-carbon security trading demand through interactive trial and error and strategy evolution with the MMG real-time environment; State space s i,t It reflects the spatiotemporal perception of the MG’s operating state at time t, and plays a key role in guiding the optimization strategy learning. The three MG functional areas contain the following state information: In the formula, the price signal It is the electricity-carbon price established based on step 2, including P2P electricity, carbon quota trading price and gas purchase price; They represent the real-time heat load of MG respectively; They represent the energy storage and discharge power of EES and TES respectively; The MG agents in the three functional areas dynamically dispatch the operating power of energy equipment at time t, and their action space a i,t Defined as: Respectively represent the ratio of the power output of FC, MT and GB to their power upper limit. A value of 0 means no output. Represents EES / TES energy storage action, It represents the action of releasing energy; Based on the status i,t Execute action a i,t The reward function value r obtained after i,t Including its electricity carbon trading and gas costs Equipment operating costs And the thermal energy supply and demand balance constraint penalty If the MG agent fails to meet the heat supply and demand balance under the constraints of each component, it will be affected by the formula economic penalties; In the formula, Indicates the amount of gas purchased by CHP and GB from the external gas network; [·] + / - =max / min{·,0}; are the operation and maintenance cost coefficients of CHP, GB and electric thermal energy storage equipment respectively; Indicates the gas consumption of CHP at time t Output power or Respectively represent electric thermal energy storage devices or The state at time t and t-1; is the energy loss rate; based on whether its heat demand can be met, whether there is energy waste, and whether the equipment exceeds the limit of ramping, etc., the heat supply and demand balance and operation constraint penalty coefficient are used to calculate the energy loss rate. Quantify the economic penalty for the scheduling actions performed by the MG agent; (2) TD3 Intelligent Electricity-Carbon Trading Network Training Each MG agent adopts an optimized scheduling model trained by the improved TD3 algorithm based on the Actor-Critic architecture, including a policy network actor and a pair of Q network critics to compensate for the Q value overestimation problem that is common in the traditional DDPG algorithm; for the i-th MG at each time step t, the critic online network Batch sample B experience tuples (s) from the experience replay pool i,t ,a i,t ,r i,t ,s i,t+1 ), using two target critic networks to calculate the temporal difference error and through The two target critic network TD gradients are propagated in the direction of , guiding the update of the corresponding online critic network; at the same time, the actor network performs policy enhancement based on this Q value and updates it through stochastic gradient descent: In the formula, is the online actor network φ i,t Based on the action selected by the deterministic behavior strategy μ for the state, a small and finite random noise ε is added by smoothing regularization of the target strategy; is the online critic network θ i,t The target Q value calculated based on the state and action; The meaning of is the same as that of ; γ is the target Q value discount factor, J(·) represents the discounted cumulative reward function, α θ and α φ are the gradient descent learning rates of the respective networks; the online critic network θ i,t With actor network φ i,t Perform a soft update in the following ways: In addition, double and The network performs delayed soft updates in the following ways to improve the optimal action search capability and operation stability of the TD3 scheduling agent, where τ is the soft update coefficient; (3) Solve the transaction demand; In the TD3 model, the Actor network selects an optimal electricity-carbon trading action based on the current state, including purchasing and selling electricity from the P2P electricity market or external power grid, and purchasing and selling carbon quotas from the carbon trading market; the Critic network evaluates the Q value of the action and provides feedback on the strategy; if the Q value is high, it means that the action is reasonable, and the agent will continue to choose similar actions under similar conditions in the future; if the Q value is low, the agent will try different actions to increase future rewards; the agent selects appropriate electricity-carbon trading and energy scheduling strategies according to the state at each moment, evaluates the value of the action through the Critic network, and continuously optimizes its decision-making process; finally, the agent can dynamically adjust the energy equipment and trading strategies of the microgrid to ensure that electricity-carbon trading is carried out in the best way while meeting the heat and electricity needs.

5. According to the online measurement data driven multi-microgrid safety coordination and electricity-carbon trading method of claim 1, it is characterized by: The specific steps of step 4 include: (1) Total communication rounds N during initialization training e , learning rate η, number of MG agents i∈Ω; (2) The local model of the i-th MG agent is defined as It performs TD3 local training to obtain model parameters The iterative update is performed as follows; under the premise of ensuring privacy, each MG agent will not share its actual online measurement data, but only upload the trained model parameters to the central server; (3) The central server aggregates the model parameters of all MG agents The weighted average training parameters are calculated by To build a global model This process avoids the leakage of private data and realizes experience sharing and secure collaborative optimization among different MG agents through parameter aggregation. Subsequently, the central server broadcasts the aggregated global model parameters to all MG agents participating in the training. (4) Global parameters that the MG agent will receive Replace the local parameters, i.e. In this safe collaborative process, each MG agent jointly improves its scheduling strategy through a shared global model; when the collaborative training of each MG agent reaches convergence, several global models f are generated. G , and repeat the above steps, ending the training phase after reaching the maximum number of rounds; (5) After multiple rounds of parameter aggregation and federated learning training, a globally optimal collaborative model is output, which aggregates the local model parameters of all MG agents, including the knowledge and experience of each agent, in a secure manner without leaking privacy.

6. A multi-microgrid safety coordination and electricity carbon trading system driven by online measurement data, characterized by: include: A safety collaborative training framework establishment module is built for the grid domain and user-side multi-microgrid MMGs, and an MMG safety collaborative training framework based on multi-agent federated deep reinforcement learning FMADRL driven by online measurement data is established; A module for establishing dynamic pricing and trading methods, which establishes dynamic pricing and trading methods for point-to-point electricity-carbon trading demand response between MMGs in the framework; The local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model construction module, based on the dynamic pricing and trading method, the MMG system trains the local double-delayed deep deterministic policy gradient TD3 electricity-carbon trading model based on local online measurement data and calculates the trading demand at each moment; The global collaborative model generation module trains a multi-agent security collaborative model based on federated learning. While ensuring the privacy of electricity-carbon trading, it aggregates and generates a global collaborative model to achieve MMG's experience sharing and security collaboration.

Images