Microgrid cluster low-carbon optimal operation method based on improved MAPPO algorithm

Abstract

The micro-grid group low-carbon optimization operation method based on the improved MAPPO algorithm comprises the following steps: taking the total cost minimum in a single micro-grid operation period as an objective function, and constructing a micro-grid group optimization operation model; constructing a partially observable Markov decision process for the micro-grid optimization operation, establishing an observation space, an action space and a reward function corresponding to each micro-grid agent; introducing a nonlinear constraint of carbon management cost and a collaborative reward among micro-grids into the reward function, and introducing a negative sampling experience sharing framework into the MAPPO algorithm; embedding an LSTM network into the MAPPO algorithm, and designing a differentiated learning rate decay strategy to further improve the training speed of the LSTM-MAPPO model; and training the agent based on the improved MAPPO algorithm to obtain an optimal micro-grid group low-carbon optimization operation scheme. The optimization method can effectively reduce the operation cost of each micro-grid and reduce the system carbon emission.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of micro-grid group optimal operation, and particularly relates to a micro-grid group low-carbon optimal operation method based on an improved MAPPO algorithm. BACKGROUND

[0002] With the rapid growth of renewable energy and distributed energy, the power system is facing increasingly complex management challenges. Micro-grid group, as an intelligent system integrating multiple energy resources, can achieve efficient operation of the power system through optimal scheduling. However, the micro-grid group structure is complex, and how to realize the coordinated operation of multiple micro-grids in the system and reduce the system operation cost and carbon emissions has become a hot issue.

[0003] In the prior art, document [1]: "Micro-grid group energy optimization method based on improved bat algorithm" (Zeng Zhihui, Li Xueqiang, Yin Lulu, etc. Micro-grid group energy optimization method based on improved bat algorithm [J]. Electronic measurement technology, 2023, 46(10): 53-60.) aims to stabilize the operation of the micro-grid group and reduce the operation cost of the group to the greatest extent, and conducts research on the micro-grid group energy optimization based on the improved bat algorithm, thereby improving the economic benefit of the system. Document [2]: "Multi-energy micro-grid multi-objective optimal scheduling based on improved particle swarm algorithm" (Wang Yu, Hao Yi, Wang Lei, etc. Multi-energy micro-grid multi-objective optimal scheduling based on improved particle swarm algorithm [J]. Electrical measurement and instrumentation, 2023, 60(11): 29-36+59.) proposes a multi-energy micro-grid multi-objective optimal scheduling method based on the improved particle swarm method, thereby realizing flexible calling of internal energy of the multi-energy micro-grid and reducing the carbon emission pressure of the system.

[0004] However, the optimization method adopted in the above documents belongs to a heuristic algorithm, which is easy to fall into a local optimal solution and difficult to adapt to the complex dynamic environmental changes of multiple micro-grids.

[0005] Document [3]: "Micro-grid energy transaction based on multi-agent reinforcement learning" (Wei Guixi, Liu Xianggang, Chi Ming, etc. Micro-grid energy transaction based on multi-agent reinforcement learning [J]. Control engineering, 2023, 30(12): 2274-2279+2296.) proposes an energy transaction method based on multi-agent reinforcement learning to promote energy transactions between micro-grid users, thereby reducing the peak load of the micro-grid and the energy cost of users. However, the document adopts the traditional MAPPO algorithm to optimize the energy transaction between micro-grids, which makes it difficult to fully capture the time dependence when dealing with time series tasks, and the data utilization efficiency needs to be improved. SUMMARY

[0006] In view of the above, the application discloses a micro-grid group low-carbon optimization operation method based on an improved MAPPO algorithm.

[0007] The technical scheme adopted by the application is:

[0008] The micro-grid group low-carbon optimization operation method based on the improved MAPPO algorithm comprises the following steps:

[0009] Step 1: taking the minimum total cost in a single micro-grid operation cycle as an objective function, a micro-grid group optimization operation model is constructed;

[0010] Step 2: a partially observable Markov decision process for micro-grid optimization operation is constructed, and an observation space, an action space and a reward function corresponding to each micro-grid agent are established;

[0011] Step 3: a nonlinear constraint of carbon management cost and a collaborative reward between micro-grids are introduced into the reward function, so as to reduce carbon emissions of the micro-grid group and promote electric energy transaction between the micro-grids;

[0012] Step 4: in order to solve the problem of mutual dependence between agents and privacy protection, improve the training efficiency and sample utilization rate, a negative sampling experience sharing framework is introduced into the MAPPO algorithm;

[0013] Step 5: an LSTM network is embedded into the MAPPO algorithm, which helps the model to better capture the time sequence dependence to enhance the decision-making ability of the agent, and a differentiated learning rate decay strategy is designed to further improve the training speed of the LSTM-MAPPO model;

[0014] Step 6: the agent is trained based on the improved MAPPO algorithm, and an optimal micro-grid group low-carbon optimization operation scheme is obtained.

[0015] (1) Objective function:

[0016] In addition to the power generation cost of each distributed energy, the operation cost of the energy storage system and the electric energy power interaction cost, the application also introduces the carbon management cost in the objective function of each micro-grid to measure the carbon emission influence in the micro-grid operation, and the objective function is:

[0017] C i,MG =minC i,DG +C i,ESS +C i,cbn +C i,grid +C i,j ;

[0018] In the formula, C i,MG is the total cost of the micro-grid; C i,DGThe generation cost of each distributed power source in microgrid i; C i,ESS The operation cost of energy storage system; C i,cbn The carbon management cost of microgrid i; C i,grid The cost of power exchange between microgrid i and the upper-level grid; C i,j The cost of power exchange between microgrid i and microgrid j, and i≠j.

[0019] 1) Distributed power generation cost:

[0020] Each distributed power source in the microgrid includes wind power, photovoltaic, and gas turbine, and its cost is shown in the following formula:

[0021] C i,DG =C i,WT +C i,PV +C i,MT ;

[0022] C i,WT =C WT P i,WT ;

[0023] C i,PV =C PV P i,PV ;

[0024] C i,MT =a i P i,MT ;

[0025] In the formula, C i,WT , C i,PV , and C i,MT are the operation costs of wind turbine, photovoltaic, and gas turbine in microgrid i; K WT , K PV are the use and maintenance cost coefficients of wind turbine and photovoltaic system; a i is the cost coefficient of gas turbine in microgrid i; P i,WT , P i,PV , and P i,MT are the outputs of gas turbine, wind turbine, and photovoltaic system in microgrid i.

[0026] 2) Energy storage device operation cost:

[0027] The energy storage device can store energy when the power supply is surplus and release energy when the demand is high, helping the microgrid to achieve supply and demand balance, and its operation cost is shown in the following formula:

[0028] C i,ESS =C i,ESS (η i,on P i,on +η i,off Pi,off );

[0029] where K i,ESS is the energy storage cost coefficient of microgrid i; η i,on and η i,off are the charging and discharging efficiency of energy storage, respectively; P n,on and P n,off are the charging and discharging power of energy storage, respectively.

[0030] 3) Carbon governance cost:

[0031] By introducing the carbon governance cost, the microgrid group can be encouraged to optimize energy allocation, improve energy utilization efficiency, and promote new energy consumption. The carbon governance cost directly reflects the carbon emission of the system, and its calculation formula is as follows:

[0032] C i,cbn = Q i,D η t T-Q i,MG η T P;

[0033] Q i,D = Q i,WT + Q i,PV + Q i,MT + Q i,ESS ;

[0034] Q i,MG = Q i,D + Q i,B ;

[0035]

[0036] P i,DSG = P i,WT + P i,PV + P i,MT ;

[0037] where Q i,D is the carbon dioxide emission of microgrid i; η t is the carbon tax rate; T is the unit carbon emission tax; Q i,MG is the sum of carbon dioxide emissions generated by microgrid i and the carbon dioxide emissions caused by purchasing electricity from the upper grid; η T is the carbon dioxide emission reduction ratio; P is the transaction price; Q i,WT , Q i,PV , Q i,MT and Q i,ESS are the carbon dioxide emissions caused by wind power generation, photovoltaic power generation, gas turbine power generation and energy storage device power generation, respectively; Q i,B is the carbon dioxide emissions caused by microgrid i purchasing electricity from the upper grid; L i,BP is the total load demand in microgrid i; P i,ESS P is the power output of energy storage device; sgn is the sign function; P i,DSG P is the cumulative power output of all distributed generation devices without energy storage; D i,B P is the total amount of power purchased from the upper-level power grid; P i,WT P is the total amount of power purchased from the upper-level power grid; P i,PV P is the total amount of power purchased from the upper-level power grid; P i,MT P is the power output of wind turbine, photovoltaic turbine and micro gas turbine, respectively.

[0038] 4) Microgrid and upper-level power grid and inter-microgrid transaction cost:

[0039] The microgrid and the upper-level power grid and inter-microgrid may exist in the process of dispatching power transaction, and the transaction cost is as follows:

[0040] C i,grid = q gridbuy P i,gridbuy -q gridsell P i,gridsell ;

[0041]

[0042] In the formula, q gridbuy , q gridsell are the purchase and sale prices of the microgrid to the upper-level power grid; P i,gridbuy , P i,gridsell are the purchase and sale power of the microgrid i to the upper-level power grid; q MG is the transaction price between microgrids; P i,mbuy , P i,msell are the purchase and sale power of the microgrid i to the microgrid j.

[0043] (2) Constraint conditions:

[0044] In order to ensure the effectiveness and practicability of the microgrid group optimization dispatching model, each microgrid must comply with the following power constraint conditions to ensure that the microgrid system is technically feasible and economically efficient;

[0045] 1) Microgrid power balance constraint:

[0046]

[0047] 2) Upper and lower limit constraint of inter-microgrid power interaction:

[0048]

[0049] In the formula, P i,j , are the transaction power and maximum transaction power between microgrids, respectively.

[0050] 3) Upper and lower bounds of microturbine output:

[0051]

[0052] where, and are the upper and lower bounds of microturbine output, respectively.

[0053] 4) Ramp constraint of microturbine:

[0054]

[0055] where, are the output power of microturbine at time t and t-1, respectively; are the upper and lower bounds of output power change per unit time.

[0056] 5) Energy storage constraint:

[0057]

[0058] where, are the state of charge of energy storage at time t and t-1, respectively; u is the charge-discharge coefficient; are the upper and lower bounds of state of charge of energy storage at time t; are the charge-discharge power of energy storage at time t; are the upper bounds of charge-discharge power of energy storage at time t.

[0059] 6) Interaction power constraint between microgrid and upper grid:

[0060]

[0061] where, are the interaction power and upper bound of interaction power between microgrid i and upper grid.

[0062] 7) Trading price constraint:

[0063] In order to promote each microgrid in the microgrid cluster system to give priority to internal transactions, the price transaction between microgrids should be between the selling price and the buyback price of the upper grid. Thus, the trading price constraint can be obtained as:

[0064] q gridsell ≤q MG ≤q gridbuy ;

[0065] 8) Carbon emission constraint:

[0066]

[0067] where, is the maximum allowed carbon emission of device m in microgrid i at time period t; is the carbon emission of device m in microgrid i at time period t; is the total carbon emission of device m in a dispatch period; is the maximum allowed total carbon emission of device m in a dispatch period; is the total carbon emission of microgrid i in a dispatch period; is the maximum allowed total carbon emission of microgrid i at time period t; is the maximum allowed total carbon emission of microgrid i in a dispatch period. In step 2, the partial observable Markov decision process for the optimal operation of the microgrid is as follows:

[0068] The construction of the partial observable Markov decision process for the optimal operation of the power grid is actually a process of specifying what each microgrid agent can observe (observation space), what actions it can take (action space), and the reward (reward function) obtained according to the action. That is, the process of establishing the observation space, action space, and reward function corresponding to each microgrid agent. Specifically, it includes the following:

[0069] (1) Selection of observation space O:

[0070] The observation of microgrid i includes the load data at time t photovoltaic power generation wind power generation purchase and sale price of the upper-level power grid q grid , inter-microgrid trading electricity price q MG , and energy storage state of charge For the optimal dispatch of the microgrid group, the observation can be expressed as:

[0071]

[0072] wherein, represents the observation of microgrid agent i at time t;

[0073] (2) Selection of action space A:

[0074] The goal of the economic and low-carbon optimal dispatch of the microgrid group is to determine the optimal output of each unit and the electricity trading situation, so the output power of the micro gas turbine at time t energy storage charging and discharging power trading power between the microgrid and the upper-level network , and inter-microgrid trading power are taken as action values:

[0075]

[0076] wherein, a it represents the action of the microgrid agent i at time t;

[0077] (3) Reward function:

[0078] For the optimization scheduling problem of the microgrid group, the designed reward function needs to consider the constraints of system operation while reflecting the interests of each agent. Therefore, in addition to the cost function, the operation constraints of the energy storage device, the power exchange constraints with the upper-level grid, and the carbon emission constraints are added as penalty functions to the reward function, and the penalty terms are respectively represented as The reward function of each microgrid can be represented as:

[0079]

[0080] In the formula, r i MG represents the reward function of each microgrid agent; η i,ESS , η i,ex , and η i,co2 are penalty coefficients.

[0081] In step 3, the nonlinear penalty can ensure that when the carbon emissions are too high, the penalty will rapidly increase, thereby prompting the agent to make low-carbon decisions. The nonlinear function of carbon emission penalty is set as as follows:

[0082]

[0083] In the formula, α and β are coefficients for adjusting the intensity of the penalty.

[0084] The design of the collaborative reward mechanism can encourage microgrids to trade electricity among them to achieve the optimization of collaborative scheduling of the microgrid group. When two microgrids trade electricity, the reward can be calculated according to the traded electricity and the economic efficiency of the transaction. The collaborative reward R co between microgrids is set as follows:

[0085]

[0086] In the formula, γ co is the coefficient of the collaborative reward, controlling the strength of the reward; P ij is the amount of electricity traded between microgrid i and microgrid j; and △P ij is the economic efficiency of the electricity traded between microgrid i and microgrid j, i.e., the price difference of the transaction.

[0087] The complete reward function formula after combining the nonlinear penalty of carbon governance cost and the collaborative reward between microgrids based on the original reward function is:

[0088]

[0089] wherein, represents the reward function of each microgrid agent after introducing the nonlinear penalty of carbon governance cost and the collaborative reward among microgrids.

[0090] In step 4, in order to solve the problem of mutual dependence among agents and protect privacy, a parameter sharing framework is introduced. Under this framework, all agents contribute experience samples to a central shared experience pool for use by other agents. When an agent extracts samples from the experience pool for learning, it can benefit from the experience of other agents, thereby improving training efficiency and sample utilization. In addition, by introducing negative sampling in the shared experience pool, overfitting and local optimal solution can be effectively avoided. The steps include:

[0091] (1) Constructing a shared experience pool:

[0092] In a multi-agent environment, the experience sequence generated by each agent i is stored in a shared experience pool D:

[0093]

[0094] wherein, is the observation of agent i at time t; a i t represents the action of microgrid agent i at time t; is the reward obtained by agent i at time t; is the observation of agent i at time t+1.

[0095] (2) Negative sample judgment based on advantage function:

[0096] 1) Advantage function The calculation formula is:

[0097]

[0098] wherein, is the observation action value function of agent i, which represents the expected total return that agent i can obtain by taking action under the current observation . is the observation value function of agent i, which represents the expected return of agent i under the current observation .

[0099] 2) Negative sample judgment:

[0100]

[0101] wherein, B neg is the negative sample batch; B pos is the positive sample batch; θ(t) is a dynamically adjusted threshold; θmin is the initial threshold, usually set as a negative value; θ max is the maximum threshold, usually set as 0 or a positive value; t is the current training step; T is the total training steps.

[0102] θ(t) is set to a low value at the beginning to sample more negative samples; while at the later stage of training, the threshold is gradually increased to reduce the selection of negative samples, keeping the stability of training.

[0103] (3) Negative sample sampling:

[0104] After introducing negative samples, the negative sample sampling ratio p neg is set to control the proportion of negative samples in training to ensure that it does not bring too much interference to training. Then the final training sample batch B final is represented as:

[0105] B final = (1-p neg )·B pos +p neg ·B neg .

[0106] The step 5 includes:

[0107] 5.1: LSTM-MAPPO model:

[0108] In the problem of low-carbon optimal scheduling of microgrid clusters, the system has time sequence characteristics, that is, the decision of a certain microgrid agent not only depends on the current observation, but also is closely related to the historical observation, past decision and long-term behavior. For example, the charging and discharging strategy of the energy storage system and the start-stop decision of the gas turbine are often affected by previous operations. As a kind of recurrent neural network for processing time series data, LSTM (Long Short-Term Memory Network) can effectively capture this time sequence dependence. Therefore, introducing LSTM network into MAPPO algorithm can help the model better identify the time sequence relationship and improve the learning ability of multi-agent system.

[0109] The specific steps are as follows:

[0110] 1) LSTM network hidden state input:

[0111] In the traditional MAPPO algorithm, the observation of each agent is input into the policy network and value network as input. After introducing the LSTM network, the output of the LSTM network will be transmitted to the policy network as new observation information for decision making, so that each agent can consider the influence of historical observations when considering the current observation. This will affect the observation representation used in the advantage function and the objective function. Define the output of the LSTM network as

[0112]

[0113] In the formula, arrive It is the historical observation information of agent i at time k in the past; This represents the historical observation information of agent i at time t-k+1;

[0114] 2) Updating the advantage function and objective function:

[0115] In the traditional MAPPO algorithm, the advantage function is determined by the current observations. The advantage function is calculated based on the estimation of the value network. However, with the introduction of the LSTM network, the calculation of the advantage function will not only depend on the current observations. It also relies on the hidden state output of the LSTM network. To account for temporal dependencies, the updated advantage function and objective function are as follows:

[0116]

[0117] In the formula, To consider h t The subsequent advantage function; and Considering hidden states respectively The action value function and observation value function are given afterward; N represents the total number of samples; L clip (θ,h t ) is the objective function for embedding an LSTM network; p t (θ) is for considering hidden states Post-strategy ratio. ε represents the hyperparameter of the clipping amplitude, typically taken as 0.1 or 0.2; clip(p t (θ), 1-ε, 1+ε) represents the pruning operation, which is used to prevent the policy update from being too large, leading to training instability.

[0118] 5.2: Differentiated learning rate decay strategy:

[0119] To improve the training efficiency, stability, and generalization ability of LSTM in complex multi-agent systems, a differentiated learning rate decay strategy is introduced into the LSTM-MAPPO model. The tasks and learning requirements of the LSTM network and the MAPPO algorithm differ during joint training. To address this, different learning rate strategies can better meet their respective needs: the LSTM network part requires stable learning rate changes to accurately extract temporal features; while the MAPPO algorithm part requires faster policy updates to better adapt to the exploration and optimization process.

[0120] 1) Cosine restart decay:

[0121] The cosine restart learning rate decay strategy is introduced into the LSTM network to help the LSTM network adjust the learning rate periodically during training, thereby avoiding premature convergence and promoting exploration. Each cycle can be gradually increased as needed, so that the LSTM network can repeatedly "explore" and "converge" during training, avoiding premature falling into local optimal solution. The specific formula is as follows:

[0122]

[0123] In the formula, η LSTM (t) is the learning rate of the LSTM network at the t-th step; η max is the maximum learning rate of each cycle; η min is the minimum learning rate; T max is the maximum number of steps in a cycle; t is the current training step; and T restart is the starting time of each cycle.

[0124] 2) Linear decay:

[0125] The linear decay learning rate strategy is suitable for optimizing the learning rate of the MAPPO algorithm, helping to explore more in the early stage and gradually converge in the later stage to improve stability. The specific formula is as follows:

[0126]

[0127] In the formula, η MAPPO (t) is the learning rate of the MAPPO algorithm at the t-th step; η initial is the initial learning rate; T total is the total training step; and t is the current training step.

[0128] The step 6 includes the following steps:

[0129] S6.1: Initialize the low-carbon optimization operation model of the micro-grid group, define the observation space, action space, reward function and corresponding constraint conditions for each micro-grid agent.

[0130] S6.2: Each agent selects an action according to the current strategy, records the current observation, action and reward, stores these experiences into a shared experience pool, and balances exploration and utilization at the same time.

[0131] S6.3: When the agent samples the experience sample, a negative sampling method is used to optimize the diversity and representativeness of the training sample. S6.4: Use the LSTM-MAPPO model with a differentiated learning rate strategy. Calculate the advantage function and objective function of each microgrid agent. Calculate the policy gradient based on the objective function and update the policy of the microgrid agent to ensure the collaborative optimization of the policy. S6.5: Repeat S6.2-S6.4 until the performance of the LSTM-MAPPO model is stable or the maximum training round is reached.

[0132] S6.6: After training, the optimal policy is derived, offline verification is performed, and it is deployed to the low-carbon optimization scheduling system of the microgrid group to ensure effectiveness in the actual environment.

[0133] The microgrid group low-carbon optimization operation method based on the improved MAPPO algorithm has the following technical effects:

[0134] 1) The step 1 of the present application takes the minimization of the total cost in a single microgrid operation cycle as the objective function, and constructs a microgrid group optimization operation model. The model innovatively combines economic optimization with green development goals, not only clearly defines the optimization direction, but also provides key theoretical support and technical foundation for subsequent steps.

[0135] 2) The step 2 of the present application is to construct a partially observable Markov decision process for microgrid optimization operation, and establish the observation space, action space and reward function corresponding to each microgrid agent. This step clearly defines the state and behavior mapping relationship of the agent in the partially observable environment, so that the complex microgrid operation problem can be decomposed into a learnable multi-agent decision process. At the same time, this step lays a theoretical foundation for the construction of the MAPPO model, ensuring that the agent can make efficient learning and optimization decisions in a dynamic and incomplete information environment, and is an important prerequisite for the entire optimization algorithm.

[0136] 3) The step 3 of the present application introduces a nonlinear constraint of carbon governance cost and a collaborative reward among microgrids in the reward function to reduce carbon emissions of the microgrid group and promote electricity trading among microgrids. The nonlinear penalty mechanism ensures that when the carbon emissions are too high, the punishment will intensify rapidly, thereby effectively promoting the agent to give priority to low-carbon operation strategies in the decision-making process and actively reduce carbon emissions.

[0137] Meanwhile, a collaborative reward mechanism is designed to encourage energy trading between microgrids, thereby optimizing the collaborative scheduling of the microgrid group. This mechanism calculates appropriate reward values by analyzing the trading power between two microgrids, the trading economy, and the impact of trading behavior on the overall system efficiency, not only encouraging collaborative behavior among microgrids, but also improving energy utilization efficiency and reducing energy waste. Through the introduction of this double mechanism, the model's ability to balance between low-carbon goals and economic optimization is further enhanced, providing innovative support for the overall operation optimization of the microgrid group.

[0138] 4) The step 4 of the present application is to solve the problem of mutual dependence between agents and privacy protection, while improving the training efficiency and learning effect, and introduces a negative sampling experience sharing framework in the MAPPO algorithm. Specifically, through the mechanism of parameter sharing, the experience samples of all agents are collected into a central shared experience pool for use by other agents. This design not only enhances the collaboration ability between agents, but also fully taps the learning potential of multi-agent under the premise of ensuring privacy.

[0139] When the agent extracts samples from the shared experience pool for training, it can utilize the experience of other agents to make up for the lack of its own observation information, thereby significantly improving sample utilization and training efficiency. At the same time, by introducing a negative sampling mechanism in the shared experience pool, the overfitting of the model to suboptimal strategies is effectively avoided, reducing the risk of falling into a local optimal solution. This framework design fully balances the collaboration and independence between agents, providing key support for the stability and convergence of the optimization algorithm.

[0140] 5) The step 5 of the present application embeds the LSTM network into the MAPPO algorithm to help the model better capture the time series dependency to enhance the decision-making ability of the agent, and designs a differentiated learning rate decay strategy to further improve the training speed of LSTM-MAPPO.

[0141] In the problem of low-carbon optimization scheduling of microgrid groups, the system has obvious time series characteristics, that is, the decision of a certain microgrid agent not only depends on the current observation, but also is closely related to the historical observation, past decision and long-term behavior. For example, the charging and discharging strategy of the energy storage system and the start-stop decision of the gas turbine are often strongly influenced by previous operations. LSTM (Long Short-Term Memory Network) as a recurrent neural network specially designed to handle time series data can effectively capture this time series dependency, helping the model identify complex time-dependent relationships and enhance the learning ability of agents in dynamic environments. Therefore, embedding the LSTM network into the MAPPO algorithm can significantly improve the decision-making effect of multi-agent systems in complex time series scenarios.

[0142] In addition, to further improve the training efficiency, stability and generalization ability of the LSTM-MAPPO model in complex multi-agent systems, a differentiated learning rate decay strategy is proposed. Due to the differences in task requirements of LSTM and MAPPO in joint training, it is particularly important to design independent learning rate decay strategies for different parts. Specifically: LSTM part: requires more stable learning rate changes to ensure accurate extraction of time series features and avoid feature extraction errors caused by large fluctuations in learning rate.

[0143] MAPPO part: requires faster policy updates to speed up the exploration and optimization process of the environment, enabling the agent to quickly adapt to dynamic environments.

[0144] This differentiated learning rate strategy fully meets the different needs of LSTM and MAPPO in joint training, thereby improving the overall training efficiency and adaptability of the model, providing more efficient technical support for solving complex multi-agent optimization problems.

[0145] 6) The step 6 of the present application is based on training the agent by improving MAPPO to obtain an optimal micro-grid group optimal operation scheme. As the final key step, after the optimization and improvement of the MAPPO algorithm are completed, it needs to be trained comprehensively, and the complex dynamic behavior of the micro-grid group and the multi-agent cooperation mechanism are learned through the model, and finally the optimal operation strategy meeting the low-carbon target and economic requirement is output. This step ensures that the improved algorithm can effectively play a role in practical application, and provides a reliable solution for efficient and green operation of the micro-grid group. BRIEF DESCRIPTION OF DRAWINGS

[0146] The present application will be further described below in conjunction with the drawings and examples;

[0147] Figure 1 The optimization method flowchart of the present application.

[0148] Figure 2 The negative sampling shared experience pool structure diagram.

[0149] Figure 3 The LSTM-MAPPO framework structure diagram.

[0150] Figure 4 The reward convergence results of the three reinforcement learning algorithms.

[0151] Figure 5 The dispatching result of the micro-grid 1. DETAILED DESCRIPTION

[0152] The micro-grid group low-carbon optimal operation method based on the improved MAPPO algorithm, Figure 1The optimization method flowchart is provided in the application. Firstly, the application establishes a micro-grid optimization operation model with the minimum total cost in a single micro-grid operation cycle as the objective function; secondly, a partially observable Markov decision process for the optimization operation of the micro-grid is constructed, and the observation space, action space and reward function corresponding to each micro-grid agent are established; then, the nonlinear constraint of the carbon management cost and the collaborative reward among micro-grids are introduced into the reward function to reduce the carbon emissions of the micro-grid group and promote the electricity trading among the micro-grids; in order to solve the mutual dependence and privacy protection problems among the agents, improve the training efficiency and sample utilization rate, the negative sampling experience sharing framework is introduced into the MAPPO algorithm; the LSTM network is embedded into the MAPPO algorithm to help the model better capture the time sequence dependence and enhance the decision-making ability of the agent, and a differentiated learning rate decay strategy is designed to further improve the training speed of the LSTM-MAPPO model; finally, the agent is trained based on the improved MAPPO algorithm to obtain an optimal low-carbon optimization operation scheme of the micro-grid group. The optimization method provided in the application can effectively reduce the operation cost of each micro-grid and reduce the carbon emissions of the system. The improved MAPPO algorithm is not a simple LSTM-MAPPO model, and the application improves the MAPPO algorithm in three places, i.e., steps 3, 4 and 5, namely:

[0153] 1) The nonlinear constraint of the carbon management cost and the collaborative reward among micro-grids are introduced into the reward function to reduce the carbon emissions of the micro-grid group and promote the electricity trading among the micro-grids.

[0154] 2) In order to solve the mutual dependence and privacy protection problems among the agents, improve the training efficiency and sample utilization rate, the negative sampling experience sharing framework is introduced into the MAPPO algorithm.

[0155] 3) The LSTM network is embedded into the MAPPO algorithm to help the model better capture the time sequence dependence and enhance the decision-making ability of the agent, and a differentiated learning rate decay strategy is designed to further improve the training speed of the LSTM-MAPPO. Therefore, the improved MAPPO algorithm integrates the above three innovations.

[0156] Figure 2 The negative sampling shared experience pool structure is provided. As can be seen from Figure 2 , all agents share their experience samples into a central experience pool for all agents to access and use. When the agent extracts samples from the experience pool for learning, the negative sample sampling mechanism is introduced to selectively filter out those negative samples that are not beneficial or inefficient for strategy optimization, thereby effectively reducing unnecessary noise interference and improving the efficiency of the learning process and the quality of the training results. Through this mechanism, the agent can focus more on valuable experiences, accelerate convergence, and optimize the final strategy performance.

[0157] Figure 3 is the framework structure diagram of LSTM-MAPPO. Figure 3 Critic and Actor represent the policy network and action network of MAPPO respectively. After introducing the LSTM network, the LSTM network will input the current observation and the past observation into the MAPPO algorithm network, so that the model can better capture the time dependence and enhance the time perception ability of the agent in the decision-making process, and improve the performance and decision optimization of the model in the dynamic environment.

[0158] Figure 4 are the reward convergence results of three reinforcement learning algorithms. From Figure 4 , it can be seen that the improved MAPPO algorithm has a rapid growth in cumulative reward at the beginning of training, and tends to be stable after about 700 training rounds, finally stabilizing at about 0.95, indicating that the algorithm has achieved good learning effect and strategy optimization in training. Although compared with the PPO algorithm, the reward growth rate of the MAPPO algorithm is faster, and the final cumulative reward value is higher. However, both the MAPPO algorithm and the PPO algorithm have the problem of insufficient stability in the training process, and the final reward is much lower than the improved MAPPO algorithm. In summary, the improved MAPPO algorithm has the characteristics of fast convergence speed and high reward value, and has strong stability.

[0159] Figure 5 is the scheduling result of microgrid 1. In the 00:00-04:00 period, due to the low price of the power grid, the load demand of microgrid 1 is large, and at this time the new energy power generation is insufficient to meet its demand, so microgrid 1 tends to buy electricity from other microgrids or the upper distribution network, rather than scheduling the gas turbine to increase power generation. In the 05:00-07:00 period, microgrid 1 buys low-price electricity and stores it for subsequent use. In the 08:00-14:00 period, photovoltaic power generation increases, and load demand gradually increases, microgrid 1 increases output by scheduling gas turbines, and buys electricity from other microgrids to meet demand. In the 18:00-20:00 period, when the price is high, microgrid 1 chooses not to trade with the distribution network, but to discharge the electric energy storage device, increase power generation, and trade with other microgrids to maintain balance. In the scheduling process, the microgrid only buys electricity from the upper distribution network when other microgrids cannot meet its load. In addition, the three microgrids do not sell electricity to the distribution network throughout the scheduling process, but prefer to trade electricity among microgrids to improve economic efficiency.

[0160] Table 1 Comparison of optimization schemes

[0161]

[0162] Table 1 is an optimization scheme comparison. Three schemes are set to verify the effectiveness and superiority of the micro-grid group optimization scheduling method based on the improved MAPPO proposed in the application.

[0163] Scheme 1: micro-grid group optimization operation method based on PPO;

[0164] Scheme 2: micro-grid group optimization operation method based on MAPPO;

[0165] Scheme 3: micro-grid group optimization operation method based on the method proposed in the application.

[0166] The results show that the operation cost of each micro-grid in scheme 3 is the lowest, and the average carbon management cost of the micro-grid group is reduced by 13.4% and 25.2% respectively compared with schemes 1 and 2. This result proves that the proposed method not only can effectively reduce the operation cost of the micro-grid, but also can significantly reduce the carbon emission of the system.

Claims

1. A low-carbon optimized operation method for microgrid clusters based on an improved MAPPO algorithm, characterized in that... Includes the following steps: Step 1: Construct an optimal operation model for the microgrid cluster, with the objective function being the minimum total cost within a single microgrid operating cycle; Step 2: Construct a partially observable Markov decision process for the optimized operation of the microgrid, and establish the observation space, action space, and reward function for each microgrid agent; Step 3: Introduce nonlinear constraints on carbon governance costs and collaborative rewards among microgrids into the reward function; Step 4: Introduce a negative sampling experience sharing framework into the MAPPO algorithm; Step 5: Embed the LSTM network into the MAPPO algorithm and design a differentiated learning rate decay strategy to further improve the training speed of the LSTM-MAPPO model; Step 6: Train the agent based on the improved MAPPO algorithm to obtain the optimal low-carbon operation scheme for the microgrid cluster; Step 4 includes the following steps: (1) Construct a shared experience pool: In a multi-agent environment, each agent The generated experience sequences are stored in a shared experience pool. middle: ； In the formula, It is an intelligent agent At any moment Observations; Microgrid intelligent agents exist Actions at any given moment; It is an intelligent agent At any moment The rewards received; It is an intelligent agent At any moment Observations; (2) Negative sample judgment based on the dominance function: 1) Advantage function The calculation formula is as follows: ； In the formula, For intelligent agents The observed action value function represents the agent. In the current observation Take action below The expected total return that can be obtained; For intelligent agents The observation function represents the agent. In the current observation The expected return; 2) Negative sample judgment: ； ； In the formula, For negative sample batches; This is a positive sample batch; To dynamically adjust the threshold; This is the initial threshold, which is usually set to a negative value; The maximum threshold is set to 0 or a positive value; This represents the current number of training steps. Total training steps; By initially setting a lower threshold, more negative samples are sampled; in the later stages of training, as the strategy is gradually optimized, the threshold is gradually increased to reduce the selection of negative samples and maintain the stability of training. (3) Negative sample sampling: After introducing negative samples, set the negative sample sampling ratio. Controlling the proportion of negative samples in training ensures that they do not cause excessive interference to the training process; thus, the final training sample batch size is determined. Represented as: 。 2. The low-carbon optimized operation method for microgrid clusters based on the improved MAPPO algorithm according to claim 1, characterized in that: In step 1, the constructed microgrid cluster optimization operation model includes an objective function: In addition to the electricity production costs of each distributed energy source, the operating costs of the energy storage system, and the power exchange costs, carbon control costs are introduced into the objective function of each microgrid to measure the carbon emission impact of microgrid operation. The objective function is as follows: ； In the formula, This represents the total cost of the microgrid. For micro-network The generation cost of each distributed power source in China; For the operating costs of energy storage systems; microgrid The cost of carbon governance; For micro-network Power interaction cost with the upstream power grid; For micro-network With micro-network The interaction power cost, and ; 1) Distributed power generation cost: The distributed power sources in a microgrid include wind power, solar power, and gas turbines, and their costs are shown in the following formula: ； ； ； ； In the formula, , and microgrids The operating costs of wind turbine units, photovoltaic units, and gas turbines; , These are the operating and maintenance cost coefficients for wind turbines and photovoltaic systems, respectively. For micro-network The gas turbine cost coefficient; , and microgrids The output of the gas turbine, the output of the wind turbine, and the output of the photovoltaic system; 2) Operating costs of energy storage devices: The operating cost of an energy storage device is shown in the following formula: ； In the formula, For micro-network Cost coefficient of energy storage devices; , These are the charging and discharging efficiencies of the energy storage device; , These are the charging and discharging power of the energy storage device; 3) Carbon treatment costs: The cost of carbon remediation directly reflects the carbon emissions of the system, and its calculation formula is as follows: ； ； ； ； ； In the formula, For micro-network The amount of carbon dioxide emissions generated; The carbon tax collection rate; Taxes levied per unit of carbon emissions; For micro-network The sum of carbon dioxide emissions generated and carbon dioxide emissions from purchasing electricity from the upper-level power grid; This represents the percentage of carbon dioxide emission reductions. The transaction price; , , and These are the carbon dioxide emissions from wind power generation, photovoltaic power generation, gas turbine power generation, and energy storage device power generation, respectively. For micro-network Carbon dioxide emissions from purchasing electricity from the upper-level power grid; For micro-network Total load demand within the area; For the power generation output of energy storage devices; It is a symbolic function; This represents the cumulative power generation of all distributed generation devices excluding energy storage. The total amount of electricity purchased from the higher-level power grid; , and These are the outputs of wind turbines, photovoltaic units, and micro gas turbines, respectively. 4) Transaction costs between microgrids and the upper-level power grid, and between microgrids: During the dispatching process, power transactions may occur between microgrids and their upstream grid, as well as between microgrids themselves. The transaction costs are shown below: ； ； In the formula, , These are the electricity purchase and sales prices of the microgrid to the upstream power grid, respectively; , microgrids The amount of electricity purchased from and sold to the higher-level power grid; The electricity price for transactions between microgrids; , microgrids To Micro Network Electricity purchased and sold.

3. The low-carbon optimized operation method for microgrid clusters based on the improved MAPPO algorithm according to claim 2, characterized in that: The constraints of the objective function include: 1) Microgrid power balance constraints: ； 2) Upper and lower limits constraints for power interaction between microgrids: ； In the formula, , These represent the transaction power and maximum transaction power between microgrids, respectively. 3) Upper and lower limits of output constraints for micro gas turbines: ； In the formula, and These represent the upper and lower limits of the gas turbine's output, respectively. 4) Micro gas turbine ramping constraints: ； In the formula, , They are respectively Time and The output power of the gas turbine at any given time; , These represent the upper and lower limits of the change in unit output per unit time; 5) Constraints of energy storage devices: ； ； ； ； In the formula, , They are respectively Time and State of charge (SOC) that stores energy at all times; The charge / discharge coefficient; , for The upper and lower limits of the state of charge of the energy storage at any given time; , They are respectively The charging and discharging power of energy stored at all times; , They are respectively The upper limit of charging and discharging power for energy storage at all times; 6) Power constraints between microgrids and the upstream power grid: ； In the formula, , For micro-network The trading power and its upper limit with the upstream power grid; 7) Electricity trading price constraints: To encourage sub-microgrids within a microgrid cluster to prioritize intra-system transactions, the electricity price for inter-microgrid transactions should be guaranteed to fall between the upstream network's selling price and buyback price. This results in the following transaction price constraint: ； 8) Carbon emission constraints: ； ； ； ； In the formula, For micro-network medium equipment exist Maximum permissible carbon emissions for a given period; For micro-network medium equipment exist Carbon emissions over a given period; For a scheduling cycle of devices Total carbon emissions; For a scheduling cycle of devices Maximum permissible total carbon emissions; For micro-network Total carbon emissions within a scheduling cycle; for Time-based microgrid The maximum allowable total carbon emissions; For micro-network The maximum permissible carbon emissions within a scheduling cycle.

4. The low-carbon optimized operation method for microgrid clusters based on the improved MAPPO algorithm according to claim 2, characterized in that: In step 2, the partially observable Markov decision process for microgrid optimization operation includes the process of establishing the observation space, action space and reward function corresponding to each microgrid agent. Specifically, it includes: (1) Observation space Selection: microgrid The observations include Load data at any time Photovoltaic power generation Wind power generation capacity Electricity purchase and sale price of the superior power grid Microgrid inter-grid electricity price and energy storage state of charge For microgrid group optimal scheduling, its observation can be expressed as: ； In the formula, Microgrid intelligent agents exist Observation of time; (2) Action space Selection: The goal of microgrid cluster economic low-carbon optimized dispatch is to determine the optimal output and electricity trading situation for each generating unit, therefore... The output power of the micro gas turbine at any time Energy storage charging and discharging power Microgrid and its parent network transaction power and inter-micro-network transaction power As an action value: ； In the formula, Microgrid intelligent agents exist Actions at any given moment; (3) Reward function: For the optimal scheduling problem of microgrid clusters, the designed reward function must consider the constraints of system operation while reflecting the interests of each agent. Therefore, in addition to the cost function, constraints on energy storage device operation, power interaction with the upstream grid, and carbon emission constraints are added as penalty functions to the reward function. The penalty terms are expressed as follows: , , Then the reward function of each micronetwork is expressed as: ； In the formula, Represents the reward function for each micronet agent; , , All are penalty coefficients.

5. The low-carbon optimized operation method for microgrid clusters based on the improved MAPPO algorithm according to claim 4, characterized in that: In step 3, a nonlinear function for carbon emission penalties is set. as follows: ； In the formula, and It is a coefficient that adjusts the intensity of the punishment; When two microgrids engage in electricity trading, rewards can be calculated based on the amount of electricity traded and the economic viability of the trade. Set up inter-micronetwork collaboration rewards as follows: ； In the formula, It is the coefficient of the synergistic reward, which controls the intensity of the reward; It is a micro-network With micro-network Electricity trading volume between them; It is a micro-network Hewei.com The economics of trading electricity, i.e., the price difference in electricity trading; Based on the original reward function, the complete reward function formula, after incorporating the nonlinear penalty for carbon governance costs and the synergistic reward between microgrids, is as follows: ； In the formula, This represents the reward function for each microgrid agent after introducing a nonlinear penalty for carbon governance costs and a cooperative reward between microgrids.

6. The low-carbon optimized operation method for microgrid clusters based on the improved MAPPO algorithm according to claim 1, characterized in that: Step 5 includes: 5.1: LSTM-MAPPO model: The specific steps for introducing an LSTM network into the MAPPO algorithm are as follows: 1) Hidden input of LSTM network: In the MAPPO algorithm, the observations of each agent... The LSTM network is used as input to both the policy network and the value network; after its introduction, the output of the LSTM network is... The new observation information is passed to the policy network for decision-making, allowing each agent to consider the influence of historical observations when considering the current observation; this affects the observation representations used in the advantage function and objective function; the output of the LSTM network is defined as... : ； In the formula, arrive It's the past Time-based intelligent agent Historical observation information; Represents intelligent agents exist Historical observation information at any given moment; 2) Updating the advantage function and objective function: In the MAPPO algorithm, the advantage function is determined by the current observations. The advantage function is calculated based on the estimation of the value network; however, with the introduction of the LSTM network, the calculation of the advantage function will not only depend on the current observations. It also relies on the hidden state output of the LSTM network. To account for temporal dependencies, the updated advantage function and objective function are: ； ； In the formula, For consideration The subsequent advantage function; and Considering hidden states respectively The subsequent action value function and observation value function; Indicates the total number of samples; The objective function for embedding an LSTM network; To consider hidden states Post-strategy ratio; Hyperparameters representing the cropping range; Indicates a cropping operation; 5.2: A differentiated learning rate decay strategy was introduced into the LSTM-MAPPO model: 1) Cosine restart decay: A cosine restart learning rate decay strategy is introduced into the LSTM network to help the LSTM network periodically adjust the learning rate during training, thereby avoiding premature convergence and promoting exploration; the specific formula is as follows: ； In the formula, For LSTM networks in the first The learning rate of each step; The maximum learning rate for each cycle; Minimum learning rate; It is the maximum number of steps in a cycle; This represents the current number of training steps. The start time of each cycle; 2) Linear decay: A linear decaying learning rate strategy is suitable for optimizing the learning rate of the MAPPO algorithm, helping to explore in the early stages and gradually converge in the later stages to improve stability; the specific formula is as follows: ； In the formula, The MAPPO algorithm is in the first... The learning rate of each step; The initial learning rate; Total training steps; This represents the current number of training steps.

7. The low-carbon optimized operation method for microgrid clusters based on the improved MAPPO algorithm according to claim 6, characterized in that: Step 6 includes the following steps: S6.1: Initialize the low-carbon optimization operation model of the microgrid swarm, and define the observation space, action space, reward function and corresponding constraints for each microgrid agent; S6.2: Each agent selects an action according to the current policy, records the current observations, actions and rewards, stores these experiences in a shared experience pool, and balances exploration and utilization. S6.3: When the agent performs experience sample sampling, a negative sampling method is used to optimize the diversity and representativeness of the training samples; S6.4: LSTM-MAPPO model using a differentiated learning rate strategy; calculate the advantage function and objective function for each micro-network agent; calculate the policy gradient based on the objective function and update the policy of the micro-network agent to ensure collaborative optimization of the policy; S6.5: Repeat S6.2 to S6.4 until the LSTM-MAPPO model performance is stable or the maximum number of training rounds is reached; S6.6: After training, export the optimal strategy, perform offline verification, and deploy it to the low-carbon optimization scheduling system of the microgrid cluster to ensure its effectiveness in a real-world environment.

Images