Layered multi-agent microgrid optimization and carbon emission management method

Abstract

The invention relates to a layered multi-agent micro-grid optimization and carbon emission management method, which comprises the following steps of: constructing a layered multi-agent framework to divide a micro-grid system into upper-layer micro-grid collaborative optimization and lower-layer single micro-grid energy management, and realizing energy flow collaborative optimization between micro-grids by an upper layer through a trust domain model; the lower layer optimizes the energy management of a single micro-grid through a multi-agent reinforcement learning algorithm, so that the self-adaptive capability of the system is improved; photovoltaic and wind energy renewable energy sources and a super capacitor, pumped storage and battery hybrid energy storage system are integrated, energy fluctuation is coped, stable operation of the system is ensured, meanwhile, the carbon emission strategy of the system is optimized through a carbon emission limitation model, fossil fuel is reduced, and green energy management is promoted; the load in the peak period of the power grid is effectively reduced, and the energy management efficiency and environmental friendliness are improved; the method has the advantages that the reinforcement learning algorithm is adopted, the adaptability of the system is improved, and the energy management efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of collaborative optimization and carbon emission management technology for multi-energy microgrid systems, specifically relating to a hierarchical multi-agent microgrid optimization and carbon emission management method. Background Technology

[0002] With the continuous increase in global energy demand and the growing awareness of environmental protection, how to reduce carbon emissions while meeting load demand has become an important research direction for microgrid systems. Existing microgrid optimization methods often fail to fully consider the impact of energy volatility and random load changes, especially in complex systems with multiple energy sources. Existing optimization strategies are difficult to achieve both system robustness and sustainability simultaneously. Therefore, it is necessary to provide a hierarchical multi-agent microgrid optimization and carbon emission management method that uses reinforcement learning algorithms to improve system adaptability and energy management efficiency. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide a hierarchical multi-agent microgrid optimization and carbon emission management method that uses reinforcement learning algorithms to improve the adaptability of the system and improve energy management efficiency.

[0004] The objective of this invention is achieved as follows: a hierarchical multi-agent microgrid optimization and carbon emission management method, comprising the following steps:

[0005] Step 1: Construct a hierarchical multi-agent architecture, dividing the microgrid system into upper-layer microgrid collaborative optimization and lower-layer single microgrid energy management;

[0006] Step 2: In the upper layer, collaborative optimization between microgrids is achieved through the trust domain model to manage the energy flow between microgrids;

[0007] Step 3: In the lower layer, the energy management within a single microgrid is optimized through a multi-agent reinforcement learning algorithm to improve the system's adaptability;

[0008] Step 4: Integrate renewable energy with hybrid energy storage systems to enhance the system's adaptability to energy fluctuations;

[0009] Step 5: Optimize energy storage and distribution in different time periods based on electricity time-based pricing strategies to reduce reliance on the power grid during peak hours;

[0010] Step 6: Optimize the system's carbon emission strategy through carbon emission limitation models, reduce the use of fossil fuels, and achieve green energy management.

[0011] Step 2, which utilizes the trust domain model to address the mutual exclusion problem in multi-agent decision-making, specifically includes the following steps:

[0012] Step 2.1: Using the multi-agent advantage decomposition lemma, a trust domain is established among multiple device control agents. The trust domain is represented by the decomposition of the joint advantage function as follows: in, Let be the joint advantage function in state s, representing the overall decision performance of all agents given a joint policy π; This represents the local advantage of the ij-th agent given the definite actions of other agents;

[0013] Step 2.2: Through the decomposition of the trust domain model, each agent can effectively reduce the mutual exclusion effect in the decision-making sequence during the optimization process. This model decomposes the joint advantage function into local advantage functions, so that each agent only makes decisions within its "trust domain", reducing interference with other agents and enabling each agent to optimize more independently, avoiding decision conflicts caused by mutual influence.

[0014] The trust domain model in step 2 uses the KL divergence constraint formula to constrain the policy update step size. By controlling the policy update magnitude through KL divergence, it ensures that each update does not deviate too much from the original policy, thus maintaining the stability of the system. In the formula, This normalizes the samples in the experience pool to standardize the KL divergence, ensuring that the divergence of all samples and time steps is considered uniformly; E is the number of samples in the experience pool, and T is the time step. It is the KL divergence, used to measure the difference between two probability distributions, representing the divergence between state s at time step t. t and joint strategy b t Under the conditions, the old strategy and new strategies The differences between them.

[0015] In step 2, the trust domain model primarily controls the magnitude of policy updates within the trust domain, allowing each agent to gradually adjust its policy during optimization without drastic changes. By limiting the step size to the KL divergence constraint, the stability of the policy update is ensured, as shown in the following formula: in, This represents the policy parameters for the microgrid m and the agent n, with the superscript 'a' representing the agent's action space; α ls δ is a learning rate parameter related to line search; δ is a parameter of the KL divergence constraint. This represents the direction vector being updated; The curvature representing the policy distribution; It is the scaling factor for the step size.

[0016] The multi-agent reinforcement learning algorithm in step 3 is as follows: Each agent independently manages a specific device in the microgrid. Through the policy optimization process in the multi-agent reinforcement learning algorithm, each agent autonomously selects the optimal action based on local environmental information and can improve its adaptability to changing environments by adjusting its strategy in real time. By introducing a hot experience pool mechanism, each agent can store and utilize historical state, action, and reward information, reducing random exploration in high-dimensional space. At the same time, agents collaborate to optimize through a global review network, ensuring that each device remains consistent with the overall system goal while optimizing independently. Each agent works collaboratively by sharing the results of local policies.

[0017] Step 4 specifically involves: integrating renewable energy and hybrid energy storage systems, and optimizing energy allocation through a fuzzy logic controller to ensure that each energy storage technology can work collaboratively based on load fluctuations and time-based usage rate strategies; the power balance formula ensures that the output of each energy system in the microgrid system can meet the load demand in each time period, and is used to balance the output power of different energy sources. The power balance formula is as follows: P load(j) =P 1(j) +P 2(j) +P 4(j) +P 6(j) +P 8(j) (1≤j≤N)(4), where P load(j) P represents the total load demand of the microgrid during time period j; 1(j) P represents the power generated by the photovoltaic system during time segment j; 2(j) P represents the power output of the battery system during time period j; 4(j) P represents the output power of a hydroelectric power generation system. 6(j) P represents the power output of the supercapacitor during time period j; 8(j) This represents the power supplied by the power grid to the load during time period j.

[0018] The strategy based on electricity time rates in step 5 is as follows: by monitoring electricity price fluctuations, the grid is prioritized to charge the energy storage system during periods of lower electricity prices, and the energy storage system is used to supply power to the load during peak periods of higher electricity prices, thereby reducing the need to draw power from the grid during peak periods. This strategy dynamically adjusts the charging and discharging behavior of the energy storage device through a fuzzy logic control system to ensure that the stored low-priced electricity is utilized to the maximum extent during peak load demand periods.

[0019] Step 6, optimizing the system's carbon emission strategy through a carbon emission limitation model, specifically involves: optimizing the system's carbon emission strategy through a carbon emission limitation model, including automatically adjusting the energy allocation strategy based on the use of fossil fuels and the supply capacity of green energy, in order to achieve the goal of minimizing carbon emissions.

[0020] The carbon emission limitation model in step 6 includes establishing the following core indicators:

[0021] ① Indicators for measuring the carbon emission level of current energy use: Analyze the emissions generated by the use of fossil fuels, including calculating the carbon dioxide emissions generated per unit of fossil fuel, as the basis for subsequent optimization strategies;

[0022] ② Integrate historical energy consumption and production pattern data indicators, especially seasonal data, to predict energy demand in different seasons in the future. This data can be used to develop targeted carbon reduction strategies, which may rely more on renewable energy during peak energy demand periods and reduce the use of fossil fuels.

[0023] ③ Integrate volatility indicators of various renewable energy outputs: especially the seasonal variations of solar and wind power, as well as the stability of hydropower. Renewable energy power generation fluctuates in different seasons. For example, solar power generation is more active in summer, while wind power performs better from March to August, in order to ensure the maximum utilization of renewable energy and reduce dependence on fossil fuels.

[0024] The beneficial effects of this invention are as follows: This invention provides a microgrid system optimization method based on a hierarchical multi-agent reinforcement learning algorithm. In use, this method constructs a hierarchical multi-agent architecture, dividing the microgrid system into an upper-layer microgrid collaborative optimization and a lower-layer single microgrid energy management. The upper layer achieves collaborative optimization of energy flow between microgrids through a trust domain model, ensuring operational stability among agents. The lower layer optimizes the energy management of a single microgrid through a multi-agent reinforcement learning algorithm, improving the system's adaptability. This invention integrates renewable energy sources such as photovoltaics and wind power with hybrid energy storage systems such as supercapacitors, pumped hydro storage, and batteries to cope with energy fluctuations and ensure stable system operation. Simultaneously, it optimizes the system's carbon emission strategy through a carbon emission constraint model, reducing the use of fossil fuels and promoting green energy management. This invention can effectively reduce the load on the power grid during peak hours, improving the energy management efficiency and environmental friendliness of the microgrid system. This invention has the advantages of employing reinforcement learning algorithms, improving system adaptability, and enhancing energy management efficiency. Attached Figure Description

[0025] Figure 1 This is a diagram of the multi-agent microgrid and carbon emission management system of the present invention.

[0026] Figure 2 This is a schematic diagram of the trust domain model of the present invention.

[0027] Figure 3 This is a schematic diagram of the carbon emission model of the present invention. Detailed Implementation

[0028] The microgrid system optimizes energy coordination among microgrids at the upper layer; at the lower layer, energy management within a single microgrid is optimized through multi-agent reinforcement learning; a trust domain model reduces conflicts between agents and improves system stability; the system coordinates the operation of photovoltaic, wind power, and hybrid energy storage systems (supercapacitors, pumped hydro storage, and batteries) to cope with energy fluctuations; the energy storage system responds rapidly during peak hours to ensure system stability; in this embodiment, a carbon emission constraint model optimizes the system's carbon emissions, adjusting fossil fuel use based on seasonal energy demand changes to minimize carbon emissions while maintaining stable system operation.

[0029] The present invention will now be further described with reference to the accompanying drawings.

[0030] Example 1

[0031] like Figure 1-3 As shown, a hierarchical multi-agent microgrid optimization and carbon emission management method includes the following steps:

[0032] Step 1: Construct a hierarchical multi-agent architecture, dividing the microgrid system into upper-layer microgrid collaborative optimization and lower-layer single microgrid energy management;

[0033] Step 2: In the upper layer, collaborative optimization between microgrids is achieved through the trust domain model to manage the energy flow between microgrids;

[0034] Step 3: In the lower layer, the energy management within a single microgrid is optimized through a multi-agent reinforcement learning algorithm to improve the system's adaptability;

[0035] Step 4: Integrate renewable energy (including photovoltaic and wind power) with hybrid energy storage systems (including supercapacitors, pumped hydro storage, and batteries) to enhance the system's adaptability to energy fluctuations;

[0036] Step 5: Optimize energy storage and distribution in different time periods based on electricity time-based pricing strategies to reduce reliance on the power grid during peak hours;

[0037] Step 6: Optimize the system's carbon emission strategy through carbon emission limitation models, reduce the use of fossil fuels, and achieve green energy management.

[0038] This invention presents a microgrid system optimization method based on a hierarchical multi-agent reinforcement learning algorithm. In practice, this method constructs a hierarchical multi-agent architecture, dividing the microgrid system into an upper-layer microgrid collaborative optimization and a lower-layer single microgrid energy management. The upper layer uses a trust domain model to achieve collaborative optimization of energy flow between microgrids, ensuring operational stability among agents. The lower layer optimizes the energy management of a single microgrid using a multi-agent reinforcement learning algorithm, improving the system's adaptability. This method integrates renewable energy sources such as photovoltaics and wind power with hybrid energy storage systems such as supercapacitors, pumped hydro storage, and batteries to cope with energy fluctuations and ensure stable system operation. Simultaneously, it optimizes the system's carbon emission strategy through a carbon emission constraint model, reducing fossil fuel use and promoting green energy management. This method effectively reduces grid load during peak hours, improving the energy management efficiency and environmental friendliness of the microgrid system. This invention has the advantages of employing reinforcement learning algorithms, improving system adaptability, and enhancing energy management efficiency.

[0039] Example 2

[0040] like Figure 1-3 As shown, a hierarchical multi-agent microgrid optimization and carbon emission management method includes the following steps:

[0041] Step 1: Construct a hierarchical multi-agent architecture, dividing the microgrid system into upper-layer microgrid collaborative optimization and lower-layer single microgrid energy management;

[0042] Step 2: In the upper layer, collaborative optimization between microgrids is achieved through the trust domain model to manage the energy flow between microgrids;

[0043] In this embodiment, the trust domain model addresses the mutual exclusion problem in multi-agent decision-making to ensure the stability and robustness of collaborative operations among agents. Specifically, it includes the following steps:

[0044] Step 2.1: Using the multi-agent advantage decomposition lemma, a trust domain is established among multiple device control agents. The trust domain is represented by the decomposition of the joint advantage function as follows: in, Let be the joint advantage function in state s, representing the overall decision performance of all agents given a joint policy π; This represents the local advantage of the ij-th agent given the definite actions of other agents;

[0045] Step 2.2: Through the decomposition of the trust domain model, each agent can effectively reduce the mutual exclusion effect in the decision-making sequence during the optimization process. This model decomposes the joint advantage function into local advantage functions, so that each agent only makes decisions within its "trust domain", reducing interference with other agents. In this way, each agent can optimize more independently and avoid decision conflicts caused by mutual influence. The trust domain model not only improves the decision independence of the system, but also enhances its adaptability to the fluctuation of renewable energy and random changes in load in complex environments. Finally, the system exhibits stronger robustness and stability when dealing with these uncertainties and random factors, ensuring that the multi-agent system can work efficiently and maintain stable operating efficiency in complex and ever-changing environments.

[0046] The KL divergence constraint formula is used to constrain the policy update step size. By controlling the policy update magnitude through KL divergence, it ensures that each update does not deviate too much from the original policy, thus maintaining system stability. In the formula, It normalizes the samples in the experience pool, where E is the number of samples in the experience pool and T is the time step. This factor is used to standardize the divergence to ensure that the divergence of all samples and time steps is considered uniformly. This is the KL divergence, used to measure the difference between two probability distributions. In this formula, it represents the divergence between state s at time step t. t and joint strategy b t Under the conditions, the old strategy and new strategies The difference between them; KL divergence calculates the difference in the output distribution of the old and new strategies under the same input;

[0047] Policy updates within the trust domain primarily control the magnitude of policy updates, allowing each agent to gradually adjust its policy during optimization without drastic changes. By limiting the step size within the KL divergence constraint, the stability of policy updates is guaranteed, as shown in the following formula: in, α represents the policy parameters for the microgrid m and the agent n, with the superscript 'a' indicating the agent's action space, meaning this is a policy update about how to take actions; ls δ is a learning rate parameter related to line search, which is used to find a suitable step size to ensure that the divergence limit is not exceeded in each update; δ is a parameter of the KL divergence constraint, which is used to control the step size of the policy update to ensure that the new policy does not differ too much from the old policy. Here, it limits the maximum step size of each update. This represents the direction vector being updated; It is the second derivative of the Hessian matrix related to the KL divergence, representing the curvature of the policy distribution. This matrix is ​​very important in the optimization process, as it reflects the rate of change during policy updates. Since directly inverting the Hessian matrix would result in high computational overhead, the formula uses the conjugate gradient method for approximate calculation. It is the scaling factor of the step size, which controls the size of the step size through KL divergence constraints and the inverse of the Hessian matrix. This part ensures that the magnitude of policy updates conforms to the KL divergence constraint and prevents instability caused by excessively large update step sizes.

[0048] Step 3: In the lower layer, the energy management within a single microgrid is optimized through a multi-agent reinforcement learning algorithm to improve the system's adaptability;

[0049] In this embodiment, a hierarchical trust domain multi-agent reinforcement learning algorithm (such as the CD-EHATRPO algorithm) is used to optimize energy management within a single microgrid, improving the system's adaptability. Each agent independently manages specific devices within the microgrid, such as energy storage devices, photovoltaic power generation devices, and wind power generation devices. Each agent autonomously selects the optimal action based on local environmental information through the policy optimization process in the multi-agent reinforcement learning algorithm, and can improve its adaptability to changing environments by adjusting its strategy in real time. By introducing a hot experience pool mechanism, each agent can store and utilize historical state, action, and reward information, reducing random exploration in high-dimensional space, thereby improving learning efficiency and accelerating policy convergence. At the same time, agents collaborate to optimize through a global review network, ensuring that each device remains consistent with the overall system goal while optimizing independently. Each agent works collaboratively by sharing the results of local policies to ensure that the system maintains high efficiency and stability in the face of load fluctuations and unstable energy supply.

[0050] Step 4: Integrate renewable energy (including photovoltaic and wind power) with hybrid energy storage systems (including supercapacitors, pumped hydro storage, and batteries) to enhance the system's adaptability to energy fluctuations;

[0051] In this embodiment, the system's adaptability to energy fluctuations is enhanced by integrating renewable energy sources (including photovoltaic and wind power) with a hybrid energy storage system (including supercapacitors, pumped hydro storage, and batteries). The photovoltaic power generation system serves as the primary energy source during the day, the hybrid energy storage system responds rapidly to high load demand via supercapacitors, the pumped hydro storage system slowly releases energy with its high energy density during low load demand, and the batteries are used for standard load demand. This combination optimizes energy allocation through a fuzzy logic controller, ensuring that each energy storage technology can work collaboratively according to load fluctuations and time-based usage rate strategies, thereby achieving system stability and reliability. The power balance formula ensures that the output of each energy system (photovoltaic, energy storage, grid, etc.) in the microgrid system meets load demand in each time period. This formula is a core part of microgrid energy management, used to balance the output power of different energy sources to ensure stable system operation. The power balance formula is as follows: P load(j) =P 1(j) +P 2(j) +P 4(j) +P 6(j) +P 8(j) (1≤j≤N)(4), where P load(j) This represents the total load demand of the microgrid during time period j, which is the energy demand that the system must meet during that period, including various electrical loads from users. The contribution of each energy device to the total load at different time periods is represented by different symbols, P. 1(j) P represents the power generated by the photovoltaic system within time segment j. As a major renewable energy source in microgrids, the output power of the photovoltaic system is mainly affected by environmental factors such as light intensity; 2(j) This represents the power output of the battery system during time period j. The battery, as part of an energy storage system, is used to provide supplemental power when the load is high, or to store power when the energy supply is sufficient. 4(j) P represents the output power of a hydroelectric power system. Although not all microgrids have hydroelectric power facilities, in systems with pumped storage, hydroelectric power is used to store electrical energy when the load is low and release it when the load is high. 6(j) This represents the power output of the supercapacitor within time period j. Supercapacitors can respond quickly to load fluctuations and are particularly suitable for high load responses over short periods of time; finally, P 8(j) This represents the power provided by the grid to the load during time period j. As a backup power source, the grid can provide stable power support to the load when renewable energy and energy storage devices cannot meet the demand. The sum of these different sources of power ensures that the microgrid system can balance energy supply and load demand in each time period, thus achieving stable system operation.

[0052] Step 5: Optimize energy storage and distribution in different time periods based on electricity time-based pricing strategies to reduce reliance on the power grid during peak hours;

[0053] In this embodiment, based on a time-of-use (ToU) pricing strategy, energy storage and allocation are optimized for different time periods to reduce reliance on the power grid during peak hours. Specifically, by monitoring electricity price fluctuations, the power grid is prioritized for charging the energy storage system during periods of lower electricity prices (such as late at night), while the energy storage system (including supercapacitors, pumped hydro storage, and batteries) supplies power to the load during peak hours when electricity prices are higher, reducing the need to draw power from the grid during peak hours. This strategy dynamically adjusts the charging and discharging behavior of the energy storage devices through a fuzzy logic control system (FLC) to ensure maximum utilization of stored low-priced electricity during peak load demand periods, thereby reducing reliance on the power grid and optimizing system operating costs.

[0054] Step 6: Optimize the system's carbon emission strategy through carbon emission limitation models, reduce the use of fossil fuels, and achieve green energy management.

[0055] In this embodiment, the carbon emission strategy of the system is optimized through a carbon emission limitation model. This further includes automatically adjusting the energy allocation strategy based on fossil fuel usage and green energy supply capacity to achieve the goal of minimizing carbon emissions while maintaining the stability and efficiency of the energy system. This model is established through the following core indicators:

[0056] ① Traditional fossil fuels (such as coal, oil and natural gas) have been the main source of energy in the past, but their negative impact on the environment is obvious. Due to the continued use of fossil fuels, global carbon emissions are rising and have become the focus of global environmental issues. Therefore, analyzing the carbon emission levels of existing energy systems, especially the carbon emissions generated by fossil fuels, is a key step in establishing carbon emission limit models.

[0057] The model needs to measure the carbon emission level of current energy use, especially to analyze the emissions generated by the use of fossil fuels. This includes calculating the carbon dioxide emissions per unit of fossil fuel (such as per kilowatt-hour of coal or natural gas power generation) as the basis for subsequent optimization strategies.

[0058] ② Energy consumption and production patterns vary significantly across different seasons. For example, solar power generation is higher in summer, while energy demand may increase in winter. In particular, energy consumption drops to 580,000 kWh in May and rises to 850,000 kWh in August, indicating a significant impact of seasonal variations on energy demand.

[0059] To effectively optimize carbon emissions, the model needs to integrate historical energy consumption data, especially seasonal data, to predict energy demand in different seasons in the future. This data can be used to develop targeted carbon reduction strategies, which may rely more on renewable energy and reduce the use of fossil fuels during peak energy demand periods (such as August).

[0060] ③ Solar microgrids contributed 3.2% to 5.3% of the total electricity, wind microgrids contributed 5.9% to 7.4%, and hydropower microgrids contributed 24.4%. These data illustrate the contribution of renewable energy to the electricity of the entire system and provide direct data support for optimizing carbon emission models.

[0061] The model should take into account the output of various renewable energy sources, especially the seasonal variations of solar and wind power, as well as the stability of hydropower. The power generation of renewable energy sources fluctuates in different seasons. For example, solar power generation is more active in summer, while wind power performs better from March to August. This means that the model needs to integrate these fluctuations to ensure that renewable energy is used to the maximum extent and dependence on fossil fuels is reduced.

[0062] This invention presents a microgrid system optimization method based on a hierarchical multi-agent reinforcement learning algorithm. The method constructs a hierarchical multi-agent architecture, dividing the microgrid system into an upper-layer microgrid collaborative optimization and a lower-layer individual microgrid energy management. The upper layer uses a trust domain model to achieve collaborative optimization of energy flow between microgrids, ensuring operational stability among agents. The lower layer optimizes the energy management of individual microgrids using a multi-agent reinforcement learning algorithm, improving the system's adaptability. This invention integrates renewable energy sources such as photovoltaics and wind power with hybrid energy storage systems such as supercapacitors, pumped hydro storage, and batteries to cope with energy fluctuations and ensure stable system operation. Simultaneously, it optimizes the system's carbon emission strategy through a carbon emission constraint model, reducing fossil fuel use and promoting green energy management. This invention can effectively reduce grid load during peak hours, improving the energy management efficiency and environmental friendliness of the microgrid system. This invention has the advantages of employing reinforcement learning algorithms, improving system adaptability, and enhancing energy management efficiency.

Claims

1. A hierarchical multi-agent microgrid optimization and carbon emission management method, characterized by: The following steps are involved: Step 1: Construct a hierarchical multi-agent architecture to divide the microgrid system into upper-level microgrid collaborative optimization and lower-level single microgrid energy management; Step 2: In the upper layer, the collaborative optimization between microgrids is realized through the trust domain model to manage the energy flow between microgrids; Step 3: In the lower layer, the energy management within a single microgrid is optimized through a multi-agent reinforcement learning algorithm to improve the adaptability of the system; Step 4: Integrate renewable energy and hybrid energy storage systems to enhance the system’s adaptability to energy fluctuations; Step 5: Optimize energy storage and allocation in different time periods based on the strategy of electricity time rate; Step 6: Optimize the system’s carbon emission strategy through the carbon emission limitation model.

2. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 1, characterized in that: In step 2, the trust region model is used to solve the mutual exclusion problem in decision-making among multiple agents, which specifically includes the following steps: Step 2.1: Through the multi-agent advantage decomposition lemma, a trust domain is established between multiple device control agents. The trust domain is expressed based on the decomposition of the joint advantage function as follows: in, is the joint advantage function in state s, which represents the overall decision effect of all agents under a given joint strategy π; represents the local advantage of the ijth agent under certain actions of other agents; Step 2.2: Through the decomposition of the trust domain model, each agent can effectively reduce the mutual exclusion effect in the decision sequence during the optimization process. This model decomposes the joint advantage function into local advantage functions, so that each agent only makes decisions within its "trust domain", reducing interference with other agents, allowing each agent to optimize more independently and avoid decision conflicts caused by mutual influence.

3. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 2, characterized in that: The trust region model in step 2 uses the KL divergence constraint formula to constrain the strategy update step length. The KL divergence is used to control the strategy update amplitude to ensure that each update does not deviate too much from the original strategy and maintain the stability of the system: In the formula, Normalize the samples in the experience pool to standardize the KL divergence and ensure that the divergence of all samples and time steps is considered evenly; E is the number of samples in the experience pool, T is the time step; is the KL divergence, which is used to measure the difference between two probability distributions, indicating that at time step t, state s t and joint strategy b t Under the condition of and new strategies The difference between.

4. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 3, characterized in that: The trust domain model in step 2 mainly controls the strategy update amplitude in the trust domain, so that each agent can gradually adjust its strategy during optimization without drastic changes. By limiting the step size within the KL divergence constraint, the stability of the strategy update is guaranteed. The formula is as follows: in, is the strategy parameter representing the microgrid m and the intelligent agent n, and the superscript a represents the action space of the intelligent agent; α ls is a learning rate parameter related to line search; δ is a parameter of the KL divergence constraint; represents the updated direction vector; represents the curvature of the strategy distribution; is the scaling factor for the step size.

5. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 1, characterized in that: The multi-agent reinforcement learning algorithm in step 3 is specifically as follows: each agent is independently responsible for managing specific equipment in the microgrid, and each agent autonomously selects the optimal action based on local environmental information through the strategy optimization process in the multi-agent reinforcement learning algorithm, and can improve its adaptability to the changing environment by adjusting the strategy in real time; By introducing the hot experience pool mechanism, each agent can store and utilize historical states, actions, and reward information, reducing random exploration in high-dimensional space. At the same time, agents collaborate to optimize through a global review network to ensure that each device is independently optimized while remaining consistent with the goals of the overall system. Agents work together by sharing the results of local strategies.

6. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 1, characterized in that: The step 4 is specifically as follows: integrating renewable energy and hybrid energy storage system to optimize energy distribution through fuzzy logic controller, ensuring that each energy storage technology can work together according to load fluctuation and time usage rate strategy; the power balance formula is used to ensure that the output of each energy system in the microgrid system can meet the load demand in each time period, and is used to balance the output power of different energy sources. The power balance formula is as follows: P load(j) =P 1(j) +P 2(j) +P 4(j) +P 6(j) +P 8(j) (1≤j≤N)(4), where P load(j) represents the total load demand of the microgrid in time period j; P 1(j) represents the power generated by the photovoltaic system in the time period j; P 2(j) represents the power output of the battery system in time period j; P 4(j) Represents the output power of the hydropower system; P 6(j) represents the power output of the supercapacitor in time period j; P 8(j) It represents the power provided by the grid to the load in time period j.

7. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 1, characterized in that: The strategy based on the electricity time rate in step 5 is specifically as follows: by monitoring electricity price fluctuations, the power grid is preferentially used to charge the energy storage system during periods of low electricity prices, and the energy storage system is used to supply power to the load during peak periods when electricity prices are higher, thereby reducing the demand for electricity from the power grid during peak periods. This strategy dynamically adjusts the charging and discharging behavior of the energy storage device through a fuzzy logic control system to ensure maximum utilization of the stored low-cost electricity during peak load demand periods.

8. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 1, characterized in that: The step 6 of optimizing the system's carbon emission strategy through the carbon emission limitation model is specifically as follows: optimizing the system's carbon emission strategy through the carbon emission limitation model, including automatically adjusting the energy allocation strategy based on the use of fossil fuels and the green energy supply capacity, so as to achieve the goal of minimizing carbon emissions.

9. A hierarchical multi-agent microgrid optimization and carbon emission management method as claimed in claim 8, characterized in that: The carbon emission limitation model in step 6 includes establishing the following core indicators: ① Indicators to measure the carbon emission level of current energy use; ②Integrate historical energy consumption and production pattern data indicators; ③Integrate volatility indicators of various renewable energy production.

Images