Low-carbon economic integrated energy system energy planning method based on multi-agent deep reinforcement learning

Abstract

The invention discloses an energy planning method for a low-carbon economic comprehensive energy system based on multi-agent deep reinforcement learning. The method comprises the following steps: constructing a comprehensive energy system structure including multiple energy forms such as wind power, photovoltaic, electricity-to-gas, combined heat and power generation, energy storage and the like; establishing a carbon emission intensity model of each subsystem based on an energy input and output relationship, and constructing a segmented carbon tax function to quantify the carbon cost; a double-layer two-stage Nash optimization model is designed, and the game relationship between collaboration between subsystems and external energy interaction is considered; the optimization problem is converted into a Markov decision process, and a state, an action and a reward function are defined; a multi-agent TD3 algorithm model is constructed and trained, and strategy stability and robustness are improved through a differential evolution mechanism; and deploying the trained strategy model in an actual system to realize self-adaptive energy planning and scheduling for the carbon economic target. The method has the characteristics of high adaptability, high carbon benefit and excellent intelligent cooperation capability.

Description

Technical Field

[0001] The present invention belongs to the field of integrated energy system optimization and relates to an integrated energy system energy planning method, specifically to a low-carbon economic integrated energy system energy planning method based on multi-agent deep reinforcement learning. Background Art

[0002] As the global energy mix accelerates toward a low-carbon, clean energy future, integrated energy systems (IES) are becoming a crucial component of future energy infrastructure. By coupling multiple energy sources—electricity, thermal energy, gas energy, and renewable energy sources (such as photovoltaics and wind power)—IES achieves diversified, coordinated energy supply and optimized allocation, improving energy efficiency. However, IES faces multiple challenges in its actual operation:

[0003] On the one hand, renewable energy output is highly volatile and uncertain, particularly in microgrid or independent energy supply scenarios, which can easily lead to system instability. On the other hand, user load demand is dynamic and random, and market signals such as electricity and gas prices fluctuate frequently. The implementation of carbon tax policies further complicates energy scheduling. Traditional optimization methods, such as linear programming, mixed integer programming, and robust optimization, while able to address some uncertainties to a certain extent, generally suffer from poor real-time performance and difficulty handling high-dimensional nonlinear decision-making problems. Furthermore, most existing methods ignore the differences in operational objectives among energy subsystems, fail to model their game-playing behaviors, and lack a systematic, intelligent coordination mechanism. Therefore, there is an urgent need for an intelligent optimization and scheduling method that combines uncertainty modeling, multi-energy coordination, and distributed game-solving capabilities. Summary of the Invention

[0004] To address the challenges of slow energy flow scheduling and the difficulty in quantifying and controlling carbon emission constraints in complex operating environments within integrated energy systems, as well as the poor robustness of traditional optimization methods in dealing with electricity price fluctuations, renewable energy uncertainty, and dynamic changes in carbon costs, this paper provides an energy planning method for low-carbon integrated energy systems based on multi-agent deep reinforcement learning. This method is applicable to integrated energy systems connected to microgrids and can achieve efficient, coordinated, and low-carbon energy scheduling optimization for these systems. It features strong adaptability, high carbon efficiency, and excellent intelligent coordination capabilities.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for energy planning of a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning includes the following steps:

[0007] Step S1: Constructing a comprehensive energy system structure including wind power system (WT), photovoltaic system (PV), power-to-gas system (PtG), combined heat and power system (CHP), energy storage system (ESS) energy units and load nodes;

[0008] Step S2: Model the carbon emission intensity of the energy flow of each subsystem in step S1 to obtain a carbon emission intensity model. Calculate the node carbon emission intensity (NCI) of electricity, heat, and gas based on the real-time operation data of the system, and construct a piecewise carbon tax penalty function to quantify the carbon cost of the system.

[0009] Step S3: Based on the integrated energy system structure constructed in step S1 and the carbon emission intensity model established in step S2, a two-layer two-stage Nash optimization model is designed;

[0010] Step S4: Convert the two-layer two-stage Nash optimization model in step S3 into a Markov decision process (MDP) form, define the state space, action space, reward function, and state transition function of each subsystem, and prepare for subsequent agent training;

[0011] Step S5: Construct an independent double-delayed deep deterministic policy gradient (TD3) reinforcement learning agent for each subsystem in step S1, including WT agent, PV agent, PtG agent, CHP agent and ESS agent, and define their respective local state variables, behavior control actions and reinforcement learning objectives;

[0012] Step S6: The multi-agent TD3 algorithm (MATD3) is used to train each agent, and the differential evolution perturbation mechanism (DEM) is introduced to improve strategy diversity and stability, forming an energy scheduling strategy network that takes into account both global benefits and local game;

[0013] Step S7: Deploy the multi-agent scheduling strategy trained in step S6 in the actual integrated energy system to achieve real-time robust optimization scheduling control under factors such as renewable energy output fluctuations, electricity price changes, and carbon tax policy impacts, and output energy flow control instructions and carbon emission assessment reports for each time period.

[0014] Compared with the prior art, the present invention has the following advantages:

[0015] 1. Strong system collaborative optimization capability: This invention models the multi-energy subsystem as a game participant and introduces a two-layer, two-stage Nash game optimization framework. The first layer coordinates the operating status of each subsystem under a controllable strategy, and the second layer uses a min-max mechanism to optimize the procurement cost under the worst-case scenario, thereby ensuring that the system still has the ability to operate stably under extreme fluctuation conditions.

[0016] 2. Highly robust and adaptable strategies: By combining the Markov decision process (MDP) and the differential evolution minimax strategy (DEM-Minimax), a multi-agent TD3 (MATD3) strategy was trained that combines local optimization with global scheduling objectives. This strategy can adaptively respond to changes in electricity prices, fluctuations in renewable energy output, and load uncertainty, achieving flexible and intelligent scheduling.

[0017] 3. Introducing dynamic carbon costs to drive low-carbon optimization: Modeling the impact of carbon taxes through a step-by-step carbon emission cost function, incorporating carbon emission intensity into the intelligent agent reward function, guiding each energy subsystem to actively move towards a low-carbon path during operation, effectively improving the carbon economy of the system, and achieving active control and reduction of carbon emissions while ensuring energy supply security and economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flowchart of the energy planning method for a low-carbon economic integrated energy system based on multi-agent deep reinforcement learning.

[0019] Figure 2 A detailed flow chart was developed for the carbon emission intensity model.

[0020] Figure 3 A specific flow chart for establishing a two-layer two-stage Nash optimization model.

[0021] Figure 4 Flowchart for the energy planning strategy implementation based on the multi-agent TD3 algorithm.

[0022] Figure 5 Flowchart for real-time optimization and scheduling of integrated energy for intelligent entities.

[0023] Figure 6 Power dispatching using multi-agent deep reinforcement learning algorithms in integrated energy systems.

[0024] Figure 7 Energy storage system scheduling using multi-agent deep reinforcement learning algorithms in integrated energy systems.

[0025] Figure 8 Natural gas system scheduling using multi-agent deep reinforcement learning algorithms in integrated energy systems. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below with reference to the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention that does not depart from the spirit and scope of the technical solution of the present invention should be included in the scope of protection of the present invention.

[0027] The present invention provides an energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning. Figure 1 As shown, the method constructs a comprehensive energy system structure that includes multiple energy forms such as wind power, photovoltaics, power-to-gas, cogeneration, and energy storage; establishes a carbon emission intensity model for each subsystem based on the energy input-output relationship, and constructs a piecewise carbon tax function to quantify the carbon cost; designs a two-layer two-stage Nash optimization model that takes into account the game relationship between the coordination between subsystems and the interaction with external energy; transforms the optimization problem into a Markov decision process, defines the state, action, and reward function; constructs and trains a multi-agent TD3 algorithm model, and improves the strategy stability and robustness through the differential evolution mechanism; deploys the trained strategy model in the actual system to achieve adaptive energy planning and scheduling for carbon economy goals. The specific implementation steps are as follows:

[0028] Step S1: Construct a comprehensive energy system structure including energy units such as WT (wind power generation system), PV (photovoltaic power generation system), PtG (power-to-gas system), CHP (combined heat and power system), ESS (energy storage system) and load nodes.

[0029] In this step, the interconnected structure of three energy channels—the power grid, the heat grid, and the gas grid—was established. The energy inputs and outputs of each subsystem, as well as their interactions, were defined to form a multi-energy complementary architecture. The IES, serving as an energy support platform, integrates its electrical subsystem with the main power grid and local electricity loads, integrating the ESS and renewable energy systems (RES) to achieve optimized energy distribution and management. The renewable energy system primarily consists of wind turbines and photovoltaic subsystems. Meanwhile, the CHP unit provides thermal energy to the IES by consuming coal, natural gas supplied from the natural gas grid, and renewable gas generated by the PtG.

[0030] Step S2: Model the carbon emission intensity of the energy flow of each subsystem in step S1 to obtain a carbon emission intensity model. Calculate the node carbon emission intensity (NCI) of electricity, heat, and gas based on the real-time operation data of the system, and construct a segmented carbon tax penalty function to quantify the carbon cost of the system.

[0031] like Figure 2 As shown in the figure, the establishment of the carbon emission intensity model mainly includes the following steps:

[0032] Step S21: Collect energy input and output data of each subsystem;

[0033] Step S22: Calculate the net power load of the power system;

[0034] For the power subsystem, the net power load in IES as follows:

[0035]

[0036] In the formula: For the power demand of IES, and are WT power generation and PV power generation respectively.

[0037] Step S23: Select the carbon emission model calculation method based on whether there is electricity purchase behavior, and use different formulas to calculate NCI:

[0038] when When <0, it means that the PV and WT in IES can fully support the power load, and the NCI of the power subsystem is as follows:

[0039]

[0040] Where, is the carbon emission rate of CHP, is the electrical power of CHP.

[0041] when When > 0, the NCI of the power subsystem is as follows:

[0042]

[0043] Where, For the grid NCI, To buy electricity from the grid.

[0044] Step S24: Constructing a carbon emission intensity model for the thermal subsystem:

[0045] For the thermal subsystem, heat and electricity are generated by CHP. The thermal network includes a forward power supply network and a return power supply network. The carbon emissions of the source node as energy support satisfy the following formula:

[0046]

[0047] in, 、 、 and They are the energy of thermal power supplied to the grid, thermal power returned to the grid, gas supplied by the main grid and gas supplied by the power-to-gas system (PtG). 、 、 and They are the carbon emissions from the nodes of thermal power supply to the grid, thermal power returned to the grid, gas supply from the main grid and gas supply from the power-to-gas system.

[0048] In the thermal subsystem, NCI satisfies the following formula:

[0049]

[0050] in, To supply network node temperature, The temperature of the resource recovery network node, and are the heat component matrices in the supply and return networks respectively.

[0051] The NCI of the power supply node heating network is as follows:

[0052]

[0053] Step S25: Constructing a carbon emission intensity model for the gas combustion subsystem:

[0054] For the gas subsystem, the gas from the main grid and PtG supports the gas demand of CHP. The NCI of the gas subsystem is as follows:

[0055]

[0056]

[0057] in, is the electrical conversion efficiency.

[0058] Step S26: Establishing a penalty function based on carbon intensity segmentation intervals , using segmented penalty terms to regulate the energy usage structure.

[0059] In order to reduce carbon emissions, an optimized step-by-step carbon emission cost is proposed, which increases the carbon emission penalty during the EPS period between subsystems. The carbon emission cost is equal to the carbon emission tax and the output of NCI. The penalty factor in the carbon emission tax is and It is adjustable and helps the learning subject to comprehensively consider all operating costs including carbon emission costs, equipment costs, and energy planning costs. The optimized stepped carbon emission cost formula is as follows:

[0060]

[0061] Where, is the carbon emission cost, in cents / kWh; is the length of the carbon quota interval; is the NCI of the relevant subsystem, kg / kWh; is the cost of carbon emissions, cents / kg; is the energy consumption of the relevant subsystem.

[0062] Step S3: Based on the system structure constructed in step S1 and the carbon emission intensity model established in step S2, a two-layer two-stage Nash optimization model is designed, in which: the first layer realizes the internal energy coordination and flexible load regulation among the energy subsystems; the second layer realizes the electricity and gas purchase and sales strategy game between the integrated energy system and the main power grid; both layers are embedded with the min-max robust optimization mechanism under the worst-case scenario to deal with uncertain disturbances.

[0063] like Figure 3 As shown in Figure 2, the establishment of the two-layer two-stage Nash optimization model mainly includes the following steps:

[0064] Step S31: Design a two-stage nested Nash optimization scheduling framework that integrates system operation and transaction costs.

[0065] This step starts from the overall IES scheduling and constructs a nested two-stage Nash optimization planning framework. The goal is to consider the two-level robust optimization of operation and energy purchase under uncertain disturbances under the conditions of multi-subsystem coupling and multi-energy form coordination. The main model is as follows:

[0066]

[0067] Where, is the system cost in the current state, The cost of buying gas for the system, The cost of buying coal for the system, The cost of electricity for the system.

[0068] Step S32: Construct a first-stage operating state optimization model, formulate an initial operating state decision plan for the integrated energy system under disturbance conditions, and construct an operating cost function.

[0069] This step is used to determine the operating status of key subsystems within the IES system at the initial stage of operation, and output the optimal operating configuration with the goal of minimizing operating costs under disturbance conditions:

[0070] The state variables are defined as follows:

[0071]

[0072]

[0073] Where: Control charging and discharging of energy storage systems; is the operating power of the system; They are the heat, electricity and gas power support values ​​respectively.

[0074] The operating cost function is:

[0075]

[0076] Step S33: Construct the second-stage energy procurement robust optimization model and design the Min-Max energy procurement robust optimization mechanism for uncertain worst-case scenarios.

[0077] This step is based on the decision-making results of the first stage. It further considers the uncertain disturbances (such as price changes and load forecast errors) in the main grid's energy purchase process and constructs a Min-Max robust optimization model with the goal of minimizing the energy cost under the worst-case scenario:

[0078] Perturbation variable set:

[0079]

[0080] and These are the gas source generated by the PtG, the electricity sent to the P2G, the electricity output of the wind turbines, the heat output of the CHP system, and the electricity output of the CHP system. and represent the electricity demand and natural gas demand in the integrated energy system respectively.

[0081] The cost of electricity purchase is defined as follows:

[0082]

[0083]

[0084] Where, and It is the real-time electricity selling and buying price. and represent the electricity output surplus and electricity demand in the integrated energy system respectively.

[0085] The cost of purchasing gas is defined as follows:

[0086]

[0087]

[0088] Where, and They represent the real-time price of selling gas to the main gas grid and the price of purchasing gas from the main grid. and They are used to represent the gas output surplus and gas load demand in the integrated energy system. The coal purchase cost is defined as follows:

[0089]

[0090] Where, Indicates the real-time price of coal purchased from the main power grid.

[0091] Step S34: Construct a first-layer internal load optimization model and design an electricity, heat, gas and multi-energy net load optimization strategy based on a flexible load adjustment mechanism.

[0092] This step models the load regulation within the IES system and achieves coordinated control of load balancing and resource consumption through flexible optimization of electrical load, thermal load, and gas load.

[0093] Net electric load expression:

[0094]

[0095] Where, and Represent fixed power load and adjustable power load respectively.

[0096] Net heat load expression:

[0097]

[0098] Where, and Represent fixed heat load and adjustable heat load respectively.

[0099] Net gas load expression:

[0100]

[0101] Where, and Represent fixed heat load and adjustable heat load respectively.

[0102] Step S35: Construct a second-layer energy purchase strategy optimization model and formulate a dynamic game strategy for energy purchase and sale between the system and the main grid to maximize operating benefits.

[0103] This step models the energy purchasing behavior between the IES system and the main grid, and formulates the main grid's energy purchase / sale strategy through the Nash game to optimize the overall economic efficiency of the system.

[0104] The total cost of purchased energy is expressed as follows:

[0105]

[0106] This layer of model dynamically plans energy purchase strategies based on current load demand and market prices, while also linking the first layer’s net load calculation results.

[0107] Step S4: Convert the two-layer two-stage Nash optimization model in step S3 into a Markov decision process (MDP) form, define the state space, action space, reward function, and state transition function of each subsystem, and prepare for subsequent intelligent agent training.

[0108] This step converts the two-layer two-stage Nash optimization model into a Markov decision process, defines the state, action, and reward function of the agent, and implements multi-agent distributed decision control. It specifically includes the following steps:

[0109] Step S41: Model each energy subsystem in the system as an independent intelligent agent;

[0110] Step S42: define the state space S including load, electricity price, carbon intensity, and wind and solar energy resource state variables;

[0111] Step S43: defining action space A including electricity purchase, output dispatch, and energy storage system charging and discharging behaviors;

[0112] Step S44: constructing a reward function as the negative of the operating cost and the carbon cost as the optimization target;

[0113] Step S45: defining a state transfer function to be dynamically generated based on the environmental response.

[0114] Step S5: Build an independent TD3 reinforcement learning agent for each subsystem in step S1, and define their respective local state variables, behavior control actions, and reinforcement learning goals.

[0115] In this step, Multi-Agent Deep Reinforcement Learning (MADRL) is a multi-agent double-delayed deep deterministic policy gradient, a multi-agent learning algorithm that utilizes the TD3 algorithm. In the MADRL framework, multiple agents interact with a constantly changing IES environment, where the agents in each subsystem competitively optimize their own rewards while taking into account the actions and goals of other agents. Importantly, during execution, all TD3 agents collaborate through a centralized critic during the online training phase, enabling coordinated learning across agents. This balance between competition and collaboration enables each agent to achieve optimal performance and efficient and adaptive decision-making across the entire system. This architecture allows all TD3 agents to effectively handle collaborative scenarios and continuously operating environments. The holistic interaction between the participants and the centralized critic supports multi-agent learning in dynamic settings.

[0116] Step S6: The multi-agent TD3 algorithm (MATD3) is used to train each agent, and the differential evolution perturbation mechanism (DEM) is introduced to improve strategy diversity and stability, forming an energy scheduling strategy network that takes into account both global benefits and local games.

[0117] like Figure 4 As shown in the figure, the energy planning strategy execution flow chart based on the multi-agent TD3 algorithm mainly includes the following steps:

[0118] Step S61: Build a multi-agent reinforcement learning framework and define the local state and action space.

[0119] This step is based on the double-delayed deep deterministic policy gradient (TD3) to build a multi-agent deep reinforcement learning (MADRL) system. For each subsystem in IES (such as photovoltaic, wind power, CHP, PtG, energy storage, etc.), an independent TD3 agent is configured, and each agent uses its local state as input to make action decisions. The action space of each agent Defined as:

[0120]

[0121] Where, It represents the local decision action of the i-th agent at time t, and the action corresponds to the energy input or output adjustment instruction of the subsystem.

[0122] Step S62: Set up a centralized evaluation mechanism and build a dual-critic network for bias suppression.

[0123] To address the problem of overestimation of policy values ​​in reinforcement learning, this step introduces two sets of Critic network structures for value function estimation and uses the minimum value as the update basis. Each TD3 agent has an Actor network (which generates actions); all agents share a centralized Critic network to evaluate joint actions; and a "minimum Q value" mechanism is used to avoid policy overfitting.

[0124] The experience replay mechanism is used to store interaction data, and each agent updates network parameters with small batch samples to improve learning stability.

[0125] Step S63: Define a global reward function and build a global collaborative reward mechanism.

[0126] Each subsystem constructs a local reward function based on its own energy gain and carbon emission penalty. The system also establishes a global reward function to guide multiple agents to converge to the optimal goal of the system:

[0127] The global reward function is as follows:

[0128]

[0129] Where: is the local immediate reward of the i-th agent; is the weight of each agent; is the current operating cost of the system; is the reward discount factor. This mechanism ensures the coordination between individual learning goals and the overall operating benefits of the system.

[0130] Step S64: Introduce the differential evolution perturbation mechanism (DEM) to improve the efficiency of strategy exploration.

[0131] To enhance the diversity of policy exploration and avoid local optimality, the Differential Evolution Mechanism (DEM) is introduced into policy updates. This mechanism introduces a perturbation factor during the policy selection phase, thereby improving the action space search capability and enhancing adaptability to dynamic environments.

[0132] Step S65: Complete the strategy training and deploy it to the real IES scheduling environment to perform scheduling control.

[0133] After training, the generated TD3 strategy is embedded in the actual IES operating system, enabling adaptive, coordinated, and robust dispatch control for renewable energy fluctuations, load dynamics, electricity price signals, and carbon emission constraints. This control system supports continuous online learning and rolling optimization, making it suitable for complex real-world scenarios such as industrial parks and microgrids.

[0134] Step S7: Deploy the multi-agent scheduling strategy trained in step S6 in the actual integrated energy system to achieve real-time robust optimization scheduling control under factors such as renewable energy output fluctuations, electricity price changes, and carbon tax policy impacts, and output energy flow control instructions and carbon emission assessment reports for each time period.

[0135] like Figure 5 As shown in the figure, the multi-agent integrated energy real-time optimization scheduling flow chart mainly includes the following steps:

[0136] Step S71: Loading the trained policy models of each subsystem in the actual deployment of IES;

[0137] Step S72: Real-time collection of system status information, such as output, electricity price, load, energy storage level, etc.;

[0138] Step S73: input the state to the multi-agent energy scheduling strategy network and output the optimal action instruction;

[0139] Step S74: Execute electricity purchase and sales, charging and discharging, fuel scheduling, and energy storage system charging and discharging operations according to the agent's actions;

[0140] Step S75: Output includes energy scheduling instructions for each time period and corresponding node NCI evaluation values.

[0141] Figure 6Power dispatching using multi-agent deep reinforcement learning algorithms in integrated energy systems, Figure 7 For energy storage system scheduling using multi-agent deep reinforcement learning algorithm in integrated energy systems, Figure 8 This paper proposes a multi-agent deep reinforcement learning algorithm for natural gas system scheduling in an integrated energy system. Figures 6 to 8 It can be seen that the scheduling strategy based on the multi-agent deep reinforcement learning algorithm can effectively design the scheduling strategy for electricity and natural gas, save computing time, reduce carbon emissions and reduce the operating cost of the integrated energy system.

Claims

1. A low-carbon economy integrated energy system energy planning method based on multi-agent deep reinforcement learning, characterized by The method comprises the following steps: Step S1: constructing a comprehensive energy system structure including wind power generation system WT, photovoltaic system PV, power-to-gas system PtG, combined heat and power system CHP, energy storage system ESS energy units and load nodes; Step S2: Model the carbon emission intensity of the energy flow of each subsystem in step S1 to obtain a carbon emission intensity model. Calculate the node carbon emission intensity (NCI) of electricity, heat, and gas based on the real-time operation data of the system, and construct a piecewise carbon tax penalty function to quantify the carbon cost of the system. Step S3: Based on the integrated energy system structure constructed in step S1 and the carbon emission intensity model established in step S2, a two-layer two-stage Nash optimization model is designed; Step S4: Convert the two-layer two-stage Nash optimization model in step S3 into the Markov decision process (MDP) form, define the state space, action space, reward function, and state transition function of each subsystem, and prepare for subsequent agent training; Step S5: Construct an independent double-delayed deep deterministic policy gradient TD3 reinforcement learning agent for each subsystem in step S1, including WT agent, PV agent, PtG agent, CHP agent and ESS agent, and define their respective local state variables, behavior control actions and reinforcement learning objectives; Step S6: Use the multi-agent TD3 algorithm MATD3 to train each agent, and introduce the differential evolution perturbation mechanism DEM to improve strategy diversity and stability, forming an energy scheduling strategy network that takes into account both global benefits and local game; Step S7: Deploy the multi-agent scheduling strategy trained in step S6 in the actual integrated energy system to achieve real-time robust optimization scheduling control under factors such as renewable energy output fluctuations, electricity price changes, and carbon tax policy impacts, and output energy flow control instructions and carbon emission assessment reports for each time period.

2. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that The specific steps of step S2 are as follows: Step S21: Collect energy input and output data of each subsystem; Step S22: Calculate the net power load of the power system; For the power subsystem, the net power load in IES as follows: In the formula: For the power demand of IES, and are WT power generation and PV power generation respectively; Step S23: Select the carbon emission model calculation method based on whether there is electricity purchase behavior, and use different formulas to calculate NCI: when When <0, it means that the PV and WT in IES can fully support the power load, and the NCI of the power subsystem is as follows: Where, is the carbon emission rate of CHP, is the electrical power of CHP, when When > 0, the NCI of the power subsystem is as follows: Where, For the grid NCI, To buy electricity from the grid; Step S24: Constructing a carbon emission intensity model for the thermal subsystem: The carbon emissions of the source node as energy support satisfy the following formula: in, 、 、 and They are the energy of thermal power supplied to the grid, thermal power returned to the grid, gas supplied by the main grid and gas supplied by the power-to-gas system (PtG). 、 、 and They are the carbon emissions of grid-fed thermal power, grid-returned thermal power, main grid gas supply, and power-to-gas system gas supply system nodes; In the thermal subsystem, NCI satisfies the following formula: in, To supply network node temperature, The temperature of the resource recovery network node, and are the heat component matrices in the supply and return networks respectively; The NCI of the power supply node heating network is as follows: Step S25: Constructing a carbon emission intensity model for the gas combustion subsystem: The NCI of the gas subsystem is as follows: in, is the electrical conversion efficiency; Step S26: Establishing a penalty function based on carbon intensity segmentation intervals , use the segmented penalty term to regulate the energy usage structure: Where, is the carbon emission cost, in cents / kWh; is the length of the carbon quota interval; is the NCI of the relevant subsystem, kg / kWh; is the cost of carbon emissions, cents / kg; is the energy consumption of the relevant subsystem; and It is the penalty factor in the carbon emission tax.

3. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that The specific steps of step S3 are as follows: Step S31: Design a two-stage nested Nash optimization scheduling framework that integrates system operation and transaction costs: Where, is the system cost in the current state, The cost of buying gas for the system, The cost of buying coal for the system, The cost of purchasing electricity for the system; Step S32: Constructing a first-stage operation state optimization model, formulating an initial operation state decision plan for the integrated energy system under disturbance conditions and constructing an operation cost function, wherein: The state variables are defined as follows: Where: Control charging and discharging of energy storage systems; is the operating power of the system; are the heat, electricity and gas power support values ​​respectively; The operating cost function is: Step S33: Based on the decision results of the first stage, the uncertain disturbances in the main grid energy purchase process are further considered to construct a second-stage energy procurement robust optimization model and design a Min-Max energy procurement robust optimization mechanism for the uncertain worst-case scenario, where: Perturbation variable set: Where: and and represent electricity demand and natural gas demand in the integrated energy system respectively; The cost of electricity purchase is defined as follows: Where, and It is the real-time electricity selling and buying price. and represent the electricity output surplus and electricity demand in the integrated energy system respectively; The cost of purchasing gas is defined as follows: Where, and They represent the real-time price of selling gas to the main gas grid and the price of purchasing gas from the main grid. and They are used to represent the gas output surplus and gas load demand in the integrated energy system respectively; The cost of purchasing coal is defined as follows: Where, Indicates the real-time price of coal purchased from the main power grid; Step S34: Construct a first-layer internal load optimization model and design a multi-energy net load optimization strategy for electricity, heat, and gas based on a flexible load adjustment mechanism. Through flexible optimization of electric load, heat load, and gas load, coordinated control of load balance and resource consumption is achieved, wherein: The net electric load expression is as follows: Where, and Represent fixed power load and adjustable power load respectively; The net heat load expression is as follows: Where, and represent fixed heat load and adjustable heat load respectively; The net gas load expression is as follows: Where, and represent fixed heat load and adjustable heat load respectively; Step S35: Construct a second-layer energy purchase strategy optimization model, formulate a dynamic game strategy for energy purchase and sale between the system and the main grid to maximize operating benefits, and formulate the main grid energy purchase / sale strategy through the Nash game form to optimize the overall economic efficiency of the system, where: The total cost of purchased energy is expressed as follows: 。 4. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that The specific steps of step S4 are as follows: Step S41: Model each energy subsystem in the system as an independent intelligent agent; Step S42: define the state space S including load, electricity price, carbon intensity, and wind and solar energy resource state variables; Step S43: defining action space A including electricity purchase, output dispatch, and energy storage system charging and discharging behaviors; Step S44: constructing a reward function as the negative of the operating cost and the carbon cost as the optimization target; Step S45: defining a state transfer function to be dynamically generated based on the environmental response.

5. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that The specific steps of step S5 are as follows: Step S61: Build a multi-agent reinforcement learning framework and define the local state and action space; Step S62: Setting up a centralized evaluation mechanism and building a dual-critic network for bias suppression; Step S63: define a global reward function and build a global collaborative reward mechanism; Step S64: Introducing the differential evolution perturbation mechanism DEM to improve the efficiency of strategy exploration; Step S65: Complete the strategy training and deploy it to the real IES scheduling environment to perform scheduling control.

6. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 5 is characterized in that The specific steps of step S61 are as follows: Based on the double-delayed deep deterministic policy gradient TD3, a multi-agent deep reinforcement learning MADRL system is constructed. For each subsystem in IES, an independent TD3 agent is configured. Each agent uses its local state as input to make action decisions. The action space of each agent is Defined as: Where, It represents the local decision action of the i-th agent at time t, and the action corresponds to the energy input or output adjustment instruction of the subsystem.

7. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 5 is characterized in that The global reward function is as follows: Where: is the local immediate reward of the i-th agent; is the weight of each agent; is the current operating cost of the system; is the reward discount factor. This mechanism ensures the coordination between individual learning goals and the overall operating benefits of the system.

8. The energy planning method for a low-carbon economy integrated energy system based on multi-agent deep reinforcement learning according to claim 1 is characterized in that The specific steps of step S7 are as follows: Step S71: Loading the trained policy models of each subsystem in the actual deployment of IES; Step S72: collecting system status information in real time; Step S73: input the state to the multi-agent energy scheduling strategy network and output the optimal action instruction; Step S74: Execute electricity purchase and sales, charging and discharging, fuel scheduling, and energy storage system charging and discharging operations according to the agent's actions; Step S75: Output includes energy scheduling instructions for each time period and corresponding node NCI evaluation values.

Images