Comprehensive energy system microgrid group optimization method, system and related equipment

Abstract

The invention discloses an integrated energy system micro-grid group optimization method and system and related equipment. The method comprises the following steps: collecting observation data of an operator agent and a micro-grid agent; on the basis of the collected observation data of the operator intelligent agent and each micro-grid intelligent agent, alternately training an action network and an evaluation network of a double-layer intelligent agent, and repeatedly iterating to obtain a stably converged operator intelligent agent and a plurality of micro-grid intelligent agents; on the basis of the one-master multi-slave double-layer game model, a micro-grid group autonomous collaborative optimization model considering the multi-agent game is constructed; based on the state space, the action space, the environment and the reward function and the micro-grid group autonomous collaborative optimization model considering the multi-agent game, a micro-grid group autonomous collaborative optimization model based on deep reinforcement learning under the multi-agent environment is constructed; and solving the micro-grid group autonomous collaborative optimization model based on multi-agent deep reinforcement learning to obtain a micro-grid group optimization scheme. The method is beneficial to protecting the privacy of each micro-grid.

Description

Technical Field

[0001] The present invention relates to the field of microgrid group optimization and scheduling, and in particular to an integrated energy system microgrid group optimization method and related equipment. Background Art

[0002] Currently, a large number of distributed power sources are integrated into the distribution network, enriching its dispatching resources and increasing its operational flexibility. As the proportion of distributed power sources continues to increase, the uncertainty of renewable energy output exacerbates fluctuations in grid operation and exacerbates system voltage and current overshoots. To address the conflict between the security and stability of the large power grid and the high penetration of distributed power sources, the concept of microgrids has emerged. With the large-scale development of microgrids, multiple geographically adjacent microgrids are interconnected to form microgrid clusters. By sharing energy among microgrids, the utilization rate of renewable energy can be further improved, enabling more efficient energy transfer and coordination.

[0003] Currently, centralized optimization methods are primarily used to address the collaborative optimization of microgrid clusters. However, with the integration of large-scale distributed power generation equipment and the interconnectedness of multiple energy devices, the operational complexity of microgrid systems has increased significantly. The intermittent output of renewable energy sources and the random nature of user energy consumption lead to various uncertainties in microgrid optimization scheduling, making traditional centralized optimization algorithms increasingly difficult to solve. Furthermore, the market landscape for energy services is becoming increasingly diversified, with an increasing number of market players participating in grid operations based on their own strengths and market demands. This has led to increasingly complex game-playing among multiple stakeholders in microgrids, posing significant challenges to data privacy and security. Summary of the Invention

[0004] In response to the data privacy security issues in the existing autonomous collaborative optimization of microgrid groups considering multi-agent game interaction, the present invention proposes a microgrid group optimization method, system and related equipment for an integrated energy system. The method is based on hierarchical game deep reinforcement learning for autonomous collaborative optimization of microgrid groups, which is conducive to protecting the privacy of each microgrid.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: In a first aspect, the present invention provides a method for optimizing a microgrid cluster in an integrated energy system, comprising: Collect observation data from operator agents and microgrid agents; Based on the collected observation data of the operator agent and each microgrid agent, the action network and evaluation network of the two-layer agent are alternately trained and iterated repeatedly to obtain a stable convergence operator agent and multiple microgrid agents. Based on the one-master-multiple-slave two-layer game model, combined with the constraints of the operator agent and multiple microgrid agents, a microgrid cluster autonomous collaborative optimization model considering multi-agent game is constructed; based on the state space, action space, environment and reward function and the microgrid cluster autonomous collaborative optimization model considering multi-agent game, a microgrid cluster autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment is constructed; The microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning is solved to obtain a microgrid cluster optimization solution.

[0006] As a further improvement of the present invention, the observation data of the operator intelligent body and the microgrid intelligent body include: load data, actual output power of renewable energy, price per unit volume of natural gas, low calorific value of natural gas, energy conversion efficiency of micro gas turbines and charging and discharging efficiency of energy storage equipment, and random charging power of electric vehicles.

[0007] As a further improvement of the present invention, the one-master-multiple-slave two-layer game model includes: First, a microgrid equipment model is established: the integrated energy system is divided into multiple microgrids, each of which includes photovoltaics, wind turbines, micro gas turbines, battery energy storage systems, base loads, and electric vehicle charging loads; A two-tiered game model for active energy management in microgrids, with one master and multiple followers, is established. The model consists of two layers, one in a sequential order and with interactive iteration. The operator DSO, as the leader, aims to maximize its own revenue; the microgrid, as a follower, aims to minimize its own output costs. Determine the objective function and constraints: the objective function of the DSO is to maximize its own electricity sales revenue, and the objective function of each microgrid is to minimize its own operating costs; the equality constraint is the power balance constraint, and the system's inequality constraints include the upper and lower limits of gas turbine output power, energy storage equipment operation constraints, electric vehicle load adjustment constraints, unit climbing constraints, and interconnection line transmission power constraints.

[0008] As a further improvement of the present invention, the above-mentioned two-layer game model with one master and multiple slaves is based on the constraints of the operator agent and multiple microgrid agents to construct a microgrid group autonomous collaborative optimization model considering multi-agent game, including: As a leader, DSO aims to maximize its own benefits. The objective function is as follows:

[0009] Where:

[0010]

[0011] Where: 、 are the revenues from electricity trading between the DSO and the main grid and microgrid cluster respectively; and They are t The amount of electricity sold and purchased by DSO to the main grid during this period, and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; N is the number of microgrids; For each microgrid agent, the objective function is to minimize the operating cost of each microgrid within the cycle. The objective function is:

[0012] Where:

[0013]

[0014]

[0015] Where: T is the total scheduling time period, 、 、 They are the microgrid’s transaction costs, fuel costs, and energy storage equipment operation and maintenance costs; and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; is the price per unit volume of natural gas, Hi is the heat released per unit volume of gas, for t The electric power output of the gas turbine during the period, The energy conversion efficiency of the gas turbine; is the converted unit charge and discharge cost, and They are t The charging and discharging power of the energy storage device during the time period, and are charge and discharge efficiency respectively.

[0016] As a further improvement of the present invention, the constraints of the operator agent and the plurality of microgrid agents include: Constraints are divided into system power balance constraints, transaction price and power constraints, and microgrid unit operation constraints; The power balance constraints that need to be met within each microgrid and the DSO are:

[0017]

[0018] Where: and They are t Time period microgrid n The amount of electricity sold or purchased from DSOs; and Microgrid exist t The output of wind turbines and photovoltaics during the period, for t The output of the gas turbine during the period, For microgrids n exist t Active load during the period, 、 for t Discharging and charging power of energy storage equipment during time periods; and They are t The amount of electricity sold or purchased by DSO from the main grid during the period; The transaction price and power constraints are:

[0019] Where: and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period;

[0020]

[0021]

[0022] Where: and They are t The amount of electricity sold and purchased by the microgrid to the DSO during the period, 、 are the upper and lower limits of the tie line power between the microgrid and the DSO, and are the average prices of electricity sales and electricity purchase respectively; Microgrid unit operation constraints include gas turbine operation constraints and energy storage device operation constraints:

[0023]

[0024] Where: For gas turbine equipment t Output power during the time period, and They are the upper and lower limits of the output allowed for gas turbine equipment respectively; and Micro gas turbines in t Time period and t-1 Output power during the time period, is the maximum value of the climbing power in one scheduling period; Energy storage devices have two states: charging and discharging. Constraints include upper and lower limits of charge, state of charge, and charge and discharge power constraints. Specifically:

[0025]

[0026]

[0027]

[0028] Where: is the rated capacity of the energy storage unit, 、 Energy storage equipment in t Time period and t+1 The state of charge during the time period, and are the charging and discharging conversion efficiency of energy storage, 、 for t Charging and discharging power during the time period; 、 The upper and lower limits of the state of charge allowed by the system; It is the maximum charging and discharging power of the energy storage device.

[0029] As a further improvement of the present invention, the autonomous collaborative optimization model of the microgrid group based on the state space, action space, environment and reward function and the multi-agent game is constructed to obtain the autonomous collaborative optimization model of the microgrid group based on deep reinforcement learning in a multi-agent environment, including: The state space is composed of the state space of the master agent and multiple slave agents. In each period, the state space of the DSO master agent includes the microgrid n Electricity purchase from DSO and electricity sales , the electricity purchase price of DSO to the main grid and electricity sales price ,for

[0030] In each period, each microgrid includes the user load of each microgrid from the state space of the intelligent agent , photovoltaic unit output , wind turbine output , the charging power of electric vehicle load , the microgrid's electricity purchase and sales price to DSO and And the energy storage charge status in the previous period and the gas turbine output power ,for

[0031] The action space of the DSO master agent includes its microgrid-oriented n Electricity purchase and sales price and , and sent to each microgrid, i.e.

[0032] Each microgrid dispatches the output of the equipment in each microgrid from the action space of the intelligent agent, taking into account the charging and discharging power of the energy storage and , the output of the micro gas turbine unit , the charging power of electric vehicle load after electricity price guidance ,Right now

[0033] Each agent takes action based on its current state, interacts with the environment, receives a reward, and transitions to the next state. This action is evaluated and used to guide the next agent's action. Constraints are established in the operator-microgrid group master-slave game model as the environment. At each moment, after each agent takes an action, the DSO's revenue and the operating costs of each microgrid are calculated. The state is fed back to calculate the reward function and transition to the next state, repeating the cycle. For DSO agents, the objective function is directly As an immediate reward, if the DSO does not satisfy the constraints, a penalty coefficient k is set s 、k b and penalty value , together with the immediate reward, serves as the agent's final reward function ; The calculation is as follows:

[0034]

[0035] For each microgrid agent, the cost function of the microgrid The opposite number of is used as the instant reward of the microgrid. If a microgrid does not meet the constraints, the penalty coefficient k is set soc 、k EV and penalty value , together with the immediate reward, serves as the final reward function of the corresponding agent , calculated as follows:

[0036] .

[0037] As a further improvement of the present invention, the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning is solved by using the MADDPG algorithm according to the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning; The MADDPG algorithm generates an action using the Actor strategy network in the current state, and inputs the action and the current state into the Critic evaluation network to calculate Q value, get Q The action with the largest value; among them, the parameters of the Critic network are updated by minimizing the loss value ; Loss function when critic network is updated The calculation formula is as follows:

[0038]

[0039] Where: , are the action state value functions of the current Critic network and the target Critic network respectively; To obtain the reward, is the discount factor; Combined with Critic network Q Value, calculate the policy gradient of the Actor network update parameter:

[0040] Where: is the Actor network action strategy function, is the current Critic network action state value function.

[0041] As a further improvement of the present invention, the solving of the microgrid group autonomous collaborative optimization model based on multi-agent deep reinforcement learning also includes: The soft update mechanism is used to update only some parameters during each learning process. The method is as follows:

[0042]

[0043] Where: is the update rate, its value is less than 1; and They are the parameters of the target Critic network and target Actor network of the agent respectively.

[0044] As a further improvement of the present invention, the microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning is solved to obtain a microgrid cluster optimization solution, including: Initialize parameters and environment of each agent: set optimization scheduling period T And set the number of training rounds for each agent M , the initial values ​​are all set to 1, and the agent network parameters are randomly set at the beginning ; Load the microgrid cluster autonomous collaborative optimization model into the environment and set the interface file with the state and action in the MADDPG algorithm; The leader DSO selects actions and updates network parameters: In the leader update round, it first selects actions based on the state information extracted from the microgrid and distribution network. Select your own action , after executing this action, the reward calculated based on the action of the microgrid agent , and update the network parameters of the master agent according to formulas (31)-(35), and judge whether the set upper limit of the leader update round is reached. If it is satisfied, it enters the follower update round. If not, the leader continues training; Each follower microgrid selects an action and updates the network parameters: In the follower microgrid update round, each microgrid acts as a follower according to the current state Select each action , perform actions and calculate rewards , and update the network parameters of multiple slave agents to determine whether the set upper limit of follower update rounds is reached. If it is met, it will enter the next step. If not, the follower will continue training; Determine whether the leader and follower reward curves both meet the convergence conditions. If so, end the training, output the results and save them. Otherwise, go to the leader DSO to select actions and update network parameters, and perform a new round of iteration until the convergence requirements are met.

[0045] In a second aspect, the present invention provides a microgrid cluster optimization system for an integrated energy system, comprising: The collection module is used to collect observation data from the operator agent and the microgrid agent; The training module is used to alternately train the action network and evaluation network of the two-layer agent based on the collected observation data of the operator agent and each microgrid agent, and iterate repeatedly to obtain the operator agent and multiple microgrid agents; A construction module is used to construct a microgrid cluster autonomous collaborative optimization model considering multi-agent game based on a two-layer game model with one master and multiple slaves, combined with the constraints of the operator agent and multiple microgrid agents; a microgrid cluster autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment is constructed based on the state space, action space, environment and reward function and the microgrid cluster autonomous collaborative optimization model considering multi-agent game; The solution module is used to solve the microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning to obtain a microgrid cluster optimization solution.

[0046] As a further improvement of the present invention, the observation data of the operator intelligent body and the microgrid intelligent body include: load data, actual output power of renewable energy, price per unit volume of natural gas, low calorific value of natural gas, energy conversion efficiency of micro gas turbines and charging and discharging efficiency of energy storage equipment, and random charging power of electric vehicles.

[0047] As a further improvement of the present invention, the one-master-multiple-slave two-layer game model includes: First, a microgrid equipment model is established: the integrated energy system is divided into multiple microgrids, each of which includes photovoltaics, wind turbines, micro gas turbines, battery energy storage systems, base loads, and electric vehicle charging loads; A two-tiered game model for active energy management in microgrids, with one master and multiple followers, is established. The model consists of two layers, one in a sequential order and with interactive iteration. The operator DSO, as the leader, aims to maximize its own revenue; the microgrid, as a follower, aims to minimize its own output costs. Determine the objective function and constraints: the objective function of the DSO is to maximize its own electricity sales revenue, and the objective function of each microgrid is to minimize its own operating costs; the equality constraint is the power balance constraint, and the system's inequality constraints include the upper and lower limits of gas turbine output power, energy storage equipment operation constraints, electric vehicle load adjustment constraints, unit climbing constraints, and interconnection line transmission power constraints.

[0048] As a further improvement of the present invention, the building blocks are specifically used for: As a leader, DSO aims to maximize its own benefits. The objective function is as follows:

[0049] Where:

[0050]

[0051] Where: 、 are the revenues from electricity trading between the DSO and the main grid and microgrid cluster respectively; and They are t The amount of electricity sold and purchased by DSO to the main grid during this period, and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; N is the number of microgrids; For each microgrid agent, the objective function is to minimize the operating cost of each microgrid within the cycle. The objective function is:

[0052] Where:

[0053]

[0054]

[0055] Where: T is the total scheduling time period, 、 、 They are the microgrid’s transaction costs, fuel costs, and energy storage equipment operation and maintenance costs; and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; is the price per unit volume of natural gas, Hi is the heat released per unit volume of gas, for t The electric power output of the gas turbine during the period, The energy conversion efficiency of the gas turbine; is the converted unit charge and discharge cost, and They are t The charging and discharging power of the energy storage device during the time period, and are charge and discharge efficiency respectively.

[0056] As a further improvement of the present invention, the constraints of the operator agent and the plurality of microgrid agents include: Constraints are divided into system power balance constraints, transaction price and power constraints, and microgrid unit operation constraints; The power balance constraints that need to be met within each microgrid and the DSO are:

[0057]

[0058] Where: and They are t Time period microgrid n The amount of electricity sold or purchased from DSOs; and Microgrid exist t The output of wind turbines and photovoltaics during the period, for t The output of the gas turbine during the period, For microgridsn exist t Active load during the period, 、 for t Discharging and charging power of energy storage equipment during time periods; and They are t The amount of electricity sold or purchased by DSO from the main grid during the period; The transaction price and power constraints are:

[0059] Where: and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period;

[0060]

[0061]

[0062] Where: and They are t The amount of electricity sold and purchased by the microgrid to the DSO during the period, 、 are the upper and lower limits of the tie line power between the microgrid and the DSO, and are the average prices of electricity sales and electricity purchase respectively; Microgrid unit operation constraints include gas turbine operation constraints and energy storage device operation constraints:

[0063]

[0064] Where: For gas turbine equipment t Output power during the time period, and They are the upper and lower limits of the output allowed for gas turbine equipment respectively; and Micro gas turbines in t Time period and t-1 Output power during the time period, is the maximum value of the climbing power in one scheduling period; Energy storage devices have two states: charging and discharging. Constraints include upper and lower limits of charge, state of charge, and charge and discharge power constraints. Specifically:

[0065]

[0066]

[0067]

[0068] Where: is the rated capacity of the energy storage unit, 、 Energy storage equipment in t Time period and t+1 The state of charge during the time period, and are the charging and discharging conversion efficiency of energy storage, 、 for t Charging and discharging power during the time period; 、 The upper and lower limits of the state of charge allowed by the system; It is the maximum charging and discharging power of the energy storage device.

[0069] As a further improvement of the present invention, the building blocks are specifically used for: The state space is composed of the state space of the master agent and multiple slave agents. In each period, the state space of the DSO master agent includes the microgrid n Electricity purchase from DSO and electricity sales , the electricity purchase price of DSO to the main grid and electricity sales price ,for

[0070] In each period, each microgrid includes the user load of each microgrid from the state space of the intelligent agent , photovoltaic unit output , wind turbine output , the charging power of electric vehicle load , the microgrid's electricity purchase and sales price to DSO and And the energy storage charge status in the previous period and the gas turbine output power ,for

[0071] The action space of the DSO master agent includes its microgrid-oriented n Electricity purchase and sales price and , and sent to each microgrid, i.e.

[0072] Each microgrid dispatches the output of the equipment in each microgrid from the action space of the intelligent agent, taking into account the charging and discharging power of the energy storage and , the output of the micro gas turbine unit , the charging power of electric vehicle load after electricity price guidance ,Right now

[0073] Each agent takes action based on its current state, interacts with the environment, receives a reward, and transitions to the next state. This action is evaluated and used to guide the next agent's action. Constraints are established in the operator-microgrid group master-slave game model as the environment. At each moment, after each agent takes an action, the DSO's revenue and the operating costs of each microgrid are calculated. The state is fed back to calculate the reward function and transition to the next state, repeating the cycle. For DSO agents, the objective function is directly As an immediate reward, if the DSO does not satisfy the constraints, a penalty coefficient k is set s 、k b and penalty value , together with the immediate reward, serves as the agent's final reward function ; The calculation is as follows:

[0074]

[0075] For each microgrid agent, the cost function of the microgrid The opposite number of is used as the instant reward of the microgrid. If a microgrid does not meet the constraints, the penalty coefficient k is set soc 、k EV and penalty value , together with the immediate reward, serves as the final reward function of the corresponding agent , calculated as follows:

[0076] .

[0077] As a further improvement of the present invention, the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning is solved by using the MADDPG algorithm according to the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning; The MADDPG algorithm generates an action using the Actor strategy network in the current state, and inputs the action and the current state into the Critic evaluation network to calculate Q value, get Q The action with the largest value; among them, the parameters of the Critic network are updated by minimizing the loss value ; Loss function when critic network is updated The calculation formula is as follows:

[0078]

[0079] Where: , are the action state value functions of the current Critic network and the target Critic network respectively; To obtain the reward, is the discount factor; Combined with Critic network Q Value, calculate the policy gradient of the Actor network update parameter:

[0080] Where: is the Actor network action strategy function, is the current Critic network action state value function.

[0081] As a further improvement of the present invention, the solution module is specifically used to: The soft update mechanism is used to update only some parameters during each learning process. The method is as follows:

[0082]

[0083] Where: is the update rate, its value is less than 1; and They are the parameters of the target Critic network and target Actor network of the agent respectively.

[0084] As a further improvement of the present invention, the microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning is solved to obtain a microgrid cluster optimization solution, including: Initialize parameters and environment of each agent: set optimization scheduling period T And set the number of training rounds for each agent M , the initial values ​​are all set to 1, and the agent network parameters are randomly set at the beginning ; Load the microgrid cluster autonomous collaborative optimization model into the environment and set the interface file with the state and action in the MADDPG algorithm; The leader DSO selects actions and updates network parameters: In the leader update round, it first selects actions based on the state information extracted from the microgrid and distribution network. Select your own action , after executing this action, the reward calculated based on the action of the microgrid agent , and update the network parameters of the master agent according to formulas (31)-(35), and judge whether the set upper limit of the leader update round is reached. If it is satisfied, it enters the follower update round. If not, the leader continues training; Each follower microgrid selects an action and updates the network parameters: In the follower microgrid update round, each microgrid acts as a follower according to the current state Select each action , perform actions and calculate rewards , and update the network parameters of multiple slave agents to determine whether the set upper limit of follower update rounds is reached. If it is met, it will enter the next step. If not, the follower will continue training; Determine whether the leader and follower reward curves both meet the convergence conditions. If so, end the training, output the results and save them. Otherwise, go to the leader DSO to select actions and update network parameters, and perform a new round of iteration until the convergence requirements are met.

[0085] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the integrated energy system microgrid optimization method when executing the computer program.

[0086] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program, which implements the integrated energy system microgrid optimization method when executed by a processor.

[0087] In a fifth aspect, the present invention provides a computer program product, which includes computer instructions, and the computer instructions instruct a computer to execute the integrated energy system microgrid cluster optimization method.

[0088] The beneficial effects of the present invention compared to the prior art are: This invention considers the diverging interests and game-playing mechanisms between operators and microgrids. By constructing a master-slave game model between operators and microgrids and solving for Nash equilibrium points, it achieves a balanced distribution of benefits for all parties involved. The optimization algorithm based on deep reinforcement learning is deeply aligned with this master-slave game model. Each agent iteratively updates its action network and evaluation network through training, enabling rapid and stable solution of Nash equilibrium points in online applications. Due to the "centralized training-distributed execution" nature of multi-agent systems, after training, each microgrid can make optimal decisions based solely on its own local observations, which helps protect the privacy of each microgrid. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present invention or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0090] Figure 1 The grid topology used for the microgrid collaborative optimization method based on multi-agent deep reinforcement learning; Figure 2 This is a flowchart of the microgrid collaborative optimization method based on multi-agent deep reinforcement learning; Figure 3 Flowchart for solving the overall algorithm of the microgrid cluster optimization model based on alternating training MADDPG; Figure 4 A microgrid group optimization system for an integrated energy system provided by the present invention; Figure 5 This is a schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0091] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0092] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, and connecting should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0093] In order to explain the technical solution of the present invention in detail, the following describes the content of the present invention in detail with reference to the accompanying drawings: Based on the power grid optimization dispatching method, the present invention provides a microgrid optimization method for an integrated energy system, such as Figure 2 As shown; specifically including the following steps: S01, collect observation data of operator agents and microgrid agents; Observation data of the operator intelligent agent and the microgrid intelligent agent are collected. The observation data involved in the present invention include load data, actual output power of renewable energy, price per unit volume of natural gas, low calorific value of natural gas, energy conversion efficiency of micro gas turbines, charging and discharging efficiency of energy storage equipment, and random charging power of electric vehicles.

[0094] As a specific solution, the present invention's microgrid cluster optimization method based on hierarchical game reinforcement learning collects observation data from each agent region. The data involved mainly includes renewable energy unit output data, thermal power unit output data, load data, energy storage charge state, and time-of-use electricity prices set by the upper power grid. S02, based on the collected observation data of the operator agent and each microgrid agent, alternately train the action network and evaluation network of the two-layer agent and iterate repeatedly to obtain a stably converged operator agent and multiple microgrid agents; S03, based on a two-layer game model of one master and multiple slaves, combined with the constraints of the operator agent and multiple microgrid agents, a microgrid cluster autonomous collaborative optimization model considering multi-agent game is constructed; based on the state space, action space, environment and reward function and the microgrid cluster autonomous collaborative optimization model considering multi-agent game, a microgrid cluster autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment is constructed; Specifically, a two-tiered game model for active energy management in a microgrid with one master and multiple followers is constructed. The operator, as the leader, aims to maximize its own revenue; each microgrid, as a follower, aims to minimize its own output costs. Constraints are established for the microgrid cluster optimization model. The equality constraint is a power balance constraint, while the inequality constraints include time-of-use electricity price constraints, upper and lower limits on gas turbine output power, energy storage device operation constraints, unit ramping constraints, tie line transmission power constraints, and electric vehicle charging load adjustment constraints.

[0095] As a specific solution, a distribution system operator (DSO) model and a microgrid cluster autonomous collaborative optimization model are constructed based on observation data. The DSO and microgrid cluster respectively take profit maximization and cost minimization as optimization goals, and the safe and stable operation of the system as a constraint.

[0096] First, establish a microgrid device model: the microgrid model constructed by the present invention refers to Figure 1 , taking the IEEE38-node standard system as an example, it is divided into three microgrids. Each microgrid mainly includes photovoltaic (PV), wind turbine (WT), micro gas turbine (MT), battery energy storage system (ES), electric vehicle (EV) and load.

[0097] Secondly, a master-slave game model between operators and microgrids is established: a two-tiered game model for active energy management in microgrids, with one master and multiple slaves, is established. The game interaction can be divided into two layers, with a sequential and iterative relationship. As the leader, the DSO aims to maximize its own revenue and reduce peak-to-valley load variations. As the follower, the microgrid aims to minimize its own output costs and optimize the output of distributed controllable power sources while meeting the operating constraints of various devices.

[0098] Based on a constructed microgrid cluster autonomous collaborative optimization model that considers multi-agent game theory, the objective function and constraints of the optimization model are determined: the DSO's objective function is to maximize its own electricity sales revenue, and the objective function of each microgrid is to minimize its own operating costs. The system's equality constraints are power balance constraints, and the system's inequality constraints mainly include upper and lower limits on gas turbine output power, energy storage device operation constraints, unit ramping constraints, tie line transmission power constraints, and electric vehicle charging load adjustment constraints.

[0099] The present invention additionally considers the electric vehicle load at the model level and establishes relevant constraints and penalty terms; at the algorithm level, it proposes an iterative mechanism of alternating training of double-layer reinforcement learning, which greatly shortens the convergence time and improves the convergence accuracy.

[0100] Furthermore, the autonomous collaborative optimization method of microgrid clusters based on multi-agent deep reinforcement learning includes the following steps: Observation data from the DSO master agent and each microgrid slave agent is collected. The observation data of the DSO master agent involved in the present invention includes: the amount of electricity exchanged between the DSO and each microgrid, and the purchase and sale electricity prices set by the upper power grid; the observation data of each microgrid slave agent includes: renewable energy output, user load, real-time electricity prices, energy storage charge status, and electric vehicle instant charging capacity. The DSO agent determines the market electricity price for each time period of the next day based on the benchmark load data, and formulates a strategy based on the incentive adjustment to ensure the coordinated operation of the microgrid source and load in each time period of the next day. After receiving the market electricity price quotation from the operator agent, each microgrid agent feeds back to the agent an optimized scheduling plan that takes into account the user's electricity cost under the current quotation, and arranges the output plan of each controllable generator set.

[0101] Based on the basic elements of multi-agent deep reinforcement learning, the autonomous collaborative optimization model of microgrid groups considering multi-agent games is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function, and is solved using MADDPG.

[0102] The solution is achieved using computer software. By comparing it with a centralized optimization solution, it is verified that the trained model not only has a good ability to cope with uncertainty, but also can realize real-time decision-making of the system, which is conducive to online application. By comparing it with a single-agent deep reinforcement learning method, it is verified that the MADDPG algorithm has a faster convergence speed and better training results during the training process, and has obvious advantages in dealing with complex environmental problems.

[0103] As a specific solution, the MADDPG algorithm employed in this invention adopts a "centralized training, distributed execution" model. Each agent is equipped with a separate actor and critic network. Unlike DDPG, during training, each agent's actor takes actions based on its own state, and the critic evaluates the actor's actions. The actor updates its own strategy based on the feedback. Each agent's critic obtains more accurate evaluation information by estimating the strategies of other agents. After training, each agent only needs to use the actor to take actions based on its own state. At this point, no information about other agents is required, and each agent makes decisions independently. MADDPG learns the optimal strategy through centralized training, and only local information is required for application, effectively protecting the privacy of each agent.

[0104] As a specific solution, in the microgrid group autonomous collaborative optimization method based on multi-agent deep reinforcement learning of the present invention, the dual uncertainty of renewable energy and load is taken into account in the model training process, and the uncertainty is superimposed on the input data during the model training process. That is, in the MADDPG training process, the master agent and the slave agent need to obtain observable status data within their own area. At the moment of reading the renewable energy and load data, an unknown data, that is, the fluctuation range, is superimposed on the obtained data itself to make it obey the normal distribution. Therefore, it can be guaranteed that no matter how many rounds of training are performed, the renewable energy and load data obtained each time are different, but are within the fluctuation range, and the size of the fluctuation range can be flexibly adjusted.

[0105] Furthermore, the microgrid cluster autonomous collaborative optimization model considering multi-agent game is converted into a microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning in a multi-agent environment according to the state space, action space, environment and reward function; As a specific solution, based on the basic elements of multi-agent deep reinforcement learning, the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) method is adopted to design the model into a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function.

[0106] S04: Solve the microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning to obtain a microgrid cluster optimization solution.

[0107] Specifically, according to the constructed microgrid group autonomous collaborative optimization model based on multi-agent deep reinforcement learning, the alternating training MADDPG algorithm is used to solve it, and the solution process is given.

[0108] S05 is solved using computer software. By comparing it with the centralized optimization solution, it is verified that the trained model not only has a good ability to deal with uncertainty, but also can realize real-time decision-making of the system, which is conducive to online application. By comparing it with the single-agent deep reinforcement learning method, it is verified that the MADDPG algorithm has a faster convergence speed and better training results during the training process, and has obvious advantages in dealing with complex environmental problems.

[0109] During the specific implementation, considering that there is a conflict of interest between operators and various microgrid game entities, it is difficult to balance the fairness of interest distribution and the safety of grid operation. It is suitable for the collaborative optimization of microgrid groups under complex modeling and multi-agent game situations, and is easy to be recognized by relevant staff.

[0110] Compared with centralized optimization and traditional distributed optimization methods, the microgrid cluster autonomous collaborative optimization method based on layered game deep reinforcement learning provided by the present invention has the following advantages: (1) The use of multi-agent hierarchical game theory can fully consider the differences in interests among the various agents in the microgrid group, achieve coordinated interaction among the multiple agents, and achieve fair distribution of interests among the game parties by solving the Nash equilibrium point; (2) In response to the frequent oscillation and difficulty in convergence of the reward function caused by the double-layer master-slave game in the multi-agent environment, an iterative mechanism of double-layer reinforcement learning alternating training is proposed, which greatly shortens the convergence time and improves the convergence accuracy. (3) Based on the "centralized training-distributed execution" characteristics of multi-agents, after training, each agent in the microgrid group only needs to make decisions on the controllable units based on its own local observation values, which is conducive to protecting the privacy of each microgrid.

[0111] In order to achieve the above purpose, refer to Figure 2 , is a flow chart of the present invention's autonomous collaborative optimization method for microgrid clusters based on multi-agent deep reinforcement learning. The autonomous collaborative optimization method for microgrid clusters based on multi-agent deep reinforcement learning includes the following steps: Step 1: Import observation data for each agent region.

[0112] First, collect the observation data of the DSO agent and the microgrid group agent. The observation data of the present invention includes the basic load data of each microgrid. , the actual output power of renewable energy and , the price per unit volume of natural gas , energy conversion efficiency of micro gas turbine , the charging and discharging efficiency of energy storage equipment and , load data of electric vehicle random charging , Electric vehicles can adjust the ratio δ max .

[0113] (1) Step 2: Construct a microgrid cluster autonomous collaborative optimization model considering multi-agent game.

[0114] Step 2.1: Construct the operator-microgrid group master-slave game model; A two-tiered game model for active energy management in a microgrid with one master and multiple followers is constructed. The game interaction can be divided into two layers, one in sequence and the other in an iterative and interactive manner. As a leader, the DSO aims to maximize its own revenue and reduce peak-to-valley load variations. Its strategy is to publish market electricity prices to users, guiding each microgrid to adjust its strategy based on its own needs. As followers, each microgrid aims to minimize its own output costs. Its strategy is to optimize the load that responds to the published electricity prices and the output of distributed power sources. While satisfying the operating constraints of various devices, this strategy optimizes the output of distributed controllable power sources, reducing the operating costs of distributed power sources while ensuring source-load coordination within the microgrid.

[0115] Step 2.2: Establish the objective functions of the operator and the microgrid group respectively; As a leader, the DSO aims to maximize its own revenue and reduce load peak-valley differences. The system's objective function is as follows: (2) In formula (2): (3) (4) Where: 、 are the revenues from electricity trading between the DSO and the main grid and microgrid cluster respectively; and They are t The amount of electricity sold and purchased by DSO to the main grid during this period, and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; N is the number of microgrids.

[0116] The goal of optimizing the scheduling of each microgrid is to adjust the scheduling plan based on the load characteristics of different microgrids and utilize the principle of energy complementarity to achieve reasonable energy interaction between different microgrids, thereby minimizing the total operating cost of the system. For each microgrid, the objective function of the optimized scheduling is to minimize the operating cost of each microgrid within the cycle: (5) In formula (5): (6) (7) (8) Where: T is the total scheduling time period, 、 、 They are the microgrid’s transaction costs, fuel costs, and energy storage equipment operation and maintenance costs; and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; is the price per unit volume of natural gas, Hi is the heat released per unit volume of gas, for t The electric power output of the gas turbine during the period, The energy conversion efficiency of the gas turbine; is the converted unit charge and discharge cost, and They are t The charging and discharging power of the energy storage device during the time period, and are charge and discharge efficiency respectively.

[0117] Step 2.3: Establish constraints for the microgrid optimization model; The constraints of the above-mentioned microgrid group autonomous collaborative optimization model considering multi-agent game can be divided into system power balance constraints, transaction price and power constraints, and microgrid unit operation constraints. The constraints are described as follows: 1) System power balance constraints: To ensure safe and stable system operation, each microgrid and DSO must meet the following power balance constraints: (9) (10) Where: and They are t Time period microgrid n The amount of electricity sold or purchased from DSOs; and Microgrid exist t The output of wind turbines and photovoltaics during the period, For microgrids n exist t The output of the gas turbine during the period, For microgrids nexist t Active load during the period, 、 For microgrids n exist t The discharge and charging power of the energy storage equipment during the time period, For microgrids n exist t Charging power of electric vehicle load after time-of-use electricity price guidance; and They are t The amount of electricity sold or purchased by DSO from the main grid during this period.

[0118] 2) Transaction electricity price and power constraints: In order to ensure that each microgrid can actively participate in transactions with the DSO, the electricity price set by the DSO should be within the strategy space defined by equation (11) and meet the average value constraints defined by equations (12) and (13) to ensure the rationality and effectiveness of the electricity price: (11) (12) (13) Where: and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; and are the average prices of electricity sales and electricity purchase, respectively.

[0119] The power constraints of the tie-line transactions between the microgrid and the DSO are as follows: (14) Where: and They are t The amount of electricity sold and purchased by the microgrid to the DSO during the period, 、 They are the upper and lower limits of the power of the tie line between the microgrid and the DSO, respectively.

[0120] 3) Microgrid unit operation constraints: Microgrid unit operation constraints include gas turbine operation constraints, energy storage equipment operation constraints, and electric vehicle load adjustment constraints. Gas turbine operation constraints include output power upper and lower limits and ramp constraints, as shown below: (15) (16) Where: For gas turbine equipment t Output power during the time period, and They are the upper and lower limits of the output allowed for gas turbine equipment respectively; and Micro gas turbines in t Time period and t-1 Output power during the time period, It is the maximum value of the climbing power in one scheduling period.

[0121] Energy storage devices have two states: charging and discharging. The related constraints include upper and lower limit constraints on charge, state of charge constraints, and charge and discharge power constraints, as shown below: (17) (18) (19) (20) Where: is the rated capacity of the energy storage unit, 、 Energy storage equipment in t Time period and t+1 The state of charge during the time period, and are the charging and discharging conversion efficiency of energy storage, 、 for t Charging and discharging power during the time period; 、 The upper and lower limits of the state of charge allowed by the system; It is the maximum charging and discharging power of the energy storage device.

[0122] Based on the guidance of time-of-use electricity prices, under the premise that the total charging demand of electric vehicles remains unchanged, a part of the electric vehicle load can be moved from the peak period to the flat and valley periods through optimized scheduling, thereby reducing the system's electricity purchase cost during the peak period. The constraints are as follows: (twenty one) (twenty two) Where: 、 They are t The charging power of electric vehicle loads during the period after random charging and time-of-use electricity price guidance; It is the upper limit of the adjustable ratio of electric vehicle load.

[0123] Step 3: Design a reinforcement learning model in a multi-agent environment.

[0124] Based on the basic elements of multi-agent deep reinforcement learning, the microgrid autonomous collaborative optimization model considering multi-agent game is designed as a reinforcement learning model in a multi-agent environment according to the state space, action space, environment and reward function. The details are as follows: 1) State space: The state space of the system is composed of the state space of the master agent and multiple slave agents. In each period, the state space of the DSO master agent includes the microgrid n Electricity purchase from DSO and electricity sales , the electricity purchase price of DSO to the main grid and electricity sales price ,Right now (twenty three) In each period, each microgrid includes the user load of each microgrid from the state space of the intelligent agent , photovoltaic unit output , wind turbine output , the charging power of electric vehicle load , the microgrid's electricity purchase and sales price to DSO and And the energy storage charge status in the previous period and the gas turbine output power ,Right now (twenty four) 2) Action space: The action space of the DSO master agent includes its microgrid-oriented n Electricity purchase and sales price and And send it to each microgrid, i.e. (25) Each microgrid dispatches the output of the equipment in each microgrid from the action space of the intelligent body. The present invention mainly considers the charging and discharging power of the energy storage. and , the output of the micro gas turbine unit , the charging power of electric vehicle load after electricity price guidance ,Right now (26) 3) Environment Design: Each agent's Actor network takes actions based on the state at that moment, interacts with the environment, receives rewards, and transfers to the state at the next moment. The Critic network evaluates the actions and guides the agent to take actions at the next moment. According to this process, the microgrid group optimization model (Equations (9)-(17)) is used as the environment. After each agent takes an action at each moment, the DSO's benefits and the operating costs of each microgrid are calculated. The relevant state quantities in the feedback system are used to calculate the reward function and transfer to the next moment, and the cycle continues iteratively.

[0125] 4) Reward function: The reward function will affect the convergence of the algorithm to a certain extent. Therefore, the reward signal setting needs to be able to convey the goal that the agent wants to achieve, thereby guiding the agent to improve its actions towards maximizing the reward function.

[0126] For the DSO agent, the objective function calculated by formula (2) is directly As an immediate reward, if the DSO does not satisfy the constraints (12)-(13), the penalty coefficient k is set s 、k b and penalty value , together with the immediate reward, serves as the agent's final reward function The calculation is as follows: (27) (28) For each microgrid agent, the cost function calculated by formula (5) The opposite number of is used as the instant reward of the microgrid. If a microgrid does not meet the constraints (19) and (22), the penalty coefficient k is set. soc 、k EV and penalty value , together with the immediate reward, serves as the agent's final reward function The calculation is as follows: (29) (30) Step 4: Use the MADDPG algorithm to solve.

[0127] For clarification: The MADDPG (Multi-Agent Deep Deterministic Policy Gradient) algorithm is an algorithm for multi-agent reinforcement learning environments. MADDPG is an extension of the Deep Deterministic Policy Gradient (DDPG) algorithm, specifically designed for multi-agent settings. DDPG combines deep learning and reinforcement learning for problems in continuous action spaces. By combining policy gradient methods with Q-learning (a value function approximation method), it learns a deterministic policy to solve complex decision-making problems. In MADDPG, each agent has an Actor (policy network) and a Critic (value network).

[0128] According to the above-mentioned microgrid group autonomous collaborative optimization model based on multi-agent deep reinforcement learning, the MADDPG algorithm is used to solve it. The MADDPG algorithm generates actions using the Actor strategy network in the current state, and inputs the action and the current state into the Critic evaluation network to calculate Q The goal is to find the value that Q The action with the largest value. In MADDPG, the parameters of the Critic network are updated by minimizing the loss value (mean square error loss) , the loss function when the Critic network is updated The calculation formula is as follows: (31) (32) Where: , are the action state value functions of the current Critic network and the target Critic network respectively; To obtain the reward, is the discount factor.

[0129] Combined with Critic network Q Value, can calculate the policy gradient of the Actor network update parameters: (33) Where: is the Actor network action strategy function, is the current Critic network action state value function.

[0130] Furthermore, to enhance learning stability, the MADDPG algorithm uses a soft update mechanism to ensure that only a subset of parameters are updated during each learning process, enabling a slow update of parameters. This approach effectively prevents drastic changes in parameters, further improving learning stability and effectiveness.

[0131] (34) (35) Where: is the update rate, its value is less than 1; and They are the parameters of the target Critic network and target Actor network of the agent respectively.

[0132] As a further improvement, based on the MADDPG solution principle, the present invention designs an overall algorithm flow of a multi-regional power grid collaborative optimization model based on alternating MADDPG training for the constructed two-layer game model as follows: Step 41: Initialize parameters and environment of each agent. Set optimization scheduling period T And set the number of training rounds for each agent M , the initial values ​​are all set to 1, and the agent network parameters are randomly set at the beginning Load the microgrid cluster autonomous collaborative optimization model into the environment and set the interface files with the states and actions in the MADDPG algorithm.

[0133] Step 42: The leader DSO selects an action and updates the network parameters. In the leader update round, first, based on the state information extracted from the microgrid and distribution network, Select your own action , after executing this action, the reward calculated based on the initial default action of the microgrid agent is , and update the network parameters of the master agent according to formulas (31)-(35), and judge whether the set upper limit of the leader update round is reached. If it is satisfied, it enters the follower update round. If not, the leader continues training; Step 43: Each microgrid of followers selects an action and updates network parameters. In the update round of each microgrid of followers, each microgrid acts as a follower according to the current state. Select each action , perform actions and calculate rewards , and update the network parameters of multiple slave agents according to formulas (31)-(35), and judge whether the set upper limit of follower update rounds is reached. If it is satisfied, it goes to the next step, if not, the follower continues training; Step 44: Determine whether both the leader and follower reward curves meet the convergence conditions. If so, end the training, output the results and save them. Otherwise, go to step 42 for the leader training round and perform a new round of iteration until the convergence requirements are met.

[0134] Step 5: Use simulation software to program and solve the above model.

[0135] The above model is programmed and solved using simulation software. By comparing it with the centralized optimization solution, it is verified that the trained model not only has a good ability to deal with uncertainty, but also can realize real-time decision-making of the system, which is conducive to online application. By comparing it with the single-agent deep reinforcement learning method, it is verified that the MADDPG algorithm has a faster convergence speed and better training results during the training process, and has obvious advantages in dealing with complex environmental problems.

[0136] As an example, the present invention proposes a microgrid cluster autonomous collaborative optimization method based on multi-agent deep reinforcement learning that considers multi-agent games. This method is mainly aimed at the current microgrid cluster collaborative optimization problem in the power grid, which presents high-dimensional, non-convex and nonlinear characteristics, leading to problems such as complex modeling and dimensionality explosion. At the same time, the differences in the interests of the various parties in the game make it difficult to achieve a balanced distribution of benefits. Therefore, the present invention constructs a microgrid cluster autonomous collaborative optimization model based on multi-agent reinforcement learning that considers multi-agent games. It adopts a multi-agent deep deterministic policy gradient method and designs the optimization problem as a distributed optimization problem in a multi-agent reinforcement learning environment. This method can effectively solve the high-dimensional, non-convex and nonlinear microgrid cluster autonomous collaborative optimization problem.

[0137] like Figure 3 As shown, the second object of the present invention is to provide an integrated energy system microgrid group optimization system. Based on the above integrated energy system microgrid group optimization method, the system includes: The collection module 100 is used to collect observation data of the operator agent and the microgrid agent; The training module 200 is used to alternately train the action network and the evaluation network of the two-layer agent based on the collected observation data of the operator agent and each microgrid agent, and repeatedly iterate to obtain the operator agent and multiple microgrid agents; Construction module 300 is used to construct a microgrid cluster autonomous collaborative optimization model that considers multi-agent game based on a two-layer game model with one master and multiple slaves, combined with the constraints of the operator agent and multiple microgrid agents; based on the state space, action space, environment and reward function and the microgrid cluster autonomous collaborative optimization model that considers multi-agent game, a microgrid cluster autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment is constructed; The solution module 400 is used to solve the microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning to obtain a microgrid cluster optimization solution.

[0138] Based on the above scheme, the present invention considers a multi-agent operator-microgrid master-slave game model. Taking into account the diverging interests and game interaction mechanisms between operators and microgrids, the present invention achieves a balanced distribution of benefits among all parties by constructing an operator-microgrid master-slave game model and solving for Nash equilibrium points. Based on the fundamental elements of reinforcement learning, solving the master-slave game model is formulated as a microgrid collaborative optimization problem within a reinforcement learning environment. The proposed optimization algorithm based on deep reinforcement learning is highly compatible with the constructed master-slave game model. Each agent iteratively updates its action network and evaluation network during training, enabling rapid and stable solution of Nash equilibrium points in online applications. An algorithm employing alternating multi-agent deep deterministic policy gradient training improves the convergence speed and solution efficiency of each agent. Leveraging the "centralized training-distributed execution" nature of multi-agent systems, after training, each microgrid can make optimal decisions based solely on its local observations, effectively protecting the privacy of each microgrid.

[0139] like Figure 4 As shown, a third object of an embodiment of the present invention is to provide an electronic device comprising a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor, wherein the processor implements the above-mentioned integrated energy system microgrid optimization method when executing the computer program. The electronic device also includes a communication interface 703 and a bus 704.

[0140] A fourth object of an embodiment of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned integrated energy system microgrid group optimization method is implemented.

[0141] A fifth objective of an embodiment of the present invention is to provide a computer program product, which includes computer instructions, and the computer instructions instruct a computer to execute the above-mentioned integrated energy system microgrid optimization method.

[0142] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0144] The present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to magnetic disk storage, readable storage media, optical storage, etc.) containing computer-usable program code.

[0145] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0146] Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for optimizing a microgrid group in an integrated energy system, characterized in that: include: Collect observation data from operator agents and microgrid agents; Based on the collected observation data of the operator agent and each microgrid agent, the action network and evaluation network of the two-layer agent are alternately trained and iterated repeatedly to obtain a stable convergence operator agent and multiple microgrid agents. Based on a two-layer game model with one master and multiple slaves, a microgrid cluster autonomous collaborative optimization model considering multi-agent game is constructed by combining the constraints of the operator agent and multiple microgrid agents. Based on the state space, action space, environment and reward function and the microgrid group autonomous collaborative optimization model considering multi-agent game, a microgrid group autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment is constructed; The microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning is solved to obtain a microgrid cluster optimization solution.

2. The integrated energy system microgrid optimization method according to claim 1, characterized in that: The observation data of the operator intelligent agent and the microgrid intelligent agent include: load data, actual output power of renewable energy, price per unit volume of natural gas, low calorific value of natural gas, energy conversion efficiency of micro gas turbines and charging and discharging efficiency of energy storage equipment, and random charging power of electric vehicles.

3. The integrated energy system microgrid optimization method according to claim 1, characterized in that: The one-master-multiple-slave two-layer game model includes: First, a microgrid equipment model is established: the integrated energy system is divided into multiple microgrids, each of which includes photovoltaics, wind turbines, micro gas turbines, battery energy storage systems, base loads, and electric vehicle charging loads; A two-tiered game model for active energy management in microgrids, with one master and multiple followers, is established. The model consists of two layers, one in a sequential order and with interactive iteration. The operator DSO, as the leader, aims to maximize its own revenue; the microgrid, as a follower, aims to minimize its own output costs. Determine the objective function and constraints: the objective function of the DSO is to maximize its own electricity sales revenue, and the objective function of each microgrid is to minimize its own operating costs; the equality constraint is the power balance constraint, and the system's inequality constraints include the upper and lower limits of gas turbine output power, energy storage equipment operation constraints, electric vehicle load adjustment constraints, unit climbing constraints, and interconnection line transmission power constraints.

4. The integrated energy system microgrid optimization method according to claim 3, characterized in that: Based on the one-master-multiple-slave two-layer game model, combined with the constraints of the operator agent and multiple microgrid agents, a microgrid group autonomous collaborative optimization model considering multi-agent game is constructed, including: As a leader, DSO aims to maximize its own benefits. The objective function is as follows: Where: Where: 、 are the revenues from electricity trading between the DSO and the main grid and microgrid cluster respectively; and They are t The amount of electricity sold and purchased by DSO to the main grid during this period, and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; N is the number of microgrids; For each microgrid agent, the objective function is to minimize the operating cost of each microgrid within the cycle. The objective function is: Where: Where: T is the total scheduling time period, 、 、 They are the microgrid’s transaction costs, fuel costs, and energy storage equipment operation and maintenance costs; and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; is the price per unit volume of natural gas, Hi is the heat released per unit volume of gas, for t The electric power output of the gas turbine during the period, The energy conversion efficiency of the gas turbine; is the converted unit charge and discharge cost, and They are t The charging and discharging power of the energy storage device during the time period, and are charge and discharge efficiency respectively.

5. The integrated energy system microgrid optimization method according to claim 4, characterized in that: The constraints of the operator agent and the multiple microgrid agents include: Constraints are divided into system power balance constraints, transaction price and power constraints, and microgrid unit operation constraints; The power balance constraints that need to be met within each microgrid and the DSO are: Where: and They are t Time period microgrid n The amount of electricity sold or purchased from DSOs; and Microgrid exist t The output of wind turbines and photovoltaics during the period, for t The output of the gas turbine during the period, For microgrids n exist t Active load during the period, 、 for t Discharging and charging power of energy storage equipment during time periods; and They are t The amount of electricity sold or purchased by DSO from the main grid during the period; The transaction price and power constraints are: Where: and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; Where: and They are t The amount of electricity sold and purchased by the microgrid to the DSO during the period, 、 are the upper and lower limits of the tie line power between the microgrid and the DSO, and are the average prices of electricity sales and electricity purchase respectively; Microgrid unit operation constraints include gas turbine operation constraints and energy storage device operation constraints: Where: For gas turbine equipment t Output power during the time period, and They are the upper and lower limits of the output allowed for gas turbine equipment respectively; and Micro gas turbines in t Time period and t-1 Output power during the time period, is the maximum value of the climbing power in one scheduling period; Energy storage devices have two states: charging and discharging. Constraints include upper and lower limits of charge, state of charge, and charge and discharge power constraints. Specifically: Where: is the rated capacity of the energy storage unit, 、 Energy storage equipment in t Time period and t+1 The state of charge during the time period, and are the charging and discharging conversion efficiency of energy storage, 、 for t Charging and discharging power during the time period; 、 The upper and lower limits of the state of charge allowed by the system; It is the maximum charging and discharging power of the energy storage device.

6. The integrated energy system microgrid optimization method according to claim 3, characterized in that: The microgrid group autonomous collaborative optimization model based on the state space, action space, environment and reward function and the multi-agent game consideration is constructed to obtain a microgrid group autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment, including: The state space is composed of the state space of the master agent and multiple slave agents. In each period, the state space of the DSO master agent includes the microgrid n Electricity purchase from DSO and electricity sales , the electricity purchase price of DSO to the main grid and electricity sales price ,for In each period, each microgrid includes the user load of each microgrid from the state space of the intelligent agent , photovoltaic unit output , wind turbine output , the charging power of electric vehicle load , the microgrid's electricity purchase and sales price to DSO and And the energy storage charge status in the previous period and the gas turbine output power ,for The action space of the DSO master agent includes its microgrid-oriented n Electricity purchase and sales price and , and sent to each microgrid, i.e. Each microgrid dispatches the output of the equipment in each microgrid from the action space of the intelligent agent, taking into account the charging and discharging power of the energy storage and , the output of the micro gas turbine unit , the charging power of electric vehicle load after electricity price guidance ,Right now Each agent takes action based on its current state, interacts with the environment, receives a reward, and transitions to the next state. This action is evaluated and used to guide the next agent's action. Constraints are established in the operator-microgrid group master-slave game model as the environment. At each moment, after each agent takes an action, the DSO's revenue and the operating costs of each microgrid are calculated. The state is fed back to calculate the reward function and transition to the next state, repeating the cycle. For DSO agents, the objective function is directly As an immediate reward, if the DSO does not satisfy the constraints, a penalty coefficient k is set s 、k b and penalty value , together with the immediate reward, serves as the agent's final reward function ; The calculation is as follows: For each microgrid agent, the cost function of the microgrid The opposite number of is used as the instant reward of the microgrid. If a microgrid does not meet the constraints, the penalty coefficient k is set soc 、k EV and penalty value , together with the immediate reward, serves as the final reward function of the corresponding agent , calculated as follows: 。 7. The integrated energy system microgrid optimization method according to claim 1, characterized in that: The solution to the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning is solved by using the MADDPG algorithm based on the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning; The MADDPG algorithm generates an action using the Actor strategy network in the current state, and inputs the action and the current state into the Critic evaluation network to calculate Q value, get Q The action with the largest value; among them, the parameters of the Critic network are updated by minimizing the loss value ; Loss function when critic network is updated The calculation formula is as follows: Where: , are the action state value functions of the current Critic network and the target Critic network respectively; To obtain the reward, is the discount factor; Combined with Critic network Q Value, calculate the policy gradient of the Actor network update parameter: Where: is the Actor network action strategy function, is the current Critic network action state value function.

8. The integrated energy system microgrid optimization method according to claim 7, characterized in that: The solution of the microgrid group autonomous collaborative optimization model based on multi-agent deep reinforcement learning also includes: The soft update mechanism is used to update only some parameters during each learning process. The method is as follows: Where: is the update rate, its value is less than 1; and They are the parameters of the target Critic network and target Actor network of the agent respectively.

9. The integrated energy system microgrid optimization method according to claim 8, characterized in that: The microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning is solved to obtain a microgrid cluster optimization solution, including: Initialize parameters and environment of each agent: set optimization scheduling period T And set the number of training rounds for each agent M , the initial values ​​are all set to 1, and the agent network parameters are randomly set at the beginning ; Load the microgrid cluster autonomous collaborative optimization model into the environment and set the interface file with the state and action in the MADDPG algorithm; The leader DSO selects actions and updates network parameters: In the leader update round, it first selects actions based on the state information extracted from the microgrid and distribution network. Select your own action , after executing this action, the reward calculated based on the action of the microgrid agent , and update the network parameters of the master agent according to formulas (31)-(35), and judge whether the set upper limit of the leader update round is reached. If it is satisfied, it enters the follower update round. If not, the leader continues training; Each follower microgrid selects an action and updates the network parameters: In the follower microgrid update round, each microgrid acts as a follower according to the current state Select each action , perform actions and calculate rewards , and update the network parameters of multiple slave agents to determine whether the set upper limit of follower update rounds is reached. If it is met, it will enter the next step. If not, the follower will continue training; Determine whether the leader and follower reward curves both meet the convergence conditions. If so, end the training, output the results and save them. Otherwise, go to the leader DSO to select actions and update network parameters, and perform a new round of iteration until the convergence requirements are met.

10. A comprehensive energy system microgrid group optimization system, characterized in that: include: The collection module is used to collect observation data from the operator agent and the microgrid agent; The training module is used to alternately train the action network and evaluation network of the two-layer agent based on the collected observation data of the operator agent and each microgrid agent, and iterate repeatedly to obtain the operator agent and multiple microgrid agents; A construction module is used to construct a microgrid cluster autonomous collaborative optimization model that considers multi-agent game based on a two-layer game model with one master and multiple slaves, combined with the constraints of the operator agent and multiple microgrid agents; Based on the state space, action space, environment and reward function and the microgrid group autonomous collaborative optimization model considering multi-agent game, a microgrid group autonomous collaborative optimization model based on deep reinforcement learning in a multi-agent environment is constructed; The solution module is used to solve the microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning to obtain a microgrid cluster optimization solution.

11. The integrated energy system microgrid group optimization system according to claim 10, characterized in that: The observation data of the operator intelligent agent and the microgrid intelligent agent include: load data, actual output power of renewable energy, price per unit volume of natural gas, low calorific value of natural gas, energy conversion efficiency of micro gas turbines and charging and discharging efficiency of energy storage equipment, and random charging power of electric vehicles.

12. The integrated energy system microgrid group optimization system according to claim 10, characterized in that: The one-master-multiple-slave two-layer game model includes: First, a microgrid equipment model is established: the integrated energy system is divided into multiple microgrids, each of which includes photovoltaics, wind turbines, micro gas turbines, battery energy storage systems, base loads, and electric vehicle charging loads; A two-tiered game model for active energy management in microgrids, with one master and multiple followers, is established. The model consists of two layers, one in a sequential order and with interactive iteration. The operator DSO, as the leader, aims to maximize its own revenue; the microgrid, as a follower, aims to minimize its own output costs. Determine the objective function and constraints: the objective function of the DSO is to maximize its own electricity sales revenue, and the objective function of each microgrid is to minimize its own operating costs; the equality constraint is the power balance constraint, and the system's inequality constraints include the upper and lower limits of gas turbine output power, energy storage equipment operation constraints, electric vehicle load adjustment constraints, unit climbing constraints, and interconnection line transmission power constraints.

13. The integrated energy system microgrid group optimization system according to claim 12, characterized in that: The building blocks are specifically used for: As a leader, DSO aims to maximize its own benefits. The objective function is as follows: Where: Where: 、 are the revenues from electricity trading between the DSO and the main grid and microgrid cluster respectively; and They are t The amount of electricity sold and purchased by DSO to the main grid during this period, and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; N is the number of microgrids; For each microgrid agent, the objective function is to minimize the operating cost of each microgrid within the cycle. The objective function is: Where: Where: T is the total scheduling time period, 、 、 They are the microgrid’s transaction costs, fuel costs, and energy storage equipment operation and maintenance costs; and They are t Time period microgrid n The amount of electricity sold and purchased from DSO, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; is the price per unit volume of natural gas, Hi is the heat released per unit volume of gas, for t The electric power output of the gas turbine during the period, The energy conversion efficiency of the gas turbine; is the converted unit charge and discharge cost, and They are t The charging and discharging power of the energy storage device during the time period, and are charge and discharge efficiency respectively.

14. The integrated energy system microgrid group optimization system according to claim 13, characterized in that: The constraints of the operator agent and the multiple microgrid agents include: Constraints are divided into system power balance constraints, transaction price and power constraints, and microgrid unit operation constraints; The power balance constraints that need to be met within each microgrid and the DSO are: Where: and They are t Time period microgrid n The amount of electricity sold or purchased from DSOs; and Microgrid exist t The output of wind turbines and photovoltaics during the period, for t The output of the gas turbine during the period, For microgrids n exist t Active load during the period, 、 for t Discharging and charging power of energy storage equipment during time periods; and They are t The amount of electricity sold or purchased by DSO from the main grid during the period; The transaction price and power constraints are: Where: and They are t The electricity price that DSO sells and purchases from the main grid during this period, and They are t The electricity sales and purchase prices of DSO to microgrid groups during the period; Where: and They are t The amount of electricity sold and purchased by the microgrid to the DSO during the period, 、 are the upper and lower limits of the tie line power between the microgrid and the DSO, and are the average prices of electricity sales and electricity purchase respectively; Microgrid unit operation constraints include gas turbine operation constraints and energy storage device operation constraints: Where: For gas turbine equipment t Output power during the time period, and They are the upper and lower limits of the output allowed for gas turbine equipment respectively; and Micro gas turbines in t Time period and t-1 Output power during the time period, is the maximum value of the climbing power in one scheduling period; Energy storage devices have two states: charging and discharging. Constraints include upper and lower limits of charge, state of charge, and charge and discharge power constraints. Specifically: Where: is the rated capacity of the energy storage unit, 、 Energy storage equipment in t Time period and t+1 The state of charge during the time period, and are the charging and discharging conversion efficiency of energy storage, 、 for t Charging and discharging power during the time period; 、 The upper and lower limits of the state of charge allowed by the system; It is the maximum charging and discharging power of the energy storage device.

15. The integrated energy system microgrid group optimization system according to claim 12, characterized in that: The building blocks are specifically used for: The state space is composed of the state space of the master agent and multiple slave agents. In each period, the state space of the DSO master agent includes the microgrid n Electricity purchase from DSO and electricity sales , the electricity purchase price of DSO to the main grid and electricity sales price ,for In each period, each microgrid includes the user load of each microgrid from the state space of the intelligent agent , photovoltaic unit output , wind turbine output , the charging power of electric vehicle load , the microgrid's electricity purchase and sales price to DSO and And the energy storage charge status in the previous period and the gas turbine output power ,for The action space of the DSO master agent includes its microgrid-oriented n Electricity purchase and sales price and , and sent to each microgrid, i.e. Each microgrid dispatches the output of the equipment in each microgrid from the action space of the intelligent agent, taking into account the charging and discharging power of the energy storage and , the output of the micro gas turbine unit , the charging power of electric vehicle load after electricity price guidance ,Right now Each agent takes action based on its current state, interacts with the environment, receives a reward, and transitions to the next state. This action is evaluated and used to guide the next agent's action. Constraints are established in the operator-microgrid group master-slave game model as the environment. At each moment, after each agent takes an action, the DSO's revenue and the operating costs of each microgrid are calculated. The state is fed back to calculate the reward function and transition to the next state, repeating the cycle. For DSO agents, the objective function is directly As an immediate reward, if the DSO does not satisfy the constraints, a penalty coefficient k is set s 、k b and penalty value , together with the immediate reward, serves as the agent's final reward function ; The calculation is as follows: For each microgrid agent, the cost function of the microgrid The opposite number of is used as the instant reward of the microgrid. If a microgrid does not meet the constraints, the penalty coefficient k is set soc 、k EV and penalty value , together with the immediate reward, serves as the final reward function of the corresponding agent , calculated as follows: 。 16. The integrated energy system microgrid group optimization system according to claim 10, characterized in that: The solution to the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning is solved by using the MADDPG algorithm based on the autonomous collaborative optimization model of the microgrid group based on multi-agent deep reinforcement learning; The MADDPG algorithm generates an action using the Actor strategy network in the current state, and inputs the action and the current state into the Critic evaluation network to calculate Q value, get Q The action with the largest value; among them, the parameters of the Critic network are updated by minimizing the loss value ; Loss function when critic network is updated The calculation formula is as follows: Where: , are the action state value functions of the current Critic network and the target Critic network respectively; To obtain the reward, is the discount factor; Combined with Critic network Q Value, calculate the policy gradient of the Actor network update parameter: Where: is the Actor network action strategy function, is the current Critic network action state value function.

17. The integrated energy system microgrid group optimization system according to claim 16, characterized in that: The solution module is specifically used for: The soft update mechanism is used to update only some parameters during each learning process. The method is as follows: Where: is the update rate, its value is less than 1; and They are the parameters of the target Critic network and target Actor network of the agent respectively.

18. The integrated energy system microgrid group optimization system according to claim 17, characterized in that: The microgrid cluster autonomous collaborative optimization model based on multi-agent deep reinforcement learning is solved to obtain a microgrid cluster optimization solution, including: Initialize parameters and environment of each agent: set optimization scheduling period T And set the number of training rounds for each agent M , the initial values ​​are all set to 1, and the agent network parameters are randomly set at the beginning ; Load the microgrid cluster autonomous collaborative optimization model into the environment and set the interface file with the state and action in the MADDPG algorithm; The leader DSO selects actions and updates network parameters: In the leader update round, it first selects actions based on the state information extracted from the microgrid and distribution network. Select your own action , after executing this action, the reward calculated based on the action of the microgrid agent , and update the network parameters of the master agent according to formulas (31)-(35), and judge whether the set upper limit of the leader update round is reached. If it is satisfied, it enters the follower update round. If not, the leader continues training; Each follower microgrid selects an action and updates the network parameters: In the follower microgrid update round, each microgrid acts as a follower according to the current state Select each action , perform actions and calculate rewards , and update the network parameters of multiple slave agents to determine whether the set upper limit of follower update rounds is reached. If it is met, it will enter the next step. If not, the follower will continue training; Determine whether the leader and follower reward curves both meet the convergence conditions. If so, end the training, output the results and save them. Otherwise, go to the leader DSO to select actions and update network parameters, and perform a new round of iteration until the convergence requirements are met.

19. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the integrated energy system microgrid optimization method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for optimizing a microgrid cluster of an integrated energy system according to any one of claims 1 to 9 is implemented.

21. A computer program product comprising computer instructions, characterized in that: The computer instructions instruct the computer to execute the integrated energy system microgrid group optimization method according to any one of claims 1 to 9.

Images