A multi-agent hierarchical game coordination method and device for integrated energy microgrid groups

Abstract

The present invention relates to a multi-agent hierarchical game coordination method and device for an integrated energy microgrid group, including: creating multi-agent deep reinforcement learning parameters; creating an operator master agent; creating a multi-energy microgrid slave agent for each of the multi-energy microgrids in the integrated energy system; based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent, using an improved multi-agent deep determination policy gradient MADDPG algorithm to perform reinforcement learning training on a pre-created neural network to obtain a multi-agent hierarchical game coordination model. The present invention realizes the combination of centralized management and distributed control of each unit in the integrated energy system, enhances the coordinated scheduling of the integrated energy system, increases the reliability of operation, and reduces the communication and computing burden of the system. The present invention also relates to a device and a storage medium.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a multi-agent hierarchical game coordination method and device for an integrated energy microgrid group. Background Art

[0002] An integrated energy system usually refers to a new type of complex energy system that takes the power network as the energy core, uses distributed collaborative control technology, intelligent optimization control technology, and advanced information and communication technology as the main technical means, uses renewable energy as the main primary energy source, and integrates and interacts with other energy networks such as thermal networks and natural gas networks. In an integrated energy system, the coordinated management of multiple energy sources can effectively improve energy utilization, increase system flexibility, improve user satisfaction, and alleviate environmental pollution and energy supply issues.

[0003] At present, each energy microgrid in the integrated energy system is controlled by centralized control or distributed control. However, the completely centralized control method greatly increases the communication burden and computing burden. At the same time, due to the strong coupling relationship between each energy microgrid, the completely distributed control method cannot meet the minimum requirements of system coordination when only using local variables. Therefore, how to combine centralized control and distributed control for each unit in the integrated energy system is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of the problem of coordinated scheduling of integrated energy multi-microgrid systems, the present invention provides a multi-agent hierarchical game coordination method and device for integrated energy microgrid groups.

[0005] In a first aspect, the present invention provides a multi-agent hierarchical game coordination method for an integrated energy microgrid group, the method comprising:

[0006] Create multi-agent deep reinforcement learning parameters;

[0007] Create an operator master agent;

[0008] Creating a multi-energy microgrid slave agent for each of the multi-energy microgrids in the integrated energy system;

[0009] Based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent, the pre-created neural network is trained by reinforcement learning using the improved multi-agent deep determination policy gradient (MADDPG) algorithm to obtain a multi-agent hierarchical game collaborative model.

[0010] In a second aspect, the present invention further provides a multi-agent hierarchical game coordination device for an integrated energy microgrid group, the device comprising:

[0011] The first building block is used to create multi-agent deep reinforcement learning parameters;

[0012] The second building unit is used to create an operator master agent;

[0013] A third construction unit, configured to create a multi-energy microgrid slave agent for each of the multi-energy microgrids in the integrated energy system;

[0014] The reinforcement learning training unit is used to perform reinforcement learning training on a pre-created neural network based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent using an improved multi-agent deep determination policy gradient MADDPG algorithm to obtain a multi-agent hierarchical game collaborative model.

[0015] In a third aspect, the present invention further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements a multi-agent hierarchical game collaboration method for an integrated energy microgrid group as described in any one of the first aspects.

[0016] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the multi-agent hierarchical game collaboration method for an integrated energy microgrid group described in any one of the first aspects is implemented.

[0017] The present invention provides a multi-agent hierarchical game coordination method and device for an integrated energy microgrid group, including: creating multi-agent deep reinforcement learning parameters; creating an operator master agent; creating a multi-energy microgrid slave agent for each of the multi-energy microgrids in the integrated energy system; based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent, using an improved multi-agent deep determination policy gradient MADDPG algorithm to perform reinforcement learning training on a pre-created neural network to obtain a multi-agent hierarchical game coordination model. The present invention realizes the combination of centralized management and distributed control of each unit in the integrated energy system, enhances the coordinated scheduling of the integrated energy system, increases the reliability of operation, and reduces the communication and computing burden of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0019] Figure 1 It is a flowchart of a multi-agent hierarchical game coordination method for an integrated energy microgrid group provided by an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of a framework of a multi-agent hierarchical game coordination method for an integrated energy microgrid group provided by another embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of an improved MADDPG algorithm model provided by another embodiment of the present invention;

[0022] Figure 4 It is a module schematic diagram of a multi-agent hierarchical game collaboration device for an integrated energy microgrid group provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present invention can be combined with each other without conflict.

[0024] The following detailed description is an exemplary description, which is intended to provide further detailed description of the present invention. Unless otherwise specified, all technical terms used in the present invention have the same meaning as those generally understood by those skilled in the art to which the present invention belongs. The terms used in the present invention are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present invention.

[0025] The following will be combined with the attached Figure 1 , a multi-agent hierarchical game coordination method for an integrated energy microgrid group provided by an embodiment of the present invention is described, comprising the following steps:

[0026] 110. Create multi-agent deep reinforcement learning parameters.

[0027] Specifically, the multi-agent deep reinforcement learning parameters include:

[0028] 1) Environmental state parameters: including multi-energy flow data of three energy forms: electricity, gas and heat, as well as the market pricing of the operator’s main intelligent body;

[0029] 2) Action space: including the pricing decision of the operator’s master agent and the regulation strategy of various energy resources of the multi-energy microgrid slave agent;

[0030] 3) Reward function: including electricity purchase cost, operating cost and constraint penalty;

[0031] 120. Create an operator master agent;

[0032] Specifically, the operator master agent is established based on the following:

[0033] 1) The action space of the operator’s main agent;

[0034] The action space of the operator’s main agent is the relevant control variables in the upper model, including the electricity price c for energy exchange between the operator and each multi-energy microgrid. mg , as shown below:

[0035] A op ={c mg (t)}

[0036] 2) The state space of the operator’s main agent;

[0037] The operator's main intelligent agent can observe the global status information of the multi-energy network of the integrated energy system, where the electrical network status includes the quantity measurement of each node, including voltage amplitude and voltage phase angle, and the quantity measurement of each branch, including branch active power and branch reactive power; the thermal network parameters include the temperature and flow of each pipeline; the gas network parameters include the pressure and flow of each pipeline, as shown in the following formula:

[0038]

[0039] 3) Optimization objectives of the operator’s main intelligent agent;

[0040] The overall operating cost of the integrated energy system is minimized as the optimization goal of the main intelligent agent, so f PR,t Incentive benefits for integrated energy systems participating in ancillary services markets; MG,t is the comprehensive operating cost of the integrated energy multi-energy microgrid in period t, which is transmitted to the upper layer after autonomous optimization by the lower layer model; as shown in the following formula:

[0041]

[0042] 130. Create a multi-energy microgrid slave agent for each multi-energy microgrid in the integrated energy system;

[0043] Specifically, a multi-energy microgrid slave agent is established based on the following:

[0044] 1) Multi-energy microgrid from the action space of intelligent agents

[0045] The multi-energy microgrid is the energy storage charging and discharging power P from the action space of the intelligent agent. i,ch and P i,dis , energy storage charging and discharging decision variables λ i,ch , Gas boiler power Q i,GB , Electric refrigerator power P i,EC and micro gas turbine output P i,MT , as shown in the following formula. It is worth noting that the power purchased by the power grid P i,grid It can be automatically generated based on the electric power balance constraints and the above actions, and does not need to be used as a decision variable.

[0046] A i,MG= {P i,ch ,P i,dis ,λ i,ch ,Q i,GB ,P i,EC ,P i,MT}

[0047] 2) Multi-energy microgrid from the state space of intelligent agent

[0048] The multi-energy microgrid slave agent can observe the local state information of the multi-energy network of the integrated energy system, where the electrical network state includes the quantity measurement of each node within the jurisdiction, such as voltage amplitude and voltage phase angle, and each branch quantity measurement, such as branch active power and branch reactive power; the thermal network parameters include the temperature and flow of each pipeline within the jurisdiction; the gas network parameters include the pressure and flow of each pipeline within the jurisdiction; at the same time, the multi-energy microgrid can also observe the energy exchange pricing between multi-energy microgrids issued by the operator's main agent, as shown in the following formula:

[0049]

[0050] 3) Multi-energy microgrid optimization objectives from the perspective of intelligent agents

[0051] The optimal economic benefit of the multi-energy microgrid itself is selected as the optimization target of the slave intelligent agent, which includes the gas cost, the interaction cost of adjacent microgrids and the interaction cost of the distribution network.

[0052] 140. Based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent, the improved multi-agent deep deterministic policy gradient MADDPG algorithm is used to perform reinforcement learning training on the pre-created neural network to obtain a multi-agent hierarchical game collaborative model.

[0053] Based on the above embodiment, further, the multi-agent deep reinforcement learning parameters include environment state parameters, action space and reward function;

[0054] Environmental status parameters include electricity flow data, gas flow data, heat flow data, and market pricing of the operator's main intelligent body;

[0055] The action space in the multi-agent deep reinforcement learning parameters includes the action space of the operator master agent and the action space of the multi-energy microgrid slave agent;

[0056] The reward function in the multi-agent deep reinforcement learning parameters is established through the optimization objective function of the multi-energy microgrid slave agent and the penalty constraints of the multi-energy microgrid slave agent.

[0057] Based on the above embodiment, further, the reward function in the multi-agent deep reinforcement learning parameter is established by the optimization objective function of the multi-energy microgrid slave agent and the penalty constraint condition of the multi-energy microgrid slave agent, specifically including:

[0058] According to the gas cost of the i-th multi-energy microgrid in period t, the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, and the energy exchange cost of the i-th multi-energy microgrid in period t, the optimization objective function minf of the i-th multi-energy microgrid from the intelligent agent is established. MG,i , the optimization objective function is as follows:

[0059] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t)

[0060]

[0061]

[0062] C i,grid (t) = c grid (t)P i,grid (t)

[0063] In the formula, c gas is the natural gas price, C i,fuel (t) is the gas cost of the i-th multi-energy microgrid in period t, C i,mg (t) is the energy exchange cost of the i-th multi-energy microgrid in period t, C i,grid (t) is the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, P i,MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, c mg (t) is the energy exchange pricing between multi-energy microgrids issued by the operator’s main intelligent agent during period t, P ij,mg (t) is the interactive power between the i-th multi-energy microgrid and the j-th multi-energy microgrid during period t, P i,grid (t) is the interactive power between the i-th multi-energy microgrid and the distribution network in period t, V i,GB (t) is the gas consumption of the gas boiler GB in the i-th multi-energy microgrid during period t, η MT is the power generation efficiency, LHV NG is the calorific value of natural gas;

[0064] Establish a multi-energy microgrid based on the penalty constraints of the intelligent agent:

[0065]

[0066] φ2 =k 2 |SOC(0)-SOC(T)|

[0067] In the formula, k 1 and k 2 is the penalty coefficient, E(t+1) is the charge of the energy stored in period t, SOC(0) is the state of charge of all batteries in the initial period, SOC(T) is the state of charge of all batteries in the total scheduling period T, P ch,max is the maximum charging power of the battery, P dis,max is the maximum discharge power of the battery;

[0068] Through all multi-energy microgrids from the intelligent agent's optimization objective function minf MG,i , we can get the total optimization objective function f of the multi-energy microgrid from the intelligent agent MG , through the multi-energy microgrid from the overall optimization objective function f of the intelligent agent MG And penalty constraints, the reward function R of the multi-energy microgrid from the intelligent agent is obtained t =-(f MG +φ 1 +φ 2 ).

[0069] Based on the above embodiment, further, it also includes establishing operation constraints of the multi-energy microgrid slave intelligent agent;

[0070] The operating constraints of the multi-energy microgrid from the intelligent agent include:

[0071] The real-time balance constraints of the cooling load, heating load and electrical load inside the multi-energy microgrid and the photovoltaic output and wind turbine output are as follows:

[0072] Q i,MT,c (t)+Q i,EC (t) = Q i,load,c (t)

[0073] Q i,MT,h (t)+Q i,GB (t) = Q i,load,h (t)

[0074]

[0075] In the formula, Q i,load,c (t) is the cooling load of the i-th multi-energy microgrid in period t, Q i,load,h (t) is the heat load of the i-th multi-energy microgrid in period t, P i,load (t) is the number of electric loads of the i-th multi-energy microgrid in period t, P i,PV (t) is the photovoltaic output, P i,WT(t) is the fan output, Q i,load,c (t) is the cooling load of the i-th multi-energy microgrid in period t, Q i,load,h (t) is the heat load of the i-th multi-energy microgrid in period t, P i,load (t) is the load of the i-th multi-energy microgrid in period t, Q i,MT,c (t) is the heating power of the micro gas turbine MT in the i-th multi-energy microgrid during time period t, Q i,EC (t) is the cooling power of the electric refrigerator EC in the i-th multi-energy microgrid during time period t, Q i,n,EC (t) is the cooling power of the nth electric refrigerator EC in the i-th multi-energy microgrid during time period t, Q i,MT,h (t) is the cooling power of the micro gas turbine MT in the i-th multi-energy microgrid during time period t, Q i,GB (t) is the heating power of the gas boiler GB in the i-th multi-energy microgrid during period t, P i,MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, P i,PV (t) is the output power of the photovoltaic device PV in the i-th microgrid during period t, P i,WT (t) is the output power of wind turbine WT in the i-th multi-energy microgrid during period t, P i,EC (t) is the charging power of the electric refrigerator EC in the i-th multi-energy microgrid during time period t, P i,ch (t) is the charging power of the energy storage ES in the i-th multi-energy microgrid during period t, P i,dis (t) is the discharge power of the energy storage ES in the i-th multi-energy microgrid during period t;

[0076] Equipment operation constraints and power interaction constraints, where equipment operation constraints include micro gas turbine operation constraints, gas boiler operation constraints, and electric refrigerator operation constraints:

[0077] P MT,min ≤P MT (t)≤P MT,max

[0078] Q GB,min ≤Q GB (t)≤Q GB,max

[0079] P EC,min ≤P EC (t)≤P EC,max

[0080] P i,grid,min ≤P i,grid (t)≤P i,grid,max

[0081] Where P MT,minis the lower limit of gas turbine output, P MT,max is the upper limit of the gas turbine output, P MT (t) is the gas turbine output value during period t; Q GB,min is the lower limit of the thermal power output of the gas boiler, Q GB,max is the upper limit of the thermal power output of the gas boiler, Q GB (t) is the thermal power output of the gas boiler during period t, Q GB,min is the lower limit of the thermal power output of the gas boiler, Q GB,max The upper limit of the thermal power output of the gas boiler; P EC,min is the lower limit of the input power of the electric refrigerator, P EC,max is the upper limit of the input power of the electric refrigerator, P EC (t) is the input power of the refrigerator during period t, P i,grid,min is the lower limit of the interaction power between the i-th multi-energy microgrid and the distribution network, P i,grid,max are the upper limits of the interaction power between the i-th multi-energy microgrid and the distribution network, P i,grid,min is the lower limit of the interaction power between the i-th multi-energy microgrid and the distribution network, P i,grid,max is the upper limit of the interaction power between the i-th multi-energy microgrid and the distribution network;

[0082] The operation constraints of energy storage include state of charge constraints, charge and discharge power constraints, and energy balance constraints at the beginning and end of the dispatch cycle, as follows:

[0083] SOC min ≤SOC(t)≤SOC max

[0084] 0≤P ch (t)≤λ ch,t P ch,max

[0085] 0≤P dis (t)≤(1-λ ch,t ) dis,max

[0086] λ ch,t ∈{0,1}

[0087] SOC(0)=SOC(T)

[0088] In the formula, SOC min SOC is the lower limit of the battery state of charge. max is the upper limit of the battery charge state; P ch,max is the maximum charging power of the battery, P dis,max is the maximum discharge power of the battery, λ ch,tis the charge and discharge state coefficient of the battery in period t, SOC(t) is the state of charge value of all batteries in period t, P ch (t) is the charging power of the energy storage ES during period t, P dis (t) is the discharge power of the energy storage ES during period t.

[0089] Based on the above embodiment, further, step 140 specifically includes:

[0090] 141. Construct an actor-critic deep reinforcement learning model, which consists of a master agent actor network, a multi-energy microgrid slave agent actor network and a critic network;

[0091] From the environmental state parameters in the multi-agent deep reinforcement learning parameters, the global state information of the multi-energy network in the state space of the operator's main agent is obtained;

[0092] The energy exchange price of the multi-energy microgrid is used as the action space of the operator's main agent;

[0093] The output value of energy resources is used as the action space of the slave agent of the multi-energy microgrid;

[0094] 142. According to the state information of the multi-energy microgrid, the environmental state parameter s of the actor-critic deep reinforcement learning model during period t is obtained t ,Environmental state parameters include electric current data, gas current data and thermal current data;

[0095] 143. The main agent actor network is based on the environmental state parameter s t , generating the action aa of the operator master agent in period t t , update the environmental state parameter s t After that, the updated environmental state parameter s′ is obtained t ;

[0096] The i-th multi-energy microgrid is fed from the intelligent agent actor network according to the updated environmental state parameter s t ', generate the multi-energy microgrid from the action of the intelligent agent in period t

[0097] 144. According to the multi-energy microgrid slave agent actor network, the multi-energy microgrid slave agent's actions in period t Observe the environmental state parameters and calculate the environmental state parameters s for the period t+1 t+1 ;

[0098] 145. According to the multi-energy microgrid from the intelligent agent actor network in time period t, the multi-energy microgrid from the intelligent agent Environmental state parameter s at time t+1 t+1, the operating constraints of the multi-energy microgrid slave agent and the reward function of the multi-energy microgrid slave agent, and calculate the reward value R of the operator's master agent in time period t;

[0099] 146. The environmental state parameter s during period t t , the actions of the operator's main intelligent agent aa t 、Environmental state parameters s in period t+1 t+1 , the reward value R of the t period is a four-tuple {s t , aa t ,s t+1 , R} is sent to the experience playback unit as a sample;

[0100] Determine whether the experience playback unit is full;

[0101] If not, return to step 142 to continue execution;

[0102] If yes, execute step 147;

[0103] 147. Samples are obtained from the experience replay unit, and the parameters of the actor-critic deep reinforcement learning model are updated using the gradient algorithm through the samples;

[0104] Among them, the critic neural network is updated according to the loss function, environment state parameters and reward function;

[0105] 148. Output the training results of this round and determine whether the preset cycle end condition is met;

[0106] If so, the loop ends and a multi-agent hierarchical game coordination model is obtained;

[0107] Otherwise, return to step 142.

[0108] Based on the above embodiment, further, the multi-energy network global state information in the state space of the operator's main intelligent body includes electric network state parameter values, thermal network parameter values ​​and gas network parameter values.

[0109] The status information of the multi-energy microgrid includes the electric network status value, heat network parameter value, gas network parameter value of the multi-energy microgrid and the energy exchange pricing between the multi-energy microgrids issued by the operator's main intelligent body.

[0110] It should be understood that there is an existing disclosed method and device for peak-shaving energy optimization of distribution networks. This technical solution proposes a method for peak-shaving energy optimization of distribution networks, including establishing a peak-shaving energy optimization framework between distribution network operators and load aggregators, distributed energy storage operators, and photovoltaic operators based on reinforcement learning theory; constructing a distribution network operator model, whose optimization objectives include maximizing operating benefits and minimizing the standard deviation of the net load curve; constructing a cooperative alliance model based on the internal interactive electricity prices and peak-shaving prices released by the distribution network operator, including a load aggregator optimization model, a distributed energy storage operator model, a photovoltaic operator model, and a cooperative game model; comprehensively considering the interests of the cooperative alliance, constructing a peak-shaving energy optimization model; and using the Q-learning-ADMM joint solution algorithm to solve the problem to obtain the optimal electricity trading demand plan and the optimal energy strategy of each entity.

[0111] The disadvantages of the prior art 1 include:

[0112] 1. The reinforcement learning and Q-learning-ADMM joint solution algorithm proposed in the technical solution requires a large amount of high-quality data input for training and optimization. However, in an integrated energy system characterized by "multi-party investment and multi-party participation", independent autonomous entities that own the energy resources cannot provide the above private data;

[0113] 2. In a highly uncertain and information-limited shared scenario environment, reinforcement learning and

[0114] The Q-learning-ADMM joint solution algorithm may face slow convergence and stability issues.

[0115] Another technology disclosed in the prior art is a method for energy storage planning of a shared hybrid energy storage power station based on cooperative game. This technical solution proposes a method for energy storage planning of a shared hybrid energy storage power station based on cooperative game. First, a control strategy for hybrid energy storage is formulated according to the different operating characteristics of batteries and supercapacitors. The battery adopts a "low storage and high generation" charging and discharging strategy, and the supercapacitor is based on model prediction control to smooth wind and solar fluctuations and track planned power generation; secondly, a two-layer optimization configuration model for the hybrid energy storage power station is established, and an evaluation index for the energy storage configuration results is constructed; finally, the income of each new energy station is allocated based on the improved Shapley scoring method considering the energy storage configuration effect. The energy storage planning method for a shared hybrid energy storage power station based on cooperative game provided by the present invention clarifies the operating strategy and profit-making method of hybrid energy storage during planning, maximizes the annual comprehensive benefits of the energy storage power station, and reasonably allocates income to the new energy stations in the alliance, so that the shared hybrid energy storage power station obtains individual rationality and collective rationality.

[0116] Disadvantages of this technique include:

[0117] 1. This technical solution requires accurate forecast information to achieve effective wind and solar fluctuation smoothing and power generation tracking, which is difficult to achieve in the operating environment of an integrated energy system with highly dynamic and strong random characteristics;

[0118] 2. The Shapley value is a cooperative game solution with quite high computational complexity, especially in complex scenarios involving multiple participants, which limits the application of this method in large-scale systems.

[0119] The present invention proposes a multi-agent hierarchical game coordination method for a group of integrated energy microgrids. The operator, as the leader, indirectly regulates each multi-energy microgrid through price signals to form an operator master agent decision module; each multi-energy microgrid regulates the output of the energy resources under its jurisdiction according to the operator's pricing strategy to form a multi-energy microgrid slave agent decision module; the improved MADDPG multi-agent deep reinforcement learning algorithm module considering the master-slave game is adopted to construct an information interaction mechanism between each agent, and train the strategy network of each agent. In addition, the present invention proposes an action space, state space and optimization target design scheme for the operator master agent and the multi-energy microgrid slave agent for sequential decision-making, wherein the operator master agent action space is the energy exchange price between multi-energy microgrids, the state space is the operation state of the entire network, and the optimization target considers the ancillary service market revenue; the slave agent action space is the output adjustment of various energy resources, the state space is the perception state within the jurisdiction area, and the optimization target is to maximize its own operation benefit.

[0120] In addition, the present invention also proposes an improved MADDPG master-slave heterogeneous multi-agent deep reinforcement learning algorithm model and intelligent interaction method and process for the master-slave game mechanism between the operator's master agent and the multi-energy microgrid slave agent, in which the operator's master agent first acts and issues the energy exchange price; the multi-energy microgrid slave agent then makes decisions based on the actions of the master agent to form a new reinforcement learning environment state, and on this basis calculates the reward functions of the master agent and the slave agent respectively, and finally completes this round of iterative training.

[0121] Embodiment 1 of the present invention greatly improves the speed and accuracy of model training, solves the problem that traditional reinforcement learning is difficult to handle constraints, and can ensure that the intelligent agent finds the optimal strategy under the premise of satisfying the constraints. It avoids the problems of traditional reinforcement learning methods that are difficult to satisfy constraints and convergence difficulties due to manually setting penalty coefficients. The scheduling results can be given in real time, and the multi-agent hierarchical game collaborative model in this application realizes the combination of centralized management and distributed control of each unit in the integrated energy system, enhances the collaborative scheduling of the integrated energy system, increases the reliability of operation, and reduces the communication and computing burden of the system.

[0122] As attached Figure 2-3As shown, another embodiment of the present invention includes the following modules: a multi-agent deep reinforcement learning algorithm module, an operator master agent module, and a multi-energy microgrid slave agent module:

[0123] (1) Multi-agent deep reinforcement learning algorithm module

[0124] The multi-agent deep reinforcement learning model uses the multi-agent deep deterministic policy gradient algorithm (MADDPG) for training and solving. The MADDPG algorithm is one of the multi-agent reinforcement learning algorithms. In multi-agent deep reinforcement learning, the agent continuously interacts with the surrounding environment, perceives the state of the environment, and takes corresponding actions according to certain strategies. After receiving the action, the environment will feedback an action reward to the agent and enter the next state, and then start to repeat the above process in a cycle, and finally learn an optimal strategy to maximize the cumulative reward.

[0125] The parameters of the multi-agent deep reinforcement learning algorithm module are as follows:

[0126] 1) Environmental status: including multi-energy flow data of electricity, gas and heat, as well as market pricing of operator main intelligent bodies;

[0127] 2) Action: including the pricing decision of the operator’s master agent and the regulation strategy of various energy resources of the multi-energy microgrid slave agent;

[0128] 3) Rewards: including electricity purchase costs, operating costs and constraint penalties;

[0129] (2) Operator main intelligent module

[0130] 1) Action space of the operator’s main agent module

[0131] The action space of the operator’s main agent is the relevant control variables in the upper model, including the electricity price c for energy exchange between the operator and each multi-energy microgrid. mg , as shown below:

[0132] A op ={c mg (t)}

[0133] 2) State space of the operator's main agent module

[0134] The operator's main intelligent agent can observe the global status information of the multi-energy network of the integrated energy system, where the electrical network status includes the quantity measurement of each node (voltage amplitude, voltage phase angle), the quantity measurement of each branch (branch active power, branch reactive power); the thermal network parameters include the temperature and flow of each pipeline; the gas network parameters include the pressure and flow of each pipeline, as shown in the following formula:

[0135]

[0136] 3) Optimization goals of the operator’s main intelligent agent module

[0137] The overall operating cost of the integrated energy system is minimized as the optimization goal of the main intelligent agent, so f PR,t Incentive benefits for integrated energy systems participating in ancillary services markets; MG,t is the comprehensive operating cost of the integrated energy multi-energy microgrid in period t, which is transmitted to the upper layer after autonomous optimization by the lower layer model; as shown in the following formula:

[0138]

[0139] (3) Multi-energy microgrid from intelligent agent module

[0140] 1) Multi-energy microgrid from the action space of intelligent agent module

[0141] The multi-energy microgrid is the energy storage charging and discharging power P from the action space of the intelligent agent. i,ch and P i,dis , energy storage charging and discharging decision variables λ i,ch , Gas boiler power Q i,GB , Electric refrigerator power P i,EC and micro gas turbine output P i,MT , as shown in the following formula. It is worth noting that the power purchased by the power grid P i,grid It can be automatically generated based on the electric power balance constraints and the above actions, and does not need to be used as a decision variable.

[0142] A i,MG = {P i,ch ,P i,dis ,λ i,ch ,Q i,GB ,P i,EC ,P i,MT}

[0143] 2) Multi-energy microgrid from the state space of intelligent agent module

[0144] The multi-energy microgrid slave agent can observe the local state information of the multi-energy network of the integrated energy system, where the electrical network state includes the quantity measurement of each node within the jurisdiction (voltage amplitude, voltage phase angle), and the quantity measurement of each branch (branch active power, branch reactive power); the thermal network parameters include the temperature and flow of each pipeline within the jurisdiction; the gas network parameters include the pressure and flow of each pipeline within the jurisdiction; at the same time, the multi-energy microgrid can also observe the energy exchange pricing between multi-energy microgrids issued by the operator's main agent, as shown in the following formula:

[0145]

[0146] 3) Optimization objectives of multi-energy microgrid from intelligent agent module

[0147] The optimal economic benefit of the multi-energy microgrid is selected as the optimization target of the slave agent, which includes the gas cost, the interaction cost of the adjacent microgrid and the distribution network interaction cost, as shown in the following formula:

[0148] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t)

[0149]

[0150]

[0151] C i,grid (t) = c grid (t)P i,grid (t)

[0152] In the formula, c gas is the natural gas price; P i,MT (t) is the MT output power of multi-energy microgrid i in period t; c mg (t) is the energy exchange pricing between multi-energy microgrids issued by the operator’s main intelligent agent during time period t; P ij,mg (t), P i,grid (t) are the interactive electricity between microgrid i and microgrid j and between microgrid i and distribution network in period t.

[0153] The cooling, heating and electricity loads and output within the multi-energy microgrid should meet real-time balance constraints.

[0154] Q i,MT,c (t)+Q i,EC (t) = Q i,load,c (t)

[0155] Q i,MT,h (t)+Q i,GB (t) = Q i,load,h (t)

[0156]

[0157] In the formula, Q i,load,c (t), Q i,load,h (t), P i,load (t) are the cooling load, heating load and electric load of the multi-energy microgrid i in period t respectively; P i,PV (t) and P i,WT (t) are the photovoltaic and wind turbine outputs respectively.

[0158] In addition to the energy balance constraints within the microgrid, in order to ensure the safe operation of the microgrid group system, the equipment operation constraints and power interaction constraints must also be met, including the micro gas turbine operation constraints, gas boiler operation constraints, electric refrigerator operation constraints and power interaction constraints, as shown in the following formulas:

[0159] P MT,min ≤P MT (t)≤P MT,max

[0160] Q GB,min ≤Q GB (t)≤Q GB,max

[0161] P EC,min ≤P EC (t)≤P EC,max

[0162] P i,grid,min ≤P i,grid (t)≤P i,grid,max

[0163] Where P MT,min , P MT,max are the upper and lower limits of gas turbine output respectively; Q GB,min , Q GB,max are the upper and lower limits of the thermal power output of the gas boiler; P EC,min , P EC,max are the upper and lower limits of the input power of the electric refrigerator respectively; P i,grid,min , P i,grid,max are the upper and lower limits of the interaction power between microgrid i and distribution network, respectively.

[0164] The action constraint over-limit penalty is used to meet the relevant constraints of the decision action. The operation constraints of energy storage include the state of charge constraint, the charge and discharge power constraint, and the energy balance constraint at the beginning and end of the scheduling cycle, as shown in the following formulas:

[0165] SOC min ≤SOC(t)≤SOC max

[0166] 0≤P ch (t)≤λ ch,t P ch,max

[0167] 0≤P dis (t)≤(1-λ ch,t ) dis,max

[0168] λ ch,t ∈{0,1}

[0169] SOC(0)=SOC(T)

[0170] In the formula, SOC min , SOC max are the upper and lower limits of the battery state of charge respectively; P ch,max , P dis,max are the maximum charge and discharge power of the battery respectively; ch,t is the coefficient representing the charge and discharge state of the battery in period t.

[0171] To ensure the upper and lower limits of the output action, the output action is restricted to [-1, 1] by the tanh function, and the upper and lower limits can be met by linear transformation. The following penalty terms are added to ensure that the action meets the constraints:

[0172]

[0173] φ 2 =k 2 |SOC(0)-SOC(T)|

[0174] In the formula, k 1 , k 2 is the penalty coefficient.

[0175] Therefore, through all multi-energy microgrids from the agent's optimization objective function minf MG,i , we can get the total optimization objective function f of the multi-energy microgrid from the intelligent agent MG , through the multi-energy microgrid from the overall optimization objective function f of the intelligent agent MG And penalty constraints, the reward function R of the multi-energy microgrid from the intelligent agent is obtained t =-(f MG +φ 1 +φ 2 ).

[0176] (4) Multi-agent reinforcement learning training based on the improved MADDPG algorithm

[0177] The overall training framework of the improved MADPPG algorithm is as follows Figure 3 As shown, the following steps are included:

[0178] 1) Step 1: Construct the actor and critic neural network. According to the perceived environment state, define the state space of the operator master agent, which contains the global state information of the multi-energy network of the integrated energy system; define the action space of the operator master agent as the energy exchange price of the integrated energy multi-energy microgrid. Define the state space of the multi-energy microgrid slave agent as the local state information of the multi-energy network of the integrated energy system that the subject can observe; define the action space of the multi-energy microgrid slave agent as the output of various energy resources. According to the size, select the number of hidden layers and the number of neuron parameters.

[0179] 2) Step 2: The training cycle starts. According to the initial parameters of each device in the energy system, the power flow parameters of the electric / gas / heat energy system are calculated to generate the environmental state s at time t t ;

[0180] 3) Step 3: The main agent actor network generates action aa at time t according to the environment state st t , and update the environmental parameter s′ t ; Then, the agent is based on the latest environment state s′ t Generate action at time t

[0181] 4) Step 4: Each agent observes the environmental state of the energy system according to the action at time t, calculates the power flow and generates the environmental state st+1 at time t+1;

[0182] 5) Step 5: Each agent takes action according to time t and the environmental state s at time t+1 t+1 Calculate the action at time t The reward value of

[0183] 6) Step 6: Set the environmental state parameter s in period t t , the actions of the operator's main intelligent agent aa t 、Environmental state parameters s in period t+1 t+1 , the reward value R of the t period is a four-tuple {s t , aa t ,s t+1 , R} is sent to the experience playback unit as a sample;

[0184] The experience replay unit is set as an update mechanism. New samples are continuously generated from step 2 to step 6 and transmitted to the experience replay unit, and old samples that exceed the limit are automatically deleted. When the experience replay unit is not full, steps 2 to step 6 are repeated, and step 7 is executed when the experience replay unit is full.

[0185] 7) Step 7: Use fixed Batch for sampling, calculate the gradient and then update the parameters of the deep neural network constructed in step 1; the review network performs network learning according to the following loss function:

[0186]

[0187] Where y is the target action network Q value; is the Q value of the target review network; R tis the reward function, as shown in formula (22); s is the state; a is the relationship vector transmitted from the target action network to the target comment network; γ is the discount factor; L(θ) is the square loss of the Q value of the target action network and the Q value of the target comment network; θ is the parameter set of the target action network; E represents the average value.

[0188]

[0189] Where J is the objective function of the target action network; θ is the parameter set of the target action network; s is the state; D is the full set of state space; μ represents the deterministic action output by the target action network; Q μ (s,a) is the Q value of the deterministic action μ; a is the relationship transmitted from the target action network to the target comment network; is the gradient representation.

[0190] 8) Step 8: Output the results of this round of training and determine whether the loop has ended. If it has ended, save the neural network model. If it has not ended, repeat steps 2 to 7.

[0191] The present invention establishes an improved MADDPG method for master-slave games. Compared with the existing methods, the multi-agent hierarchical game collaborative model in this application realizes the combination of centralized management and distributed control of each unit in the integrated energy system, enhances the collaborative scheduling of the integrated energy system, increases the reliability of operation, reduces the communication and computing burden of the system, and also enhances the adaptability, generalization ability and decision-making ability in the face of complex environments.

[0192] The following will be combined with the attached Figure 4 , a multi-agent hierarchical game coordination device for an integrated energy microgrid group provided by an embodiment of the present invention is described, and the device includes:

[0193] The first building block is used to create multi-agent deep reinforcement learning parameters;

[0194] The second building unit is used to create an operator master agent;

[0195] A third construction unit, configured to create a multi-energy microgrid slave agent for each of the multi-energy microgrids in the integrated energy system;

[0196] The reinforcement learning training unit is used to perform reinforcement learning training on the pre-created neural network based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent using the improved multi-agent deep deterministic policy gradient MADDPG algorithm to obtain a multi-agent hierarchical game collaborative model.

[0197] Based on the above embodiment, further, the multi-agent deep reinforcement learning parameters include environment state parameters, action space and reward function;

[0198] Environmental status parameters include electricity flow data, gas flow data, heat flow data, and market pricing of the operator's main intelligent body;

[0199] The action space in the multi-agent deep reinforcement learning parameters includes the action space of the operator master agent and the action space of the multi-energy microgrid slave agent;

[0200] The reward function in the multi-agent deep reinforcement learning parameters is established through the optimization objective function of the multi-energy microgrid slave agent and the penalty constraints of the multi-energy microgrid slave agent.

[0201] Based on the above embodiment, further, a reward function establishment module is included, which is used to establish the optimization objective function minf of the i-th multi-energy microgrid from the intelligent agent according to the gas cost of the i-th multi-energy microgrid in period t, the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, and the energy exchange cost of the i-th multi-energy microgrid in period t. MG,i , the optimization objective function is as follows:

[0202] minf MG,i =C i,fuel (t)+C i,mg (t)+C i,grid (t)

[0203]

[0204]

[0205] C i,grid (t) = c grid (t)P i,grid (t)

[0206] In the formula, c gas is the natural gas price, C i,fuel (t) is the gas cost of the i-th multi-energy microgrid in period t, C i,mg (t) is the energy exchange cost of the i-th multi-energy microgrid in period t, C i,grid (t) is the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, P i,MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, c mg (t) is the energy exchange pricing between multi-energy microgrids issued by the operator master agent during time period t, P ij,mg (t) is the interactive power between the i-th multi-energy microgrid and the j-th multi-energy microgrid during period t, P i,grid (t) is the interactive power between the i-th multi-energy microgrid and the distribution network in period t, V i,GB(t) is the gas consumption of the gas boiler GB in the i-th multi-energy microgrid during period t, η MT is the power generation efficiency, LHV NG is the calorific value of natural gas;

[0207] Establish a multi-energy microgrid based on the penalty constraints of the intelligent agent:

[0208]

[0209] φ 2 =k 2 |SOC(0)-SOC(T)|

[0210] In the formula, k 1 and k 2 is the penalty coefficient, E(t+1) is the charge of the energy stored in period t, SOC(0) is the state of charge of all batteries in the initial period, SOC(T) is the state of charge of all batteries in the total scheduling period T, P ch,max is the maximum charging power of the battery, P dis,max is the maximum discharge power of the battery;

[0211] Through all multi-energy microgrids from the intelligent agent's optimization objective function minf MG,i , we can get the total optimization objective function f of the multi-energy microgrid from the intelligent agent MG , through the multi-energy microgrid from the overall optimization objective function f of the intelligent agent MG And penalty constraints, the reward function R of the multi-energy microgrid from the intelligent agent is obtained t =-(f MG +φ 1 +φ 2 ).

[0212] Based on the above embodiment, further, it also includes an operation constraint condition establishment module, which is used to establish the operation constraint conditions of the multi-energy microgrid slave intelligent body;

[0213] The operating constraints of the multi-energy microgrid from the intelligent agent include:

[0214] The real-time balance constraints of the cooling load, heating load and electrical load inside the multi-energy microgrid and the photovoltaic output and wind turbine output are as follows:

[0215] Q i,MT,c (t)+Q i,EC (t) = Q i,load,c (t)

[0216] Q i,MT,h (t)+Q i,GB (t) = Q i,load,h (t)

[0217]

[0218] In the formula, Q i,load,c (t) is the cooling load of the i-th multi-energy microgrid in period t, Q i,load,h (t) is the heat load of the i-th multi-energy microgrid in period t, P i,load (t) is the number of electric loads of the i-th multi-energy microgrid in period t, P i,PV (t) is the photovoltaic output, P i,WT (t) is the fan output, Q i,load,c (t) is the cooling load of the i-th multi-energy microgrid in period t, Q i,load,h (t) is the heat load of the i-th multi-energy microgrid in period t, P i,load (t) is the load of the i-th multi-energy microgrid in period t, Q i,MT,c (t) is the heating power of the micro gas turbine MT in the i-th multi-energy microgrid during time period t, Q i,EC (t) is the cooling power of the electric refrigerator EC in the i-th multi-energy microgrid during time period t, Q i,n,EC (t) is the cooling power of the nth electric refrigerator EC in the i-th multi-energy microgrid during time period t, Q i,MT,h (t) is the cooling power of the micro gas turbine MT in the i-th multi-energy microgrid during time period t, Q i,GB (t) is the heating power of the gas boiler GB in the i-th multi-energy microgrid during period t, P i,MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, P i,PV (t) is the output power of the photovoltaic device PV in the i-th microgrid during period t, P i,WT (t) is the output power of wind turbine WT in the i-th multi-energy microgrid during period t, P i,EC (t) is the charging power of the electric refrigerator EC in the i-th multi-energy microgrid during time period t, P i,ch (t) is the charging power of the energy storage ES in the i-th multi-energy microgrid during period t, P i,dis (t) is the discharge power of the energy storage ES in the i-th multi-energy microgrid during period t;

[0219] Equipment operation constraints and power interaction constraints, where equipment operation constraints include micro gas turbine operation constraints, gas boiler operation constraints, and electric refrigerator operation constraints:

[0220] P MT,min ≤P MT (t)≤P MT,max

[0221] Q GB,min ≤Q GB (t)≤Q GB,max

[0222] P EC,min ≤P EC (t)≤P EC,max

[0223] P i,grid,min ≤P i,grid (t)≤P i,grid,max

[0224] Where P MT,min is the lower limit of gas turbine output, P MT,max is the upper limit of the gas turbine output, P MT (t) is the gas turbine output value during period t; Q GB,min is the lower limit of the thermal power output of the gas boiler, Q GB,max is the upper limit of the thermal power output of the gas boiler, Q GB (t) is the thermal power output of the gas boiler during period t, Q GB,min is the lower limit of the thermal power output of the gas boiler, Q GB,max The upper limit of the thermal power output of the gas boiler; P EC,min is the lower limit of the input power of the electric refrigerator, P EC,max is the upper limit of the input power of the electric refrigerator, P EC (t) is the input power of the refrigerator during period t, P i,grid,min is the lower limit of the interaction power between the i-th multi-energy microgrid and the distribution network, P i,grid,max are the upper limits of the interaction power between microgrid i and distribution network, P i,grid,min is the lower limit of the interaction power between the i-th multi-energy microgrid and the distribution network, P i,grid,max is the upper limit of the interaction power between the i-th multi-energy microgrid and the distribution network;

[0225] The operation constraints of energy storage include state of charge constraints, charge and discharge power constraints, and energy balance constraints at the beginning and end of the dispatch cycle, as follows:

[0226] SOC min ≤SOC(t)≤SOC max

[0227] 0≤P ch (t)≤λ ch,t P ch,max

[0228] 0≤P dis (t)≤(1-λ ch,t ) dis,max

[0229] λ ch,t ∈{0,1}

[0230] SOC(0)=SOC(T)

[0231] In the formula, SOC min SOC is the lower limit of the battery state of charge. max is the upper limit of the battery charge state; P ch,max is the maximum charging power of the battery, P dis,max is the maximum discharge power of the battery, λ ch,t is the charge and discharge state coefficient of the battery in period t, SOC(t) is the state of charge value of all batteries in period t, P ch (t) is the charging power of the energy storage ES during period t, P dis (t) is the discharge power of the energy storage ES during period t.

[0232] Based on the above embodiment, further, the reinforcement learning training unit also includes a first processing module, a second processing module, a third processing module, a fourth processing module, a fifth processing module, a sixth processing module, a seventh processing module and an eighth processing module.

[0233] The first processing module is used to build an actor-critic deep reinforcement learning model, which consists of a master agent actor network, a multi-energy microgrid slave agent actor network, and a critic network;

[0234] From the environmental state parameters in the multi-agent deep reinforcement learning parameters, the global state information of the multi-energy network in the state space of the operator's main agent is obtained;

[0235] The energy exchange price of the multi-energy microgrid is used as the action space of the operator's main agent;

[0236] The output value of energy resources is used as the action space of the slave agent of the multi-energy microgrid;

[0237] The second processing module is used to obtain the environmental state parameter s of the actor-critic deep reinforcement learning model during period t according to the state information of the multi-energy microgrid. t ,Environmental state parameters include electric current data, gas current data and thermal current data;

[0238] The third processing module is used for the main agent actor network to process the environment state parameter s t , generating the action aa of the operator master agent in period t t , update the environmental state parameter s t After that, the updated environmental state parameter s′ is obtained t ;

[0239] The i-th multi-energy microgrid is fed from the intelligent agent actor network according to the updated environmental state parameter s t ', generate the multi-energy microgrid from the action of the intelligent agent in period t

[0240] The fourth processing module is used to perform the actions of the multi-energy microgrid slave agent in the multi-energy microgrid slave agent actor network during the t period. Observe the environmental state parameters and calculate the environmental state parameters s for the period t+1 t+1 ;

[0241] The fifth processing module is used to process the multi-energy microgrid from the intelligent agent actor network in the t period Environmental state parameter s at time t+1 t+1 , the operating constraints of the multi-energy microgrid slave agent and the reward function of the multi-energy microgrid slave agent, and calculate the reward value R of the operator's master agent in time period t;

[0242] The sixth processing module is used to convert the environmental state parameter s in the t period into t , the actions of the operator's main intelligent agent aa t 、Environmental state parameters s in period t+1 t+1 , the reward value R of the t period is a four-tuple {s t , aa t ,s t+1 , R} is sent to the experience playback unit as a sample;

[0243] Determine whether the experience playback unit is full;

[0244] If not, return to step 2 and continue to execute;

[0245] If yes, execute step 7 processing module;

[0246] The seventh processing module is used to obtain samples from the experience replay unit and update the parameters of the actor-critic deep reinforcement learning model using the gradient algorithm through the samples;

[0247] Among them, the critic neural network is updated according to the loss function, environment state parameters and reward function;

[0248] The eighth processing module is used to output the training results of this round and determine whether the preset cycle end condition is met;

[0249] If so, the loop ends and a multi-agent hierarchical game coordination model is obtained;

[0250] Otherwise, return to step 2 of the processing module.

[0251] Based on the above embodiment, further, the multi-energy network global state information in the state space of the operator's main intelligent body includes electric network state parameter values, thermal network parameter values ​​and gas network parameter values.

[0252] Based on the above embodiments, further, the status information of the multi-energy microgrid includes the electric network status value, thermal network parameter value, gas network parameter value of the multi-energy microgrid and the energy exchange pricing between the multi-energy microgrids issued by the operator's main intelligent body.

[0253] The embodiments of the present invention realize the combination of centralized management and distributed control of each unit in the integrated energy system, thereby enhancing the reliable operation and control of the integrated energy system.

[0254] In addition, an embodiment of the present invention includes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements a multi-agent hierarchical game collaboration method for an integrated energy microgrid group described in any one of the above technical solutions.

[0255] An embodiment of the present invention also includes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for multi-agent hierarchical game coordination of an integrated energy microgrid group described in any one of the above technical solutions is implemented.

[0256] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the specific implementation modes of the present invention can still be modified or replaced by equivalents, and any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A multi-agent hierarchical game coordination method for integrated energy microgrids, characterized in that: The method comprises: Create multi-agent deep reinforcement learning parameters; Create an operator master agent; Create a multi-energy microgrid slave agent for each multi-energy microgrid in the integrated energy system; Based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent, the pre-created neural network is trained by using the improved multi-agent deep determination policy gradient MADDPG algorithm to obtain a multi-agent hierarchical game coordination model; The multi-agent deep reinforcement learning parameters include environment state parameters, action space and reward function; The environmental state parameters include electric power flow data, gas power flow data, thermal power flow data and the market pricing of the operator's main intelligent body; The action space in the multi-agent deep reinforcement learning parameters includes the action space of the operator master agent and the action space of the multi-energy microgrid slave agent; The reward function in the multi-agent deep reinforcement learning parameters is established by the optimization objective function of the multi-energy microgrid slave agent and the penalty constraint condition of the multi-energy microgrid slave agent; The reward function in the multi-agent deep reinforcement learning parameters is established by the optimization objective function of the multi-energy microgrid slave agent and the penalty constraint condition of the multi-energy microgrid slave agent, specifically including: According to the gas cost of the i-th multi-energy microgrid in period t, the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, and the energy exchange cost of the i-th multi-energy microgrid in period t, the optimization objective function min f of the i-th multi-energy microgrid from the intelligent agent is established. MG，i , the optimization objective function is as follows: min f MG，i =C i，fuel (t)+C i，mg (t)+C i，grid (t) C i，grid (t)=c grid (t)P i，grid (t) In the formula, c gas is the natural gas price, C i，fuel (t) is the gas cost of the i-th multi-energy microgrid in period t, C i，mg (t) is the energy exchange cost of the i-th multi-energy microgrid in period t, C i，grid (t) is the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, P i，MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, c mg (t) is the energy exchange pricing between multi-energy microgrids issued by the operator master agent during time period t, P ij，mg (t) is the interactive power between the i-th multi-energy microgrid and the j-th multi-energy microgrid during period t, P i，grid (t) is the interactive power between the i-th multi-energy microgrid and the distribution network in period t, V i，GB (t) is the gas consumption of the gas boiler GB in the i-th multi-energy microgrid during period t, η MT is the power generation efficiency, LHV NG is the calorific value of natural gas; Establish the penalty constraints of the multi-energy microgrid from the intelligent agent: (E(t)-E(t+1)-P dis，max ),0]} φ2=k2|SOC(0)-SOC(T)| Where k1 and k2 are penalty coefficients, E(t+1) is the charge of the energy stored in period t, SOC(0) is the state of charge of all batteries in the initial period, SOC(T) is the state of charge of all batteries in the total scheduling period T, and P ch，max is the maximum charging power of the battery, P dis，max is the maximum discharge power of the battery; Through all the multi-energy microgrids, the optimization objective function minf of the intelligent agent MG，i , the total optimization objective function f of the multi-energy microgrid from the intelligent agent is obtained MG , through the multi-energy microgrid from the total optimization objective function f of the intelligent agent MG and the penalty constraint, the reward function R of the multi-energy microgrid from the intelligent agent is obtained t =-(f MG +φ1+φ2).

2. The method according to claim 1, characterized in that The method further includes establishing operating constraints of the multi-energy microgrid slave agent; The operating constraints of the multi-energy microgrid slave agent include: The cooling load, heating load and electric load in the multi-energy microgrid are respectively balanced with the photovoltaic output and the wind turbine output in real time: Q i，MT，c (t)+Q i，EC (t)=Q i，load，c (t) Q i，MT，h (t)+Q i，GB (t)=Q i，load，h (t) Where P i，PV (t) is the photovoltaic output, P i，WT (t) is the fan output, Q i，load，c (t) is the cooling load of the i-th multi-energy microgrid in period t, Q i，load，h (t) is the heat load of the i-th multi-energy microgrid in period t, P i，load (t) is the load of the i-th multi-energy microgrid in period t, Q i，MT，c (t) is the heating power of the micro gas turbine MT in the i-th multi-energy microgrid during time period t, Q i，EC (t) is the cooling power of the electric refrigerator EC in the i-th multi-energy microgrid during time period t, Q i，MT，h (t) is the cooling power of the micro gas turbine MT in the i-th multi-energy microgrid during time period t, Q i，GB (t) is the heating power of the gas boiler GB in the i-th multi-energy microgrid during period t, P i，MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, P i，PV (t) is the output power of the photovoltaic device PV in the i-th microgrid during period t, P i，WT (t) is the output power of wind turbine WT in the i-th multi-energy microgrid during period t, P i，EC (t) is the charging power of the electric refrigerator EC in the i-th multi-energy microgrid during time period t, P i，ch (t) is the charging power of the energy storage ES in the i-th multi-energy microgrid during period t, P i，dis (t) is the discharge power of the energy storage ES in the i-th multi-energy microgrid during period t; Equipment operation constraints and power interaction constraints, where equipment operation constraints include micro gas turbine operation constraints, gas boiler operation constraints, and electric refrigerator operation constraints: P MT，min ≤P MT (t)≤P MT，max Q GB，min ≤Q GB (t)≤Q GB，max P EC，min ≤P EC (t)≤P EC，max P i，grid，min ≤P i，grid (t)≤P i，grid，max Where P MT，min is the lower limit of gas turbine output, P MT，max is the upper limit of the gas turbine output, P MT (t) is the gas turbine output value during period t; Q GB (t) is the thermal power output of the gas boiler during period t, Q GB，min is the lower limit of the thermal power output of the gas boiler, Q GB，max The upper limit of the thermal power output of the gas boiler; P EC，min is the lower limit of the input power of the electric refrigerator, P EC，max is the upper limit of the input power of the electric refrigerator, P EC (t) is the input power of the refrigerator during period t, P i，grid，min is the lower limit of the interaction power between the i-th multi-energy microgrid and the distribution network, P i，grid，max are the upper limits of the interaction power between microgrid i and distribution network respectively; The operation constraints of energy storage include state of charge constraints, charge and discharge power constraints, and energy balance constraints at the beginning and end of the scheduling period, as follows: SOC min ≤SOC(t)≤SOC max 0≤P ch (t)≤λ ch，t P ch，max 0≤P dis (t)≤(1-λ ch，t )P dis，max l ch，t ∈{0,1} SOC(0)=SOC(T) In the formula, SOC min SOC is the lower limit of the battery state of charge. max is the upper limit of the battery charge state; P ch，max is the maximum charging power of the battery, P dis，max is the maximum discharge power of the battery, λ ch，t is the charge and discharge state coefficient of the battery in period t, SOC(t) is the state of charge value of all batteries in period t, P ch (t) is the charging power of the energy storage ES during period t, P dis (t) is the discharge power of the energy storage ES during period t.

3. The method according to claim 2, characterized in that The multi-agent hierarchical game coordination model is obtained by performing reinforcement learning training on the pre-created neural network based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent using the improved MADDPG algorithm, specifically including: S1. Construct an actor-critic deep reinforcement learning model, which consists of a master agent actor network, a multi-energy microgrid slave agent actor network, and a critic network. Obtaining the multi-energy network global state information in the state space of the operator master agent from the environment state parameters in the multi-agent deep reinforcement learning parameters; The energy exchange price of the multi-energy microgrid is used as the action space of the operator's main intelligent agent; The output value of the energy resource is used as the action space of the slave agent of the multi-energy microgrid; S2. According to the state information of the multi-energy microgrid, the environmental state parameter s of the actor-critic deep reinforcement learning model during period t is obtained. t , the environmental state parameters include the electric current data, the gas current data and the thermal current data; S3, the main agent actor network according to the environmental state parameter s t , generating the action aa of the operator master agent in period t t , update the environmental state parameter s t After that, the updated environmental state parameter s′ is obtained t ; The i-th multi-energy microgrid is derived from the intelligent agent actor network according to the updated environmental state parameter s t , generating the multi-energy microgrid from the actions of the intelligent agent in period t S4, according to the multi-energy microgrid slave intelligent agent actor network in the multi-energy microgrid slave intelligent agent action in time period t Observe the environmental state parameters and calculate the environmental state parameters s for the period t+1 t+1 ; S5, according to the multi-energy microgrid from the intelligent agent actor network in the t period the multi-energy microgrid from the intelligent agent Environmental state parameter s at time t+1 t+1 , the operating constraints of the multi-energy microgrid slave agent and the reward function of the multi-energy microgrid slave agent, and calculate the reward value R of the operator master agent in time period t; S6, the environmental state parameter s described in time period t t , the action of the operator's main intelligent agent aa t , the environmental state parameter s in the period t+1 t+1 , the reward value R of the t period is composed of a four-tuple {s t , aa t ,s t+1 , R} is sent to the experience playback unit as a sample; Determining whether the experience playback unit is full; If not, return to step S2 to continue execution; If yes, execute step S7; S7, sampling from the experience replay unit to obtain samples, and using the samples to update parameters of the actor-critic deep reinforcement learning model using a gradient algorithm; wherein the critic neural network is updated according to the loss function, the environment state parameter and the reward function; S8, output the training results of this round, and determine whether the preset cycle end condition is met; If yes, then the loop ends and the multi-agent hierarchical game coordination model is obtained; Otherwise, return to step S2.

4. The method according to claim 3, characterized in that The multi-energy network global state information in the state space of the operator master intelligent body includes electric network state parameter values, thermal network parameter values ​​and gas network parameter values.

5. The method according to claim 4, characterized in that The status information of the multi-energy microgrid includes the electric network status value, thermal network parameter value, gas network parameter value of the multi-energy microgrid and the energy exchange pricing between the multi-energy microgrids issued by the operator's main intelligent body.

6. A multi-agent hierarchical game coordination device for integrated energy microgrid groups, characterized in that: The device comprises: The first building block is used to create multi-agent deep reinforcement learning parameters; The second building unit is used to create an operator master agent; A third construction unit is used to create a multi-energy microgrid slave agent for each multi-energy microgrid in the integrated energy system; A reinforcement learning training unit is used to perform reinforcement learning training on a pre-created neural network based on the multi-agent deep reinforcement learning parameters, the operator master agent and the multi-energy microgrid slave agent using an improved multi-agent deep determination policy gradient MADDPG algorithm to obtain a multi-agent hierarchical game coordination model; The multi-agent deep reinforcement learning parameters include environment state parameters, action space and reward function; The environmental state parameters include electric power flow data, gas power flow data, thermal power flow data and the market pricing of the operator's main intelligent body; The action space in the multi-agent deep reinforcement learning parameters includes the action space of the operator master agent and the action space of the multi-energy microgrid slave agent; The reward function in the multi-agent deep reinforcement learning parameters is established by the optimization objective function of the multi-energy microgrid slave agent and the penalty constraint condition of the multi-energy microgrid slave agent; The reward function in the multi-agent deep reinforcement learning parameters is established by the optimization objective function of the multi-energy microgrid slave agent and the penalty constraint condition of the multi-energy microgrid slave agent, specifically including: According to the gas cost of the i-th multi-energy microgrid in period t, the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, and the energy exchange cost of the i-th multi-energy microgrid in period t, the optimization objective function min f of the i-th multi-energy microgrid from the intelligent agent is established. MG，i , the optimization objective function is as follows: min f MG，i =C i，fuel (t)+C i，mg (t)+C i，grid (t) C i，grid (t)=c grid (t)P i，grid (t) In the formula, c gas is the natural gas price, C i，fuel (t) is the gas cost of the i-th multi-energy microgrid in period t, C i，mg (t) is the energy exchange cost of the i-th multi-energy microgrid in period t, C i，grid (t) is the interaction cost between the i-th multi-energy microgrid and the distribution network in period t, P i，MT (t) is the output power of the micro gas turbine MT in the i-th multi-energy microgrid during period t, c mg (t) is the energy exchange pricing between multi-energy microgrids issued by the operator master agent during time period t, P ij，mg (t) is the interactive power between the i-th multi-energy microgrid and the j-th multi-energy microgrid during period t, P i，grid (t) is the interactive power between the i-th multi-energy microgrid and the distribution network in period t, V i，GB (t) is the gas consumption of the gas boiler GB in the i-th multi-energy microgrid during period t, η MT is the power generation efficiency, LHV NG is the calorific value of natural gas; Establish the penalty constraints of the multi-energy microgrid from the intelligent agent: (E(t)-E(t+1)-P dis，max ),0]} φ2=k2|SOC(0)-SOC(T)| Where k1 and k2 are penalty coefficients, E(t+1) is the charge of the energy stored in period t, SOC(0) is the state of charge of all batteries in the initial period, SOC(T) is the state of charge of all batteries in the total scheduling period T, and P ch，max is the maximum charging power of the battery, P dis，max is the maximum discharge power of the battery; Through all the multi-energy microgrids, the optimization objective function minf of the intelligent agent MG，i , the total optimization objective function f of the multi-energy microgrid from the intelligent agent is obtained MG , through the multi-energy microgrid from the total optimization objective function f of the intelligent agent MG and the penalty constraint, the reward function R of the multi-energy microgrid from the intelligent agent is obtained t =-(f MG +φ1+φ2).

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the multi-agent hierarchical game collaboration method for the integrated energy microgrid group described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-agent hierarchical game collaboration method for an integrated energy microgrid group described in any one of claims 1 to 5 is implemented.

Images