Interconnected micro-grid group optimization scheduling method based on multi-agent deep learning

Abstract

The invention discloses an interconnected micro-grid group optimization scheduling method based on multi-agent deep learning, and the method comprises the steps: building an optimization scheduling network model of an interconnected micro-grid group system, the interconnected micro-grid group system comprises a main power grid and n micro-grids, and the optimization scheduling network model comprises n agents and M hybrid network modules, and the micro-grids are in one-to-one correspondence with the intelligent agents. According to the method, the strategy action of each micro-grid is dynamically adjusted under the continuously changing environment information through the optimization scheduling model, more efficient and more collaborative energy management is realized, and the optimization scheduling model is composed of a plurality of intelligent agents and a hybrid network module, so that the energy management efficiency is improved. In the training process, the total target Q value and the total estimation Q value are adopted to calculate the loss function of each module, the overhigh deviation in Q value estimation is effectively reduced, the strategy is prevented from being excessively optimistic, and the scheduling strategy is more stable and reliable in practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of microgrid cooperation, and specifically relates to an optimization scheduling method for interconnected microgrid groups based on multi-agent deep learning. Background Art

[0002] A microgrid is a small energy network that integrates distributed energy, loads, energy storage equipment, and monitoring systems, which can achieve flexible energy supply and efficient management. In grid-connected mode, it can enhance the stability of the power system, and can also operate independently when the main power grid fails to ensure power supply in key areas. Therefore, microgrids are of great significance in improving energy utilization efficiency and power supply security. However, the energy production and storage capacity of a single microgrid is limited, making it difficult to cope with large-scale energy demand fluctuations, and lacks flexibility under external interference. For this reason, interconnected microgrid groups have emerged. Through the interconnection of multiple microgrids, energy and information sharing can be achieved, energy production and distribution can be optimized, operating costs can be reduced, and system flexibility can be improved. As the scale of interconnected microgrid groups expands, their management and scheduling become more complex.

[0003] Traditional control methods such as mathematical programming and heuristic algorithms usually have the disadvantages of high computational complexity and strong dependence on precise models. They find it difficult to handle the complex dynamic relationships in the system and the uncertainty of the external environment. Especially when faced with load fluctuations and the intermittent nature of renewable energy, the scheduling effect is easily affected. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and to propose an optimization scheduling method for interconnected microgrid groups based on multi-agent deep learning.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes an interconnected microgrid group optimization scheduling method based on multi-agent deep learning, including establishing an optimization scheduling network model of an interconnected microgrid group system, the interconnected microgrid group system includes a main grid and n microgrids, the optimization scheduling network model includes n agents and M hybrid network modules, and the microgrids correspond to the agents one by one, the agents include Actor network modules and M Critic network modules, and a Critic network module and a hybrid network module are selected from each agent as a module, so there are a total of M modules, and the Critic network module and the hybrid network module in each module are different;

[0007] Obtain the current environmental information of each microgrid in the interconnected microgrid group system and initialize the parameters of the optimization scheduling network model;

[0008] Obtain the current state information from the environmental information of each microgrid, and input it into the Actor network module of the intelligent body accordingly to obtain the current strategic action of each microgrid. Based on the strategic action and current state information, calculate the internal reward of each microgrid at the current moment.

[0009] According to the energy supply and consumption of each microgrid in the interconnected microgrid group system, the amount of electricity that needs to be traded in each microgrid at the current moment is calculated through the electricity balance equation;

[0010] According to the balance of energy input and output between microgrids and between microgrids and the main grid at the current moment in the interconnected microgrid group system, the trading strategy of each microgrid at the current moment is solved, and the trading reward of each microgrid at the current moment is calculated;

[0011] The sum of the internal rewards and transaction rewards of each microgrid at the current moment is taken as the total reward at the current moment. The current state information, strategy actions, total rewards of all microgrids and the state information of all microgrids at the next moment are stored in the experience pool as the same batch of experience, and the optimization scheduling network model is trained based on the experience pool:

[0012] Randomly sample a preset batch of experience from the experience pool. For each batch of experience sampled, input the state information of each microgrid at the next moment into the target Actor network of the corresponding Actor network module to obtain the strategy action at the next moment.

[0013] For the target critic network of the critic network module, randomly select G modules, input the state information of each microgrid at the next moment and the corresponding strategy action at the next moment into the target critic network corresponding to the G modules to obtain the target Q value, then input all the target Q values ​​in each module into the hybrid network module of the corresponding module to obtain the total target Q value, and then select the minimum total target Q value from the G total target Q values ​​as the final total target Q value;

[0014] For the estimated critic network of the critic network module, the current state information of each microgrid and the current strategic action are input into the target critic network corresponding to the M modules to obtain the estimated Q value, and then all the estimated Q values ​​of each module are input into the hybrid network module of the corresponding module to obtain the total estimated Q value of each module;

[0015] The final total target Q value and the total estimated Q value are used to calculate the loss function of each module. The parameters of the Critic network module and the hybrid network module of each module are updated through the loss function, and the parameters of the Actor network module are updated using the centralized gradient estimator.

[0016] Until the strategic actions output by the target Actor network of each intelligent agent make the sum of all total rewards in the entire optimization scheduling process converge to the optimal value, the optimal strategy of each microgrid is obtained.

[0017] Preferably, the n microgrids in the interconnected microgrid system are numbered as a total set N, and N={1,2,...,i,...n}, and the equipment included in each microgrid includes: at least one of wind power generation equipment, photovoltaic power generation equipment, electric load equipment, water electrolysis hydrogen production equipment, hydrogen storage tank equipment, fuel cell equipment, energy storage battery equipment, diesel generator equipment, thermal load equipment, electric boiler equipment and cogeneration equipment, and the environmental information of each microgrid is the environmental information of the equipment in each microgrid;

[0018] The environmental information of the wind power generation equipment is defined as follows: The electric power output of the wind power generation equipment of the i-th microgrid at time t is defined as i∈N;

[0019] The environmental information of the photovoltaic power generation equipment is defined as follows: The power output of the photovoltaic power generation equipment of the i-th microgrid at time t is i∈N;

[0020] The environmental information of the electric load device is defined as follows: The electric power consumed by the electric load device of the i-th microgrid at time t is defined as i∈N;

[0021] The environmental information of the thermal load device is defined as follows: The power consumed by the thermal load device of the i-th microgrid at time t is defined as i∈N;

[0022] The environmental information of the water electrolysis hydrogen production equipment satisfies the following relationship:

[0023]

[0024] in,

[0025]

[0026] in, represents the working cost of the water electrolysis hydrogen production equipment of the i-th microgrid at time t, represents the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t and time t-1, a EHP , b EHP and c EHP These are the coefficients for calculating the working cost of water electrolysis hydrogen production equipment. represents the maximum electric power input by the water electrolysis hydrogen production equipment of the i-th microgrid, represents the ramp power of the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t (the ramp power is the difference between the electric power at the current moment and the electric power at the previous moment), represents the maximum and minimum values ​​of the ramp power input by the water electrolysis hydrogen production equipment of the i-th microgrid, η EHP It indicates the efficiency of water electrolysis hydrogen production equipment in converting input electrical energy into hydrogen energy. It represents the hydrogen power after the water electrolysis hydrogen production equipment of the i-th microgrid converts the input electrical energy into hydrogen energy at time t, i∈N;

[0027] The environmental information of the hydrogen storage tank equipment satisfies the following relationship:

[0028]

[0029] in,

[0030]

[0031] in, represents the hydrogen energy stored in the hydrogen storage tank equipment of the i-th microgrid at time t+1 and time t, α HT , They represent the hydrogen storage attenuation coefficient, hydrogen storage energy efficiency and hydrogen release energy efficiency respectively. represents the hydrogen storage power of the hydrogen storage tank equipment of the i-th microgrid at time t, Indicates the upper limit of hydrogen storage power, The hydrogen release power of the hydrogen storage tank equipment of the i-th microgrid at time t is: Indicates the upper limit of hydrogen release power, They represent the upper and lower limits of the hydrogen storage tank’s capacity for storing hydrogen energy, respectively, Δt represents the time interval between adjacent moments, i∈N;

[0032] The environmental information of the fuel cell device satisfies the following relationship:

[0033]

[0034] in,

[0035]

[0036] in, represents the operating cost of the fuel cell equipment of the i-th microgrid at time t, and represents the electric power output of the fuel cell device of the i-th microgrid at time t and time t-1, a FC , b FC and c FCThese are coefficients for calculating the operating cost of fuel cell equipment. represents the maximum electric power output by the fuel cell device of the i-th microgrid, represents the ramp power of the electric power output by the fuel cell device of the i-th microgrid at time t, represents the maximum and minimum values ​​of the ramp power input by the fuel cell device of the i-th microgrid, η FC Indicates the efficiency of the fuel cell device in converting the input hydrogen energy into electrical energy. It represents the power of hydrogen when the fuel cell device of the i-th microgrid converts the input hydrogen energy into electrical energy at time t, i∈N;

[0037] The environmental information of the energy storage battery equipment satisfies the following relationship:

[0038]

[0039] in,

[0040]

[0041] in, represents the power of the energy storage battery equipment of the i-th microgrid at time t+1 and time t, represents the charging power and discharging power of the energy storage battery equipment of the i-th microgrid at time t, Indicates the upper limit of charging power. Indicates the upper limit of discharge power, α, η ch and η dis They represent the energy storage attenuation coefficient, charging efficiency and discharging efficiency respectively. They represent the upper and lower limits of the amount of electricity stored in the energy storage battery, i∈N;

[0042] The environmental information of the diesel generator equipment satisfies the following relationship:

[0043]

[0044] in,

[0045]

[0046] in, represents the operating cost of the diesel generator equipment of the i-th microgrid at time t, and represents the electric power output of the diesel generator equipment of the i-th microgrid at time t and time t-1, represents the maximum power output of the diesel generator equipment in the i-th microgrid, a DG 、b DG and cDG These are coefficients for calculating the operating cost of diesel generator equipment. represents the ramp power of the diesel generator equipment of the i-th microgrid output at time t, represents the maximum and minimum values ​​of the ramp electric power output by the diesel generator equipment of the i-th microgrid, i∈N;

[0047] The environmental information of the electric boiler equipment satisfies the following relationship:

[0048]

[0049] in,

[0050]

[0051] in, represents the operating cost of the electric boiler equipment of the i-th microgrid at time t, represents the electric power input to the electric boiler equipment of the i-th microgrid at time t, represents the maximum electric power input by the electric boiler equipment of the i-th microgrid, a EB , b EB and c EB Both are coefficients for calculating the working cost of electric boiler equipment, η EB Indicates the efficiency of electric boiler equipment in converting input electrical energy into thermal energy. It represents the thermal power after the electric boiler equipment of the i-th microgrid converts the input electrical energy into thermal energy at time t, i∈N;

[0052] The environmental information of the cogeneration equipment satisfies the following relationship:

[0053]

[0054] in,

[0055]

[0056] in, represents the operating cost of the cogeneration equipment of the i-th microgrid at time t, represents the electric power output of the cogeneration equipment of the i-th microgrid at time t, a CHP , b CHP and c CHP These are coefficients for calculating the operating costs of cogeneration equipment. represents the maximum electric power output by the cogeneration equipment of the i-th microgrid, Indicates the efficiency of the cogeneration equipment in converting hydrogen energy into heat energy, It means that the cogeneration equipment of the i-th microgrid converts the hydrogen energy power The thermal power after conversion into thermal energy, represents the efficiency of the cogeneration equipment in converting hydrogen energy into electrical energy, i∈N;

[0057] The state information of each microgrid in the interconnected microgrid system includes: the electric power output of the wind power generation equipment of the i-th microgrid at time t The power output of the photovoltaic power generation equipment of the i-th microgrid at time t The power consumed by the electric load equipment of the i-th microgrid at time t The electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t The hydrogen energy stored in the hydrogen storage tank equipment of the i-th microgrid at time t The power output of the fuel cell device of the i-th microgrid at time t The amount of electricity stored in the energy storage battery device of the i-th microgrid at time t The power output of the diesel generator equipment of the i-th microgrid at time t The power consumed by the thermal load equipment of the i-th microgrid at time t The unit price of electricity sold by each microgrid to the main grid at time t and the unit price of electricity purchased by each microgrid from the main grid at time t At least one of them, i∈N;

[0058] The state information of each microgrid is input into the Actor network module of the corresponding intelligent agent to obtain the strategic action of each microgrid, and the strategic action of each microgrid includes: the electric power charged by the energy storage battery equipment of the i-th microgrid at time t The electric power discharged by the energy storage battery equipment of the i-th microgrid at time t The ramp power of the electric power output by the fuel cell device of the i-th microgrid at time t The ramp power of the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t The ramp power of the diesel generator equipment of the i-th microgrid output at time t The amount of hydrogen purchased by the i-th microgrid at time t and the electric power input to the electric boiler equipment of the i-th microgrid at time t At least one of them, i∈N.

[0059] Preferably, the interconnected microgrid system includes three types of microgrids, namely, a first type of microgrid, a second type of microgrid and a third type of microgrid, and the first type of microgrids are numbered as a first set N 1 , the second type of microgrids are numbered as the second set N 2, the third type of microgrid is numbered as the third set N 3 ,in:

[0060] The equipment included in the first type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, water electrolysis hydrogen production equipment, hydrogen storage tank equipment, fuel cell equipment and energy storage battery equipment;

[0061] And the state information of the i-th microgrid at time t in the first type of microgrid is i∈N 1 , the strategic action of the i-th microgrid at time t is i∈N 1 ;

[0062] The equipment included in the second type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, diesel generator equipment, hydrogen storage tank equipment, fuel cell equipment and energy storage battery equipment;

[0063] And the state information of the i-th microgrid at time t in the second type microgrid is i∈N 2 , the strategic action of the i-th microgrid at time t is i∈N 2 ;

[0064] The equipment included in the third type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, thermal load equipment, electric boiler equipment, cogeneration equipment and energy storage battery equipment;

[0065] And the state information of the i-th microgrid at time t in the third type of microgrid is i∈N 3 , the strategic action of the i-th microgrid at time t is i∈N 3 .

[0066] Preferably, based on the strategy actions and state information, the internal rewards of each microgrid are calculated, including:

[0067] For the first type of microgrid, i∈N 1 , the calculation formula of the internal reward of each microgrid is as follows:

[0068]

[0069] in, represents the internal reward of the i-th microgrid at time t in the first type of microgrid;

[0070] For the second type of microgrid, i∈N 2 , the calculation formula of the internal reward of each microgrid is as follows:

[0071]

[0072] in, represents the internal reward of the i-th microgrid at time t in the second type of microgrid, represents the unit price of hydrogen purchased at time t;

[0073] For the third type of microgrid, i∈N 3 , the calculation formula of the internal reward of each microgrid is as follows:

[0074]

[0075] in, Internal reward of the i-th microgrid at time t in the third type of microgrid.

[0076] Preferably, according to the energy supply and consumption within each microgrid in the interconnected microgrid group system, the amount of electricity to be traded in each microgrid at the current moment is calculated by an electricity balance equation, including:

[0077] For the first type of microgrid, i∈N 1 , the amount of electricity that needs to be traded by the i-th microgrid at time t Satisfies the following relationship:

[0078]

[0079] In the first type of microgrid, during the process of electrolyzing water to produce hydrogen, electrical energy is converted into hydrogen energy and stored in a hydrogen storage tank. When there is a power shortage, a fuel cell device is used to convert the hydrogen energy in the hydrogen storage tank into electrical energy. Then formula (5) is converted into:

[0080]

[0081] For the second type of microgrid, i∈N 2 , the amount of electricity that needs to be traded by the i-th microgrid at time t Satisfies the following relationship:

[0082]

[0083] The second type of microgrid purchases hydrogen and stores it in a hydrogen storage tank. When there is a power shortage, a fuel cell device is used to convert the hydrogen energy in the hydrogen storage tank into electrical energy. Then formula (5) is converted into:

[0084]

[0085] For the third type of microgrid, i∈N 3 , the amount of electricity that needs to be traded by the i-th microgrid at time t Satisfies the following relationship:

[0086]

[0087] The third type of microgrid uses purchased hydrogen. The demand of heat load equipment is met through cogeneration equipment and electric boiler equipment to achieve heat network balance, and the heat network balance equation is expressed as:

[0088]

[0089] Preferably, the trading strategy of each microgrid is solved according to the balance of energy input and output between each microgrid and between each microgrid and the main grid in the interconnected microgrid group system, and the relationship between each microgrid and the main grid is as follows:

[0090]

[0091] in,

[0092]

[0093] Among them, i,j∈N, represents the excess power output from the i-th microgrid to the j-th microgrid at time t, It means that the i-th microgrid receives the excess power transmitted by the j-th microgrid at time t. It means that the i-th microgrid sells excess electricity to the main grid at time t. It indicates that the i-th microgrid purchases electricity from the main grid at time t;

[0094] There is a loss in the transmission of electricity between microgrids, and the relationship satisfied is as follows:

[0095]

[0096] in,

[0097]

[0098] in, represents the total power output of all microgrids, represents the total amount of electricity received by all microgrids, Indicates the amount of power lost during the transmission process, β ij represents the loss coefficient of the power output from the i-th microgrid to the j-th microgrid;

[0099] The overall benefits of each microgrid and the main grid meet the following requirements:

[0100]

[0101] Among them, i,j∈N, represents the total benefits of each microgrid and the main grid at time t, represents the unit price of the excess electricity output from the i-th microgrid to the j-th microgrid at time t, The unit price of the excess electricity that the i-th microgrid exports to the j-th microgrid at time t;

[0102] Combine formulas (38) to (43) to obtain the trading strategy of each microgrid And i,j∈N.

[0103] Preferably, the transaction rewards for each microgrid are as follows:

[0104]

[0105] in, represents the transaction reward of the i-th microgrid at time t, ξ∈{1,2,3}, when ξ=1, i∈N 1 ,Right now represents the transaction reward of the i-th microgrid in the first type of microgrid at time t, when ξ=2, i∈N 2 ,Right now represents the transaction reward of the i-th microgrid at time t in the second type of microgrid, when ξ=3, i∈N 3 ,Right now represents the transaction reward of the i-th microgrid in the third type of microgrid at time t.

[0106] Preferably, the sum of the internal reward and transaction reward of each microgrid is taken as the total reward, and the total reward function of each microgrid at each time t is the objective function of each microgrid. The calculation formula of the total reward function is as follows:

[0107] For the first type of microgrid, i∈N 1 , the calculation formula for the total reward of each microgrid is as follows:

[0108]

[0109] in, represents the total reward of the i-th microgrid in the first type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the first type of microgrid at time t;

[0110] For the second type of microgrid, i∈N 2 , the calculation formula for the total reward of each microgrid is as follows:

[0111]

[0112] in, represents the total reward of the i-th microgrid in the second type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the second type of microgrid at time t;

[0113] For the third type of microgrid, i∈N 3 , the calculation formula for the total reward of each microgrid is as follows:

[0114]

[0115] in, represents the total reward of the i-th microgrid in the third type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the third type of microgrid at time t;

[0116] The total objective function F of the interconnected microgrid group system is the accumulation of the total rewards of each microgrid during the entire optimization scheduling process, and is expressed as follows:

[0117]

[0118] And the optimization goal of the interconnected microgrid group system is to maximize the total objective function.

[0119] Preferably, for each batch of experience sampled, the batch experience at time t is expressed as The state information of each microgrid at the next moment is input into the target Actor network of the corresponding intelligent agent to obtain the strategy action a at the next moment. i,t+1 ;

[0120] Randomly select G modules from M modules, and the selected modules form a set in The size is Will gather Each element in is numbered by k, that is, The s of each microgrid i,t+1 and a i,t+1 As a whole, it is input into the target critic network corresponding to G modules. The target Q value is obtained from in represents the target critic network of the ith agent in the kth module, Represents the parameters of the target critic network of the i-th agent in the k-th module;

[0121] For each of the G modules, all the target Q values ​​are input into the hybrid network module. The total target Q value is obtained Among them, ψ k represents the parameters of the hybrid network module of the kth module, represents the hybrid network module of the kth module;

[0122] Select the smallest total target Q value from G total target Q values ​​as the final total target Q value And the final total target Q value is the final total target Q value of all agents. The specific formula is as follows:

[0123]

[0124] Among them, s t+1 represents the status information of all microgrids at time t+1, a t+1 represents the strategic actions of all microgrids at time t+1, represents the parameter set of all target Critic networks, and ψ represents the parameter set of all hybrid network modules;

[0125] The current state information s of each microgrid i,t and the current strategic action a i,t As a whole, it is input into the estimated critic network corresponding to the M modules. The estimated Q value is obtained k∈{1,...,M}, where represents the estimated Critic network of the i-th agent in the k-th module, Represents the parameters of the estimated Critic network of the i-th agent in the k-th module;

[0126] For each of the M modules, all the estimated Q values ​​are input into the hybrid network module. The total estimated Q value is k∈{1,...,M}, where φ k represents the parameters of the estimated critic network of all agents in the kth module, s t represents the status information of all microgrids at time t, a t represents the strategic actions of all microgrids at time t;

[0127] For each of the M modules, the loss function is calculated as follows:

[0128]

[0129] in,

[0130]

[0131] in,

[0132]

[0133] in, represents the loss function of the kth module, y tot Represents experience pool The target value calculated by the sample in represents the expected calculation, Represents experience pool The expected calculation is performed on the samples sampled in r t represents the sum of all total rewards of the interconnected microgrid group system at time t, γ represents the discount factor, and its value range is [0,1];

[0134] According to the loss function of each module, the estimated Critic network and hybrid network module of each module are updated using the gradient descent method, the target Critic network of each module is updated using the soft update method, and the estimated Actor network is updated using the centralized gradient estimator, and the following formula is satisfied:

[0135]

[0136] in,

[0137] a t * =(μ 1 (s 1，t θ 1 ), μ 2 (s 2，t θ 2 ), ..., μ n (s n，t θ i ), ..., μ n (s n，t θ n )) (54);

[0138] Among them, μ represents all estimated Actor networks, θ represents the parameter set of all estimated Actor networks, Represents the estimated gradient of the Actor network, μ i represents the estimated Actor network of the ith agent, θ i represents the parameters of the Actor network of the i-th agent, a t * represents the set of strategic actions of all microgrids, and each strategic action is generated by the estimated Actor network of the corresponding agent, represents the gradient of the parameter set θ, Indicates a t * The gradient of

[0139] Use soft update to update each target Actor network;

[0140] Until the strategic actions output by the target Actor network in each intelligent agent make the sum of all total rewards in the entire optimization scheduling process converge, the optimal strategy of each microgrid is obtained.

[0141] Compared with the prior art, the present invention has the following beneficial effects:

[0142] This method optimizes the scheduling model so that each microgrid can dynamically adjust its strategy actions under the constantly changing environmental information to achieve more efficient and coordinated energy management. The optimized scheduling model is composed of multiple intelligent agents and hybrid network modules. During the training process, the total target Q value and the total estimated Q value are used to calculate the loss function of each module, which effectively reduces the excessive deviation in the Q value estimation and avoids over-optimism of the strategy, making the scheduling strategy more robust and reliable in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0143] Figure 1 It is a flow chart of the interconnected microgrid group optimization scheduling method based on multi-agent deep learning of the present invention;

[0144] Figure 2 is a schematic diagram of each microgrid of the present invention;

[0145] Figure 3 A schematic diagram of electricity transactions between microgrids and with the main grid;

[0146] Figure 4 Schematic diagram of the training structure of the optimization scheduling network model. DETAILED DESCRIPTION

[0147] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0148] like Figure 1-Figure 4 As shown, a method for optimizing the scheduling of interconnected microgrids based on multi-agent deep learning is provided (this method belongs to the optimization scheduling of interconnected microgrids in a cooperative scenario), including:

[0149] Step 1, establish an optimal dispatching network model of an interconnected microgrid group system, the interconnected microgrid group system includes a main grid and n microgrids, the optimal dispatching network model includes n intelligent agents and M hybrid network modules (the hybrid network module adopts a hybrid network of the FACMAC algorithm), and the microgrid corresponds to the intelligent agent one by one, the intelligent agent includes an Actor network module and M Critic network modules, and a Critic network module and a hybrid network module are selected from each intelligent agent as a module, then there are M modules in total, and the Critic network module and the hybrid network module in each module are different (that is, the Critic network module in each module is different, and the hybrid network module in each module is different);

[0150] The n microgrids in the interconnected microgrid system are numbered as a total set N, and N = {1, 2, ..., i, ... n}.

[0151] Step 2: Obtain the current environmental information of each microgrid in the interconnected microgrid group system, and initialize the parameters of the optimization scheduling network model;

[0152] The equipment included in each microgrid includes: at least one of wind power generation equipment, photovoltaic power generation equipment, electric load equipment, water electrolysis hydrogen production equipment, hydrogen storage tank equipment, fuel cell equipment, energy storage battery equipment, diesel generator equipment, thermal load equipment, electric boiler equipment and cogeneration equipment (if the microgrid does not include the above equipment, the environmental information, state information and strategy action of the microgrid does not include the information of the equipment), and the environmental information of each microgrid is the environmental information of the equipment in each microgrid;

[0153] The environmental information of wind power generation equipment is defined as follows: The electric power output of the wind power generation equipment of the i-th microgrid at time t is defined as i∈N;

[0154] The environmental information of photovoltaic power generation equipment is defined as follows: The power output of the photovoltaic power generation equipment of the i-th microgrid at time t is i∈N;

[0155] The environmental information of the electric load device is defined as follows: The electric power consumed by the electric load device of the i-th microgrid at time t is defined as i∈N (in the power system, electrical load refers to the equipment or facilities connected to the power grid and consuming electrical energy);

[0156] The environmental information of the thermal load device is defined as follows: The power consumed by the thermal load device of the i-th microgrid at time t is defined as i∈N (in power systems, heat load refers to the total amount of heat energy required by a system or building in a specific period of time);

[0157] The environmental information of water electrolysis hydrogen production equipment meets the following relationship (water electrolysis hydrogen production is a method of using electricity to convert water (H 2 O) decomposes into hydrogen (H 2 ) and oxygen (O 2 ) technology, which is carried out through electrolyzers and can absorb excess electricity generated by renewable energy to achieve energy balance and regulation):

[0158]

[0159] in,

[0160]

[0161] in, represents the working cost of the water electrolysis hydrogen production equipment of the i-th microgrid at time t, represents the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t and time t-1, a EHP 、b EHP and c EHP These are the coefficients for calculating the working cost of water electrolysis hydrogen production equipment. represents the maximum electric power input by the water electrolysis hydrogen production equipment of the i-th microgrid, represents the ramp power of the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t (the ramp power is the difference between the electric power at the current moment and the electric power at the previous moment), represents the maximum and minimum values ​​of the ramp power input by the water electrolysis hydrogen production equipment of the i-th microgrid, η EHP It indicates the efficiency of water electrolysis hydrogen production equipment in converting input electrical energy into hydrogen energy. It represents the hydrogen power after the water electrolysis hydrogen production equipment of the i-th microgrid converts the input electrical energy into hydrogen energy at time t, i∈N;

[0162] The environmental information of hydrogen storage tank equipment satisfies the following relationship (a hydrogen storage tank is a container specially designed to store hydrogen, similar to an energy storage battery, and plays an important role in the storage and utilization of clean energy):

[0163]

[0164] in,

[0165]

[0166] in, represents the hydrogen energy stored in the hydrogen storage tank equipment of the i-th microgrid at time t+1 and time t, α HT , They represent the hydrogen storage attenuation coefficient, hydrogen storage energy efficiency and hydrogen release energy efficiency respectively. represents the hydrogen storage power of the hydrogen storage tank equipment of the i-th microgrid at time t, Indicates the upper limit of hydrogen storage power, The hydrogen release power of the hydrogen storage tank equipment of the i-th microgrid at time t is: Indicates the upper limit of hydrogen release power, They represent the upper and lower limits of the hydrogen storage tank’s capacity for storing hydrogen energy, respectively, Δt represents the time interval between adjacent moments, i∈N;

[0167] The environmental information of fuel cell equipment satisfies the following relationship (fuel cell is a technology that can efficiently and cleanly convert hydrogen into electrical energy):

[0168]

[0169] in,

[0170]

[0171] in, represents the operating cost of the fuel cell equipment of the i-th microgrid at time t, and represents the electric power output of the fuel cell device of the i-th microgrid at time t and time t-1, a FC , b FC and c FC These are coefficients for calculating the operating cost of fuel cell equipment. represents the maximum electric power output by the fuel cell device of the i-th microgrid, represents the ramp power of the electric power output by the fuel cell device of the i-th microgrid at time t, represents the maximum and minimum values ​​of the ramp power input by the fuel cell device of the i-th microgrid, η FC Indicates the efficiency of the fuel cell device in converting the input hydrogen energy into electrical energy. It represents the power of hydrogen when the fuel cell device of the i-th microgrid converts the input hydrogen energy into electrical energy at time t, i∈N;

[0172] The environmental information of the energy storage battery equipment satisfies the following relationship:

[0173]

[0174] in,

[0175]

[0176]

[0177] in, represents the power of the energy storage battery equipment of the i-th microgrid at time t+1 and time t, represents the charging power and discharging power of the energy storage battery equipment of the i-th microgrid at time t, Indicates the upper limit of charging power. Indicates the upper limit of discharge power, α, η ch and η dis They represent the energy storage attenuation coefficient, charging efficiency and discharging efficiency respectively. They represent the upper and lower limits of the amount of electricity stored in the energy storage battery, i∈N. Formula (18) indicates that the energy storage battery device can only perform one action, charging or discharging, at the same time;

[0178] The environmental information of the diesel generator equipment satisfies the following relationship (diesel generator is a very common controllable distributed dispatchable power source in microgrid):

[0179]

[0180] in,

[0181]

[0182] in, represents the operating cost of the diesel generator equipment of the i-th microgrid at time t, and represents the electric power output of the diesel generator equipment of the i-th microgrid at time t and time t-1, represents the maximum power output of the diesel generator equipment in the i-th microgrid, a DG , b DG and c DG These are coefficients for calculating the operating cost of diesel generator equipment. represents the ramp power of the diesel generator equipment of the i-th microgrid output at time t, represents the maximum and minimum values ​​of the ramp electric power output by the diesel generator equipment of the i-th microgrid, i∈N;

[0183] The environmental information of electric boiler equipment satisfies the following relationship (electric boiler is a device that converts electrical energy into thermal energy):

[0184]

[0185] in,

[0186]

[0187] in, represents the operating cost of the electric boiler equipment of the i-th microgrid at time t, represents the electric power input to the electric boiler equipment of the i-th microgrid at time t, represents the maximum electric power input by the electric boiler equipment of the i-th microgrid, a EB 、b EB and c EB Both are coefficients for calculating the working cost of electric boiler equipment, η EB Indicates the efficiency of electric boiler equipment in converting input electrical energy into thermal energy. It represents the thermal power after the electric boiler equipment of the i-th microgrid converts the input electrical energy into thermal energy at time t, i∈N;

[0188] The environmental information of cogeneration equipment satisfies the following relationship (cogeneration is a technology that produces electricity and heat energy at the same time):

[0189]

[0190] in,

[0191]

[0192] in, represents the operating cost of the cogeneration equipment of the i-th microgrid at time t, represents the electric power output of the cogeneration equipment of the i-th microgrid at time t, a CHP 、b CHP and c CHP These are coefficients for calculating the operating costs of cogeneration equipment. represents the maximum electric power output by the cogeneration equipment of the i-th microgrid, Indicates the efficiency of the cogeneration equipment in converting hydrogen energy into heat energy, It means that the cogeneration equipment of the i-th microgrid converts the hydrogen energy power The thermal power after conversion into thermal energy, represents the efficiency of the cogeneration equipment in converting hydrogen energy into electrical energy, i∈N.

[0193] Step 3: Obtain the current state information from the environmental information of each microgrid, and input it into the Actor network module of the intelligent body accordingly to obtain the current strategic action of each microgrid. Based on the strategic action and current state information, calculate the internal reward of each microgrid at the current moment.

[0194] Among them, the status information of each microgrid in the interconnected microgrid system includes: the power output of the wind power generation equipment of the i-th microgrid at time t The power output of the photovoltaic power generation equipment of the i-th microgrid at time t The power consumed by the electric load equipment of the i-th microgrid at time t The electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t The hydrogen energy stored in the hydrogen storage tank equipment of the i-th microgrid at time t The power output of the fuel cell device of the i-th microgrid at time t The amount of electricity stored in the energy storage battery device of the i-th microgrid at time t The power output of the diesel generator equipment of the i-th microgrid at time t The power consumed by the thermal load equipment of the i-th microgrid at time t The unit price of electricity sold by each microgrid to the main grid at time t and the unit price of electricity purchased by each microgrid from the main grid at time t At least one of them, i∈N;

[0195] The state information of each microgrid is input into the Actor network module of the corresponding intelligent agent to obtain the strategic action of each microgrid, and the strategic action of each microgrid includes: the electric power charged by the energy storage battery equipment of the i-th microgrid at time t The electric power discharged by the energy storage battery equipment of the i-th microgrid at time t The ramp power of the electric power output by the fuel cell device of the i-th microgrid at time t The ramp power of the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t The ramp power of the diesel generator equipment of the i-th microgrid output at time t The amount of hydrogen purchased by the i-th microgrid at time t (For the second type of microgrid, the hydrogen is purchased by the hydrogen storage tank equipment, and for the third type of microgrid, the hydrogen is purchased by the cogeneration equipment) and the electric power input of the electric boiler equipment of the i-th microgrid at time t At least one of them, i∈N;

[0196] The interconnected microgrid system includes three types of microgrids, namely, the first type of microgrid, the second type of microgrid and the third type of microgrid, and the number of the first type of microgrid is classified into the first set N 1 , the second type of microgrids are numbered as the second set N 2 , the third type of microgrid is numbered as the third set N 3 ,in:

[0197] The first type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electrical load equipment, water electrolysis hydrogen production equipment, hydrogen storage tank equipment, fuel cell equipment and energy storage battery equipment;

[0198] And the state information of the i-th microgrid at time t in the first type of microgrid is i∈N 1 , the strategic action of the i-th microgrid at time t is i∈N 1 ;

[0199] The second type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electrical load equipment, diesel generator equipment, hydrogen storage tank equipment, fuel cell equipment and energy storage battery equipment;

[0200] And the state information of the i-th microgrid at time t in the second type microgrid is i∈N 2 , the strategic action of the i-th microgrid at time t is i∈N 2 ;

[0201] The third type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, thermal load equipment, electric boiler equipment, cogeneration equipment and energy storage battery equipment;

[0202] And the state information of the i-th microgrid at time t in the third type of microgrid is i∈N 3 , the strategic action of the i-th microgrid at time t is i∈N 3 .

[0203] Among them, based on the strategy action and state information, the internal reward of each microgrid is calculated, including:

[0204] For the first type of microgrid, i∈N 1 , the calculation formula of the internal reward of each microgrid is as follows:

[0205]

[0206] in, represents the internal reward of the i-th microgrid at time t in the first type of microgrid;

[0207] For the second type of microgrid, i∈N 2 , the calculation formula of the internal reward of each microgrid is as follows:

[0208]

[0209] in, represents the internal reward of the i-th microgrid at time t in the second type of microgrid, represents the unit price of hydrogen purchased at time t;

[0210] For the third type of microgrid, i∈N3 , the calculation formula of the internal reward of each microgrid is as follows:

[0211]

[0212] in, Internal reward of the i-th microgrid at time t in the third type of microgrid.

[0213] Step 4: According to the energy supply and consumption of each microgrid in the interconnected microgrid group system, the amount of electricity that needs to be traded in each microgrid at the current moment is calculated through the electricity balance equation, including:

[0214] For the first type of microgrid, i∈N 1 , the amount of electricity that needs to be traded by the i-th microgrid at time t Satisfies the following relationship:

[0215]

[0216] In the first type of microgrid, during the process of electrolyzing water to produce hydrogen, electrical energy is converted into hydrogen energy and stored in a hydrogen storage tank. When there is a power shortage, a fuel cell device is used to convert the hydrogen energy in the hydrogen storage tank into electrical energy. Then formula (5) is converted into:

[0217]

[0218] For the second type of microgrid, i∈N 2 , the amount of electricity that needs to be traded by the i-th microgrid at time t Satisfies the following relationship:

[0219]

[0220] The second type of microgrid purchases hydrogen and stores it in a hydrogen storage tank. When there is a power shortage, a fuel cell device is used to convert the hydrogen energy in the hydrogen storage tank into electrical energy. Then formula (5) is converted into:

[0221]

[0222] For the third type of microgrid, i∈N 3 , the amount of electricity that needs to be traded by the i-th microgrid at time t Satisfies the following relationship:

[0223]

[0224] The third type of microgrid uses purchased hydrogen. The demand of heat load equipment is met through cogeneration equipment and electric boiler equipment to achieve heat network balance, and the heat network balance equation is expressed as:

[0225]

[0226] Step 5: Based on the balance of energy input and output between microgrids and between microgrids and the main grid at the current moment in the interconnected microgrid group system, solve the transaction strategy of each microgrid at the current moment, and calculate the transaction reward of each microgrid at the current moment;

[0227] Among them, according to the balance of energy input and output between each microgrid and between each microgrid and the main grid at the current moment in the interconnected microgrid group system, the trading strategy of each microgrid at the current moment is solved, and the relationship between each microgrid and the main grid is as follows:

[0228]

[0229] in,

[0230]

[0231] Among them, i,j∈N, represents the excess power output from the i-th microgrid to the j-th microgrid at time t, It means that the i-th microgrid receives the excess power transmitted by the j-th microgrid at time t. It means that the i-th microgrid sells excess electricity to the main grid at time t. It indicates that the i-th microgrid purchases electricity from the main grid at time t;

[0232] There is a loss in the transmission of electricity between microgrids, and the relationship satisfied is as follows:

[0233]

[0234] in,

[0235]

[0236] Among them, i,j∈N, represents the total power output of all microgrids, represents the total amount of electricity received by all microgrids, Indicates the amount of power lost during the transmission process, β ij represents the loss coefficient of the power output from the i-th microgrid to the j-th microgrid;

[0237] The overall benefits of each microgrid and the main grid meet the following requirements:

[0238]

[0239] Among them, i,j∈N, represents the total benefits of each microgrid and the main grid at time t, represents the unit price of the excess electricity output from the i-th microgrid to the j-th microgrid at time t, The unit price of the excess electricity that the i-th microgrid exports to the j-th microgrid at time t;

[0240] Combine formulas (38) to (43) to obtain the trading strategy of each microgrid And i,j∈N. Figure 3 As shown, each microgrid uses the amount of electricity to be traded as a known value to implement a trading strategy between microgrids and between each microgrid and the main grid.

[0241] The transaction rewards for each microgrid are as follows:

[0242]

[0243] in, represents the transaction reward of the i-th microgrid at time t, ξ∈{1,2,3}, when ξ=1, i∈N 1 ,Right now represents the transaction reward of the i-th microgrid in the first type of microgrid at time t, when ξ=2, i∈N 2 ,Right now represents the transaction reward of the i-th microgrid at time t in the second type of microgrid, when ξ=3, i∈N 3 ,Right now represents the transaction reward of the i-th microgrid in the third type of microgrid at time t.

[0244] Step 6: The sum of the internal rewards and transaction rewards of each microgrid at the current moment is taken as the total reward at the current moment, and the state information, strategy actions, total rewards of all microgrids at the current moment and the state information of all microgrids at the next moment are stored in the experience pool as the experience of the same batch (that is, the batch experience at time t is expressed as

[0245] Among them, the sum of the internal reward and transaction reward of each microgrid is taken as the total reward, and the total reward function of each microgrid at each time t is the objective function of each microgrid. The calculation formula of the total reward function is as follows:

[0246] For the first type of microgrid, i∈N 1 , the calculation formula for the total reward of each microgrid is as follows:

[0247]

[0248] in, represents the total reward of the i-th microgrid in the first type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the first type of microgrid at time t;

[0249] For the second type of microgrid, i∈N 2 , the calculation formula for the total reward of each microgrid is as follows:

[0250]

[0251] in, represents the total reward of the i-th microgrid in the second type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the second type of microgrid at time t;

[0252] For the third type of microgrid, i∈N 3 , the calculation formula for the total reward of each microgrid is as follows:

[0253]

[0254] in, represents the total reward of the i-th microgrid in the third type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the third type of microgrid at time t;

[0255] The total objective function F of the interconnected microgrid group system is the accumulation of the total rewards of each microgrid during the entire optimization scheduling process, and is expressed as follows:

[0256]

[0257] And the optimization goal of the interconnected microgrid group system is to maximize the total objective function.

[0258] Step 7: Figure 4 As shown in the figure (group 1 represents the first module, and so on, group M represents the Mth module, and Critic1 represents the Critic network (target Critic network or estimated Critic network) belonging to the first agent), the optimization scheduling network model is trained based on the experience pool:

[0259] Step 7.1, randomly sample a preset batch of experience from the experience pool, and for each batch of experience sampled, input the state information of each microgrid at the next moment into the target Actor network of the corresponding Actor network module to obtain the strategy action at the next moment;

[0260] Among them, for each batch of experience sampled, the batch experience at time t is expressed as The state information of each microgrid at the next moment is input into the target Actor network of the corresponding intelligent agent to obtain the strategy action a at the next moment. i,t+1 .

[0261] Step 7.2: For the target Critic network of the Critic network module, randomly select G modules, input the state information of each microgrid at the next moment and the corresponding strategy action at the next moment into the target Critic network corresponding to the G modules to obtain the target Q value, and then input all the target Qs in each module into the hybrid network module of the corresponding module to obtain the total target Q value, and then select the minimum total target Q value from the G total target Q values ​​as the final total target Q value, specifically:

[0262] Randomly select G modules from M modules, and the selected modules form a set in The size is Will gather Each element in is numbered by k, that is, The s of each microgrid i,t+1 and a i,t+1 As a whole, it is input into the target critic network corresponding to G modules. The target Q value is obtained from in represents the target critic network of the ith agent in the kth module, Represents the parameters of the target critic network of the i-th agent in the k-th module;

[0263] For each of the G modules, all the target Q values ​​are input into the hybrid network module. The total target Q value is obtained Among them, ψ k represents the parameters of the hybrid network module of the kth module, represents the hybrid network module of the kth module;

[0264] Select the smallest total target Q value from G total target Q values ​​as the final total target Q value And the final total target Q value is the final total target Q value of all agents. The specific formula is as follows:

[0265]

[0266] Among them, s t+1 represents the status information of all microgrids at time t+1, a t+1 represents the strategic actions of all microgrids at time t+1, represents the parameter set of all target Critic networks, and ψ represents the parameter set of all hybrid network modules.

[0267] Step 7.3: For the estimated critic network of the critic network module, the current state information of each microgrid and the current strategic action are input into the target critic network corresponding to the M modules to obtain the estimated Q value, and then all the estimated Q values ​​of each module are input into the hybrid network module of the corresponding module to obtain the total estimated Q value of each module, which is specifically:

[0268] The current state information s of each microgrid i,t and the current strategic action a i,t As a whole, it is input into the estimated critic network corresponding to the M modules. The estimated Q value is obtained k∈{1,...,M}, where represents the estimated Critic network of the i-th agent in the k-th module, Represents the parameters of the estimated Critic network of the i-th agent in the k-th module;

[0269] For each of the M modules, all the estimated Q values ​​are input into the hybrid network module. The total estimated Q value is k∈{1,...,M}, where φ k represents the parameters of the estimated critic network of all agents in the kth module, s t represents the status information of all microgrids at time t, a t Represents the strategic actions of all microgrids at time t.

[0270] Step 7.4: Use the final total target Q value and the total estimated Q value to calculate the loss function of each module, update the parameters of the Critic network module and the hybrid network module of each module through the loss function, and use the centralized gradient estimator to update the parameters of the Actor network module. Specifically:

[0271] For each of the M modules, the loss function is calculated as follows:

[0272]

[0273] in,

[0274]

[0275] in,

[0276]

[0277] in, represents the loss function of the kth module, y tot Represents experience pool The target value calculated by the sample in represents the expected calculation, Represents experience pool The expected calculation is performed on the samples sampled in r t represents the sum of all total rewards of the interconnected microgrid group system at time t, γ represents the discount factor, and its value range is [0,1];

[0278] According to the loss function of each module, the estimated critic network and hybrid network module of each module are updated using the gradient descent method. in is the learning rate when the hybrid network module is updated, It is the learning rate when estimating the critic network update. The specific content of the gradient descent method belongs to the prior art and will not be elaborated in detail in this invention. The target critic network of each module is updated using the soft update method. Where is the step size of the soft update, which is usually very small, such as 0.001. is the parameter of the target Critic network, soft update method, use the centralized gradient estimator to update each estimated Actor network, and satisfy the following formula:

[0279]

[0280] in,

[0281] a t * =(μ 1 (s 1,t θ 1 ),μ 2 (s 2,t θ 2 ),...,μ i (s i,t θ i ),...,μ n (s n,t θ n )) (54);

[0282] Among them, μ represents all estimated Actor networks, θ represents the parameter set of all estimated Actor networks, Represents the estimated gradient of the Actor network, μ i represents the estimated Actor network of the ith agent, θ i represents the parameters of the Actor network of the i-th agent, a t * represents the set of strategic actions of all microgrids, and each strategic action is generated by the estimated Actor network of the corresponding agent, represents the gradient of the parameter set θ, Indicates a t * The gradient of

[0283] Use soft update to update each target Actor network;

[0284] Go back to step 2 and repeat the operation until the policy actions output by the target Actor network of each intelligent agent make the sum of all total rewards in the entire optimization scheduling process converge to the optimal value (that is, the total objective function F converges to the optimal value, and when it converges, it maximizes the total objective function, where the convergence to the optimal value can be the convergence to the vicinity of the optimal value), then the optimal strategy of each microgrid is obtained, where the optimal strategy is the parameters of the Actor network module corresponding to each microgrid when the total objective function F converges and the policy actions generated under different state information.

[0285] This method optimizes the scheduling model so that each microgrid can dynamically adjust its strategy actions under the constantly changing environmental information to achieve more efficient and coordinated energy management. The optimized scheduling model is composed of multiple intelligent agents and hybrid network modules. During the training process, the total target Q value and the total estimated Q value are used to calculate the loss function of each module, which effectively reduces the excessive deviation in the Q value estimation and avoids over-optimism of the strategy, making the scheduling strategy more robust and reliable in practical applications.

[0286] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0287] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for optimizing the scheduling of interconnected microgrids based on multi-agent deep learning, characterized by: The interconnected microgrid group optimization scheduling method based on multi-agent deep learning includes: Establish an optimal dispatching network model for an interconnected microgrid group system, the interconnected microgrid group system includes a main grid and n microgrids, the optimal dispatching network model includes n intelligent agents and M hybrid network modules, and the microgrids correspond to the intelligent agents one by one, the intelligent agents include Actor network modules and M Critic network modules, and a Critic network module and a hybrid network module are selected from each intelligent agent as a module, then there are a total of M modules, and the Critic network module and the hybrid network module in each module are different; Obtain the current environmental information of each microgrid in the interconnected microgrid group system and initialize the parameters of the optimization scheduling network model; Obtain the current state information from the environmental information of each microgrid, and input it into the Actor network module of the intelligent body accordingly to obtain the current strategic action of each microgrid. Based on the strategic action and current state information, calculate the internal reward of each microgrid at the current moment. According to the energy supply and consumption of each microgrid in the interconnected microgrid group system, the amount of electricity that needs to be traded in each microgrid at the current moment is calculated through the electricity balance equation; According to the balance of energy input and output between microgrids and between microgrids and the main grid at the current moment in the interconnected microgrid group system, the trading strategy of each microgrid at the current moment is solved, and the trading reward of each microgrid at the current moment is calculated; The sum of the internal rewards and transaction rewards of each microgrid at the current moment is taken as the total reward at the current moment. The current state information, strategy actions, total rewards of all microgrids and the state information of all microgrids at the next moment are stored in the experience pool as the same batch of experience, and the optimization scheduling network model is trained based on the experience pool: Randomly sample a preset batch of experience from the experience pool. For each batch of experience sampled, input the state information of each microgrid at the next moment into the target Actor network of the corresponding Actor network module to obtain the strategy action at the next moment. For the target critic network of the critic network module, randomly select G modules, input the state information of each microgrid at the next moment and the corresponding strategy action at the next moment into the target critic network corresponding to the G modules to obtain the target Q value, then input all the target Q values ​​in each module into the hybrid network module of the corresponding module to obtain the total target Q value, and then select the minimum total target Q value from the G total target Q values ​​as the final total target Q value; For the estimated critic network of the critic network module, the current state information of each microgrid and the current strategic action are input into the target critic network corresponding to the M modules to obtain the estimated Q value, and then all the estimated Q values ​​of each module are input into the hybrid network module of the corresponding module to obtain the total estimated Q value of each module; The final total target Q value and the total estimated Q value are used to calculate the loss function of each module. The parameters of the Critic network module and the hybrid network module of each module are updated through the loss function, and the parameters of the Actor network module are updated using the centralized gradient estimator. Until the strategic actions output by the target Actor network of each intelligent agent make the sum of all total rewards in the entire optimization scheduling process converge to the optimal value, the optimal strategy of each microgrid is obtained.

2. The interconnected microgrid group optimization scheduling method based on multi-agent deep learning as claimed in claim 1, characterized in that: The n microgrids in the interconnected microgrid system are numbered as a total set N, and N = {1, 2, ..., i, ... n}, and the equipment included in each microgrid includes: at least one of wind power generation equipment, photovoltaic power generation equipment, electric load equipment, water electrolysis hydrogen production equipment, hydrogen storage tank equipment, fuel cell equipment, energy storage battery equipment, diesel generator equipment, thermal load equipment, electric boiler equipment and cogeneration equipment, and the environmental information of each microgrid is the environmental information of the equipment in each microgrid; The environmental information of the wind power generation equipment is defined as follows: The electric power output of the wind power generation equipment of the i-th microgrid at time t is defined as The environmental information of the photovoltaic power generation equipment is defined as follows: The power output of the photovoltaic power generation equipment of the i-th microgrid at time t is The environmental information of the electric load device is defined as follows: The electric power consumed by the electric load device of the i-th microgrid at time t is defined as The environmental information of the thermal load device is defined as follows: The power consumed by the thermal load device of the i-th microgrid at time t is defined as The environmental information of the water electrolysis hydrogen production equipment satisfies the following relationship: in, in, represents the working cost of the water electrolysis hydrogen production equipment of the i-th microgrid at time t, and represents the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t and time t-1, a EHP 、b EHP and c EHP These are the coefficients for calculating the working cost of water electrolysis hydrogen production equipment. represents the maximum electric power input by the water electrolysis hydrogen production equipment of the i-th microgrid, represents the ramp power of the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t (the ramp power is the difference between the electric power at the current moment and the electric power at the previous moment), and represents the maximum and minimum values ​​of the ramp power input by the water electrolysis hydrogen production equipment of the i-th microgrid, η EHP It indicates the efficiency of water electrolysis hydrogen production equipment in converting input electrical energy into hydrogen energy. It represents the hydrogen power after the water electrolysis hydrogen production equipment of the i-th microgrid converts the input electrical energy into hydrogen energy at time t, i∈N; The environmental information of the hydrogen storage tank equipment satisfies the following relationship: in, in, and represents the hydrogen energy stored in the hydrogen storage tank equipment of the i-th microgrid at time t+1 and time t, α HT , and They represent the hydrogen storage attenuation coefficient, hydrogen storage energy efficiency and hydrogen release energy efficiency respectively. represents the hydrogen storage power of the hydrogen storage tank equipment of the i-th microgrid at time t, Indicates the upper limit of hydrogen storage power, The hydrogen release power of the hydrogen storage tank equipment of the i-th microgrid at time t is: Indicates the upper limit of hydrogen release power, and They represent the upper and lower limits of the hydrogen storage tank’s hydrogen energy storage capacity, Δt represents the time interval between adjacent moments, i∈N; The environmental information of the fuel cell device satisfies the following relationship: in, in, represents the operating cost of the fuel cell equipment of the i-th microgrid at time t, and represents the electric power output of the fuel cell device of the i-th microgrid at time t and time t-1, a FC 、b FC and c FC These are coefficients for calculating the operating cost of fuel cell equipment. represents the maximum electric power output by the fuel cell device of the i-th microgrid, represents the ramp power of the electric power output by the fuel cell device of the i-th microgrid at time t, and represents the maximum and minimum values ​​of the ramp power input by the fuel cell device of the i-th microgrid, η FC Indicates the efficiency of the fuel cell device in converting the input hydrogen energy into electrical energy. It represents the power of hydrogen when the fuel cell device of the i-th microgrid converts the input hydrogen energy into electrical energy at time t, i∈N; The environmental information of the energy storage battery equipment satisfies the following relationship: in, in, and represents the power of the energy storage battery equipment of the i-th microgrid at time t+1 and time t, and represents the charging power and discharging power of the energy storage battery equipment of the i-th microgrid at time t, Indicates the upper limit of charging power. Indicates the upper limit of discharge power, α, η ch and η dis They represent the energy storage attenuation coefficient, charging efficiency and discharging efficiency respectively. and They represent the upper and lower limits of the amount of electricity stored in the energy storage battery, i∈N; The environmental information of the diesel generator equipment satisfies the following relationship: in, in, represents the operating cost of the diesel generator equipment of the i-th microgrid at time t, and represents the electric power output of the diesel generator equipment of the i-th microgrid at time t and time t-1, represents the maximum power output of the diesel generator equipment in the i-th microgrid, a DG 、b DG and c DG These are coefficients for calculating the operating cost of diesel generator equipment. represents the ramp power of the diesel generator equipment of the i-th microgrid output at time t, and represents the maximum and minimum values ​​of the ramp electric power output by the diesel generator equipment of the i-th microgrid, i∈N; The environmental information of the electric boiler equipment satisfies the following relationship: in, in, represents the operating cost of the electric boiler equipment of the i-th microgrid at time t, represents the electric power input to the electric boiler equipment of the i-th microgrid at time t, represents the maximum electric power input by the electric boiler equipment of the i-th microgrid, a EB 、b EB and c EB Both are coefficients for calculating the working cost of electric boiler equipment, η EB Indicates the efficiency of electric boiler equipment in converting input electrical energy into thermal energy. It represents the thermal power after the electric boiler equipment of the i-th microgrid converts the input electrical energy into thermal energy at time t, i∈N; The environmental information of the cogeneration equipment satisfies the following relationship: in, in, represents the operating cost of the cogeneration equipment of the i-th microgrid at time t, represents the electric power output of the cogeneration equipment of the i-th microgrid at time t, a CHP 、b CHP and c CHP These are coefficients for calculating the operating costs of cogeneration equipment. represents the maximum electric power output by the cogeneration equipment of the i-th microgrid, Indicates the efficiency of the cogeneration equipment in converting hydrogen energy into heat energy, It means that the cogeneration equipment of the i-th microgrid converts the hydrogen energy power The thermal power after conversion into thermal energy, represents the efficiency of the cogeneration equipment in converting hydrogen energy into electrical energy, i∈N; The state information of each microgrid in the interconnected microgrid system includes: the electric power output of the wind power generation equipment of the i-th microgrid at time t The power output of the photovoltaic power generation equipment of the i-th microgrid at time t The power consumed by the electric load equipment of the i-th microgrid at time t The power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t The hydrogen energy stored in the hydrogen storage tank equipment of the i-th microgrid at time t The power output of the fuel cell device of the i-th microgrid at time t The amount of electricity stored in the energy storage battery device of the i-th microgrid at time t The power output of the diesel generator equipment of the i-th microgrid at time t The power consumed by the thermal load equipment of the i-th microgrid at time t The unit price of electricity sold by each microgrid to the main grid at time t and the unit price of electricity purchased by each microgrid from the main grid at time t At least one of them, i∈N; The state information of each microgrid is input into the Actor network module of the corresponding intelligent agent to obtain the strategic action of each microgrid, and the strategic action of each microgrid includes: the electric power charged by the energy storage battery equipment of the i-th microgrid at time t The electric power discharged by the energy storage battery equipment of the i-th microgrid at time t The ramp power of the electric power output by the fuel cell device of the i-th microgrid at time t The ramp power of the electric power input to the water electrolysis hydrogen production equipment of the i-th microgrid at time t The ramp power of the diesel generator equipment of the i-th microgrid output at time t The amount of hydrogen purchased by the i-th microgrid at time t and the electric power input to the electric boiler equipment of the i-th microgrid at time t At least one of them, i∈N.

3. The interconnected microgrid group optimization scheduling method based on multi-agent deep learning as claimed in claim 2 is characterized by: The interconnected microgrid system includes three types of microgrids, namely, a first type of microgrid, a second type of microgrid, and a third type of microgrid, and the first type of microgrid is numbered as a first set N1, the second type of microgrid is numbered as a second set N2, and the third type of microgrid is numbered as a third set N3, wherein: The equipment included in the first type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, water electrolysis hydrogen production equipment, hydrogen storage tank equipment, fuel cell equipment and energy storage battery equipment; And the state information of the i-th microgrid at time t in the first type of microgrid is The strategic action of the i-th microgrid at time t is The equipment included in the second type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, diesel generator equipment, hydrogen storage tank equipment, fuel cell equipment and energy storage battery equipment; And the state information of the i-th microgrid at time t in the second type microgrid is The strategic action of the i-th microgrid at time t is The equipment included in the third type of microgrid includes: wind power generation equipment, photovoltaic power generation equipment, electric load equipment, thermal load equipment, electric boiler equipment, cogeneration equipment and energy storage battery equipment; And the state information of the i-th microgrid at time t in the third type of microgrid is The strategic action of the i-th microgrid at time t is 4. The method for optimizing the scheduling of interconnected microgrids based on multi-agent deep learning as claimed in claim 3, characterized in that: Based on the strategic actions and state information, the internal rewards of each microgrid are calculated, including: For the first type of microgrid, i∈N1, the calculation formula of the internal reward of each microgrid is as follows: in, represents the internal reward of the i-th microgrid at time t in the first type of microgrid; For the second type of microgrid, i∈N2, the calculation formula for the internal reward of each microgrid is as follows: in, represents the internal reward of the i-th microgrid at time t in the second type of microgrid, represents the unit price of hydrogen purchased at time t; For the third type of microgrid, i∈N3, the calculation formula of the internal reward of each microgrid is as follows: in, Internal reward of the i-th microgrid at time t in the third type of microgrid.

5. The interconnected microgrid group optimization scheduling method based on multi-agent deep learning as claimed in claim 4, characterized in that: According to the energy supply and consumption of each microgrid in the interconnected microgrid group system, the amount of electricity that needs to be traded in each microgrid at the current moment is calculated through the electricity balance equation, including: For the first type of microgrid, that is, i∈N1, the amount of electricity that needs to be traded by the i-th microgrid at time t is Satisfies the following relationship: In the first type of microgrid, during the process of electrolyzing water to produce hydrogen, electrical energy is converted into hydrogen energy and stored in a hydrogen storage tank. When there is a power shortage, a fuel cell device is used to convert the hydrogen energy in the hydrogen storage tank into electrical energy. Then formula (5) is converted into: For the second type of microgrid, that is, i∈N2, the amount of electricity that needs to be traded by the i-th microgrid at time t is Satisfies the following relationship: The second type of microgrid purchases hydrogen and stores it in a hydrogen storage tank. When there is a power shortage, a fuel cell device is used to convert the hydrogen energy in the hydrogen storage tank into electrical energy. Then formula (5) is converted into: For the third type of microgrid, that is, i∈N3, the amount of electricity that needs to be traded by the i-th microgrid at time t is Satisfies the following relationship: The third type of microgrid uses purchased hydrogen. The demand of heat load equipment is met through cogeneration equipment and electric boiler equipment to achieve heat network balance, and the heat network balance equation is expressed as:

6. The method for optimizing the scheduling of interconnected microgrids based on multi-agent deep learning as claimed in claim 5, characterized in that: According to the balance of energy input and output between microgrids and between microgrids and the main grid in the interconnected microgrid group system, the trading strategy of each microgrid is solved, and the relationship between each microgrid and the main grid is as follows: in, Among them, i,j∈N, represents the excess power output from the i-th microgrid to the j-th microgrid at time t, It means that the i-th microgrid receives the excess power transmitted by the j-th microgrid at time t. It means that the i-th microgrid sells excess electricity to the main grid at time t. It indicates that the i-th microgrid purchases electricity from the main grid at time t; There is a loss in the transmission of electricity between microgrids, and the relationship satisfied is as follows: in, Among them, i,j∈N, represents the total power output of all microgrids, represents the total amount of electricity received by all microgrids, Indicates the amount of power lost during the transmission process, β ij represents the loss coefficient of the power output from the i-th microgrid to the j-th microgrid; The overall benefits of each microgrid and the main grid meet the following requirements: Where i,j∈N, F t LP represents the total benefits of each microgrid and the main grid at time t, represents the unit price of the excess electricity output from the i-th microgrid to the j-th microgrid at time t, The unit price of the excess electricity that the i-th microgrid exports to the j-th microgrid at time t; Combine formulas (38) to (43) to obtain the trading strategy of each microgrid And i,j∈N.

7. The method for optimizing the scheduling of interconnected microgrids based on multi-agent deep learning according to claim 6, characterized in that: The transaction rewards for each microgrid are as follows: in, represents the transaction reward of the i-th microgrid at time t, ξ∈{1, 2, 3}, when ξ=1, i∈N1, that is, represents the transaction reward of the i-th microgrid at time t in the first type of microgrid, when ξ=2, i∈N2, that is, represents the transaction reward of the i-th microgrid at time t in the second type of microgrid, when ξ=3, i∈N3, that is, represents the transaction reward of the i-th microgrid in the third type of microgrid at time t.

8. The interconnected microgrid group optimization scheduling method based on multi-agent deep learning as claimed in claim 7, characterized in that: The sum of the internal rewards and transaction rewards of each microgrid is taken as the total reward, and the total reward function of each microgrid at each time t is the objective function of each microgrid. The calculation formula of the total reward function is as follows: For the first type of microgrid, i∈N1, the total reward of each microgrid is calculated as follows: in, represents the total reward of the i-th microgrid in the first type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the first type of microgrid at time t; For the second type of microgrid, i∈N2, the calculation formula for the total reward of each microgrid is as follows: in, represents the total reward of the i-th microgrid in the second type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the second type of microgrid at time t; For the third type of microgrid, i∈N3, the calculation formula for the total reward of each microgrid is as follows: in, represents the total reward of the i-th microgrid in the third type of microgrid at time t, that is, the value of the objective function of the i-th microgrid in the third type of microgrid at time t; The total objective function F of the interconnected microgrid group system is the accumulation of the total rewards of each microgrid during the entire optimization scheduling process, and is expressed as follows: And the optimization goal of the interconnected microgrid group system is to maximize the total objective function.

9. The interconnected microgrid group optimization scheduling method based on multi-agent deep learning as claimed in claim 8, characterized in that: For each batch of experience sampled, the batch experience at time t is expressed as The state information of each microgrid at the next moment is input into the target Actor network of the corresponding intelligent agent to obtain the strategy action a at the next moment. i,t+1 ; Randomly select G modules from M modules, and the selected modules form a set in The size is Will gather Each element in is numbered by k, that is, The s of each microgrid i,t+1 and a i,t+1 As a whole, it is input into the target critic network corresponding to G modules. The target Q value is obtained from in represents the target critic network of the ith agent in the kth module, Represents the parameters of the target critic network of the i-th agent in the k-th module; For each of the G modules, all the target Q values ​​are input into the hybrid network module. The total target Q value is obtained Among them, ψ k represents the parameters of the hybrid network module of the kth module, represents the hybrid network module of the kth module; Select the smallest total target Q value from G total target Q values ​​as the final total target Q value And the final total target Q value is the final total target Q value of all agents. The specific formula is as follows: Among them, s t+1 represents the status information of all microgrids at time t+1, a t+1 represents the strategic actions of all microgrids at time t+1, represents the parameter set of all target Critic networks, and ψ represents the parameter set of all hybrid network modules; The current state information s of each microgrid i,t and the current strategic action a i,t As a whole, it is input into the estimated critic network corresponding to the M modules. The estimated Q value is obtained in, represents the estimated Critic network of the i-th agent in the k-th module, Represents the parameters of the estimated Critic network of the i-th agent in the k-th module; For each of the M modules, all the estimated Q values ​​are input into the hybrid network module. The total estimated Q value is k∈{1,...,M}, where φ k represents the parameters of the estimated critic network of all agents in the kth module, s t represents the status information of all microgrids at time t, a t represents the strategic actions of all microgrids at time t; For each of the M modules, the loss function is calculated as follows: in, in, in, represents the loss function of the kth module, y tot Represents experience pool The target value calculated by the sample in represents the expected calculation, Represents experience pool The expected calculation is performed on the samples sampled in r t represents the sum of all total rewards of the interconnected microgrid group system at time t, γ represents the discount factor, and its value range is [0,1]; According to the loss function of each module, the estimated Critic network and hybrid network module of each module are updated using the gradient descent method, the target Critic network of each module is updated using the soft update method, and the estimated Actor network is updated using the centralized gradient estimator, and the following formula is satisfied: in, Among them, μ represents all estimated Actor networks, θ represents the parameter set of all estimated Actor networks, Represents the estimated gradient of the Actor network, μ i represents the estimated Actor network of the ith agent, θ i represents the parameters of the Actor network of the i-th agent, a t * represents the set of strategic actions of all microgrids, and each strategic action is generated by the estimated Actor network of the corresponding agent, represents the gradient of the parameter set θ, Indicates a t * The gradient of Use soft update to update each target Actor network; Until the strategic actions output by the target Actor network in each intelligent agent make the sum of all total rewards in the entire optimization scheduling process converge, the optimal strategy of each microgrid is obtained.

Images