Energy management method for hydrogen-electricity coupled multi-microgrid system based on improved DDPG

Abstract

The present invention provides an energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG, which comprises the following steps: obtaining historical data of the hydrogen-electricity coupled multi-microgrid system; establishing an energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG; establishing an objective function and constraint conditions of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG; initializing an operating environment of an intelligent body of the energy management system; obtaining a trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm; and obtaining an energy management strategy through the trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm. The present invention can make the actions of the microgrid have time correlation, effectively avoid the generation of unnecessary energy management actions, effectively reduce the operating cost of the hydrogen-electricity coupled multi-microgrid, and improve the operating efficiency of the hydrogen-electricity coupled multi-microgrid.

Description

Technical Field

[0001] The present invention relates to an energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG, and belongs to the technical field of microgrid system energy management. Background Art

[0002] In recent years, with the continuous expansion of installed capacity of renewable energy, issues such as curtailed solar and wind power have become increasingly prominent. The explosive growth of renewable energy, particularly photovoltaic power generation, and the increasing penetration of renewable energy in the power grid have significantly increased the complexity of grid operations and the pressure to accommodate renewable energy. Along with the massive growth in installed capacity of renewable energy generation, the proportion of electricity consumed by the tertiary industry and residential electricity consumption, such as electric vehicle charging, has continued to rise, further widening the peak-to-valley variation in system load, increasing the pressure on the grid to regulate peak and frequency, and highlighting transmission difficulties in some areas. On the other hand, the widespread application of flexible loads on the user side, such as electric vehicle charging stations, battery energy storage, and hydrogen electrolysis, has provided potential flexible resources for grid dispatch and operation. Hydrogen-electricity coupled microgrids, as a regional energy aggregation model for the intensive construction and orderly management of renewable energy generation and flexible loads, can achieve orderly management and control of renewable energy generation and flexible load systems, such as photovoltaic power generation, battery energy storage, and electric vehicle charging stations, through supporting communication technologies and coordinated control strategies. These systems, integrated into grid operations and market transactions as microgrids, are a key solution for the integrated utilization of multiple energy resources in new power systems.

[0003] Traditional reinforcement learning-based energy management algorithms, such as the DQN algorithm, require discretization of the action space, resulting in excessively large action space dimensions and suboptimal solutions. Therefore, DQN learning algorithms are difficult to apply to continuous-space decision-making problems such as energy storage charging and discharging, and hydrogen tank charging and discharging. Furthermore, the improved DDPG algorithm based on this approach is also being developed. Because hydrogen-electricity coupled microgrids can only receive limited information on hydrogen tank capacity, battery status, and charging (hydrogen) station loads, and lack prior "memory," energy management relies solely on the current state of the hydrogen-electricity coupled multi-microgrid. This results in overly complex planned energy management strategies and the potential for redundant and repeated energy allocation, severely impacting the operational efficiency and cost of the hydrogen-electricity coupled multi-microgrid.

[0004] The above problems are issues that should be considered and solved in the energy management process of hydrogen-electricity coupled multi-microgrid systems improved based on deep reinforcement learning. Summary of the Invention

[0005] The purpose of the present invention is to provide an energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG to solve the problems of repeated and redundant energy allocation, low operating efficiency and high cost of the hydrogen-electricity coupled multi-microgrid in the prior art.

[0006] The technical solution of the present invention is:

[0007] An energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG comprises the following steps:

[0008] S1. Obtain historical data of the hydrogen-electricity coupled multi-microgrid system including historical input states and historical actions;

[0009] S2. Establish an energy management model for hydrogen-electricity coupled multi-microgrid system based on improved DDPG. The energy management model for hydrogen-electricity coupled multi-microgrid system based on improved DDPG includes Actor neural network and Critic neural network. The Actor neural network includes two strategy networks with the same structure, namely current strategy network and target strategy network. The current strategy network is used to input state s t And output action a t , the target policy network is used to input the next state and output the next action; the critic neural network includes two Q networks with the same structure, namely the current Q network and the target Q network. The current Q network is used to input the state s t and action a t And output about state a t With action a t The Q value of the target Q network is used to input the next state s t+1 and the next action a t+1 And output about the next state s t+1 and the next action a t+1 Q value;

[0010] S3. Establish the objective function and constraints of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG;

[0011] S4, initializing the operating environment of the energy management system intelligent agent;

[0012] S5. Using the historical data of step S1, the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm is trained to obtain a trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm;

[0013] S6. Based on the current input state of the hydrogen-electricity coupled multi-microgrid system, the energy management strategy is obtained through the trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm.

[0014] Furthermore, in step S1, the historical input status includes the photovoltaic power generation power, charging power, hydrogen charging power, hydrogen amount in the hydrogen storage tank, and the charge state of the energy storage system in each hydrogen-electricity coupled microgrid in each time period; the historical actions include the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power, and electricity purchase price in each hydrogen-electricity coupled microgrid in each time period.

[0015] Furthermore, in step S2, each strategy network includes a first long short-term memory network, i.e., a first LSTM network, and two first fully connected layers. The first LSTM network processes the received input state of the hydrogen-electricity coupled microgrid, and then the two fully connected layers process it and output the action of the microgrid.

[0016] Furthermore, in step S2, each Q network includes a second LSTM network, a second fully connected layer and a third fully connected layer. When the Critic neural network receives the input state and action of the hydrogen-electricity coupled multi-microgrid system, the input state is processed by the second LSTM network, and the action is processed by the second fully connected layer. The output results of the second LSTM network and the second fully connected layer are processed by the third fully connected layer to output the Q value of the state and action.

[0017] Furthermore, in step S3, the objective function of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG is established. f :

[0018] (1)

[0019] Where: T is the total number of time periods corresponding to the time period; N is the number of microgrids in the hydrogen-electricity coupled multi-microgrid system; △t is the unit time; They are the electricity purchase cost, charging income, and hydrogen charging income of the hydrogen-electricity coupled multi-microgrid system in period t; are the purchased power and purchased price of the hydrogen-electricity coupled microgrid system during period t, respectively; are the charging power, charging price, hydrogen charging power, and hydrogen charging price of the i-th hydrogen-electricity coupled microgrid during period t;

[0020] Constraints include:

[0021] (1) Power balance constraints:

[0022] (2)

[0023] Where: is the photovoltaic power generation power of the i-th hydrogen-electricity coupled microgrid in period t; 、 、 is the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power, and station power load power in the i-th hydrogen-electricity coupled microgrid during period t;

[0024] (2) Operation constraints of photovoltaic power generation systems:

[0025] (3)

[0026] Where: 、 are the minimum and maximum photovoltaic output power of the i-th hydrogen-electricity coupled microgrid during period t;

[0027] (3) Operational constraints of the electrolysis hydrogen production system:

[0028] a. Operational constraints of electrolyzers:

[0029] (4)

[0030] Where: and are the lower and upper limits of the power consumed by the electrolyzer in the i-th hydrogen-electricity coupled microgrid during period t during normal operation;

[0031] b. Fuel cell operation constraints:

[0032] (5)

[0033] Where: is the operating power of the fuel cell in the i-th hydrogen-electric coupled microgrid during period t; and These are the lower and upper limits of the power consumed by a fuel cell during normal operation;

[0034] c. Hydrogen storage tank operation constraints:

[0035] (6)

[0036] Where: represents the amount of hydrogen in the hydrogen storage tank of the i-th hydrogen-electric coupled microgrid during period t; and Indicates the upper and lower limits of the hydrogen storage tank's storage capacity;

[0037] (4) Electrochemical energy storage operation constraints:

[0038] (7)

[0039] (8)

[0040] Where: and The upper and lower limits of the charging and discharging power of the energy storage system; The amount of hydrogen in the hydrogen storage tank of the i-th hydrogen-electric coupled microgrid during period t is the state of charge of the energy storage system; and The upper and lower limits of the state of charge of the energy storage system;

[0041] (5) Charging / hydrogen system operation constraints:

[0042] a. Charging / Hydrogen Load Constraints:

[0043] (9)

[0044] (10)

[0045] Where: is the initial charging load requirement; is the dispatchable charging load demand; It is the non-dispatchable charging load demand; is the initial hydrogen charging load requirement; It is the dispatchable hydrogen filling load demand; It is the non-dispatchable hydrogen filling load demand;

[0046] b. Charging / Hydrogen Pile Operation Constraints:

[0047] (11)

[0048] (12)

[0049] Where: is the rated power of the charging pile, is the rated power of the hydrogen charging pile.

[0050] Furthermore, in step S4, the operating environment of the energy management system intelligent body is initialized, specifically,

[0051] S41. Input the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG into the operating environment;

[0052] S42. Define the state signal space and the action signal space, and set the penalty function and the reward function.

[0053] Furthermore, step S42 is specifically:

[0054] The state signal space S of the intelligent agent in the hydrogen-electricity coupled multi-microgrid system is defined as:

[0055] (13)

[0056] Where: are the photovoltaic power generation power, charging power, hydrogen charging power, hydrogen volume in the hydrogen storage tank, and hydrogen volume in the hydrogen storage tank, respectively, of the i-th hydrogen-electric coupled microgrid during period t, and the state of charge of the energy storage system;

[0057] The action signal space A of the intelligent agent in the hydrogen-electricity coupled multi-microgrid system is defined as:

[0058] (14)

[0059] Where: are the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power of the i-th hydrogen-electricity coupled microgrid in period t; is the electricity purchase price of the hydrogen-electricity coupled multi-microgrid system during period t;

[0060] Define the reward function R of deep reinforcement learning as

[0061] (15)

[0062] Where: C Rewards for the hydrogen-electricity coupled multi-microgrid system:

[0063] (16)

[0064] Where: T is the total number of time periods corresponding to the time period; N is the number of microgrids in the hydrogen-electricity coupled multi-microgrid system; is the unit time; are the purchased power and purchased price of the hydrogen-electricity coupled microgrid system during period t, respectively; are the charging power, charging price, hydrogen charging power, and hydrogen charging price of the i-th hydrogen-electricity coupled microgrid during period t;

[0065] D is the penalty function:

[0066] (17)

[0067] Where: Penalties for unbalanced electricity in the power system; is the penalty for over-discharge or over-charge of the energy storage system; λ is the penalty coefficient; is the unbalanced amount of power system at moment i; is the amount of over-discharge or over-charge of the energy storage system at moment i.

[0068] Furthermore, in step S5, the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG is trained, specifically,

[0069] S51, initialize the Actor neural network, Critic neural network and experience pool, set parameters, and initialize the current number of rounds;

[0070] S52, the state of coupling hydrogen electricity to multi-microgrid system s t Input the current policy network to get a t , perform the action a t , and calculate the reward r t and the next state s t+1 ;Will( s t , a t , r t , s t+1 ) is stored in the experience pool, and the experience pool is used to train the Critic neural network and the Actor neural network;

[0071] S53, use the minimized loss function to update the Critic neural network, and the policy gradient to update the Actor neural network;

[0072] S54. Compare the current number of rounds with the set maximum number of rounds to see if they are consistent. If they are not consistent, increase the current number of rounds by one and go to step S52; otherwise, save the trained neural network parameters and end the training.

[0073] The beneficial effects of the present invention are: this energy management method of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG adopts the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG. The action of the hydrogen-electricity coupled microgrid is not only controlled by the current state of the microgrid, but also by the previous state of the microgrid. It can make the action of the microgrid have time correlation, effectively avoid unnecessary energy management actions, effectively reduce the operating cost of the hydrogen-electricity coupled multi-microgrid, and improve the operating efficiency of the hydrogen-electricity coupled multi-microgrid. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 1 is a flow chart of an energy management method for a hydrogen-electricity coupled multi-microgrid system based on improved DDPG according to an embodiment of the present invention;

[0075] Figure 2 is a schematic diagram of an energy management model of a hydrogen-electricity coupled multi-microgrid system according to an embodiment of the present invention;

[0076] Figure 3 is the deep reinforcement learning process of the DDPG algorithm in the embodiment;

[0077] Figure 4 Schematic diagrams illustrating the improved Actor neural network and Critic neural network in the DDPG algorithm in the embodiment, wherein (a) is a schematic diagram illustrating the policy network of the Actor neural network, and (b) is a schematic diagram illustrating the Critic neural network. DETAILED DESCRIPTION

[0078] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0079] The embodiment provides an energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG. Figure 1 , including the following steps,

[0080] S1. Obtain historical data of the hydrogen-electricity coupled multi-microgrid system, including historical input states and historical actions.

[0081] In step S1, the schematic diagram of the hydrogen-electricity coupled multi-microgrid system is as follows: Figure 2 The historical input states include the photovoltaic power generation power, charging power, hydrogen charging power, hydrogen volume in the hydrogen storage tank, and the state of charge of the energy storage system in each hydrogen-electricity coupled microgrid at each time period. The historical actions include the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power, and electricity purchase price in each hydrogen-electricity coupled microgrid at each time period.

[0082] S2. Establish an energy management model for hydrogen-electricity coupled multi-microgrid system based on improved DDPG. The energy management model for hydrogen-electricity coupled multi-microgrid system based on improved DDPG includes Actor neural network and Critic neural network, such as Figure 3 , the Actor neural network includes two policy networks with the same structure, namely the current policy network and the target policy network. The current policy network is used to input the state s t And output action a t , the target policy network is used to input the next state and output the next action; the critic neural network includes two Q networks with the same structure, namely the current Q network and the target Q network. The current Q network is used to input the state s t and action a t And output about state a t With action a t The Q value of the target Q network is used to input the next state s t+1 and the next action a t+1 And output about the next state s t+1 and the next action a t+1 Q value.

[0083] In step S2, the LSTM network is introduced into the Actor neural network and the Critic neural network, as shown in Figure 4 (a), each policy network of the Actor neural network includes a first long short-term memory network, namely a first LSTM network and two first fully connected layers. The first LSTM network processes the input state s of the hydrogen-electricity coupled microgrid, and then processes it through the two fully connected layers to output the action a of the microgrid. Figure 4 (b) Each Q network of the Critic neural network includes a second LSTM network, a second fully connected layer, and a third fully connected layer. When the Critic neural network receives the input state s and action a of the hydrogen-electricity coupled multi-microgrid system, the input state s is processed by the second LSTM network, and the action a is processed by the second fully connected layer. The output results of the second LSTM network and the second fully connected layer are processed by the third fully connected layer and then the Q value of the state s and action a is output.

[0084] DDPG's Actor and Critic neural networks both use an LSTM and fully connected layer architecture. In the Actor neural network, the first layer is an LSTM, replacing the first fully connected layer. The second layer is a fully connected layer with 400 nodes, and the third layer is a fully connected layer with 300 nodes. Reluctant Unit (ReLU) is used as the activation function for the fully connected layers, and a Batch Norm layer is added after each layer to ensure algorithm stability. In the Critic neural network, a second LSTM replaces the fully connected layer that processes states. States are input to the LSTM layer, and actions are input to a 400-node fully connected layer, which is then processed by a 300-node fully connected layer. Reluctant Unit (ReLU) is also used as the activation function for the fully connected layers.

[0085] S3. Establish the objective function and constraints of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG.

[0086] Energy management in a hydrogen-electricity-coupled multi-microgrid aims to coordinate the output of each unit in the system and minimize the energy cost of the hydrogen-electricity-coupled multi-microgrid while meeting technical constraints. The total energy cost of a hydrogen-electricity-coupled multi-microgrid is primarily composed of the cost of purchasing electricity from the grid (photovoltaic power generation, hydrogen fuel cells, hydrogen production equipment), as well as the operation and maintenance costs of distributed energy facilities, as shown below:

[0087] (1) In step S3, the objective function of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG is established. f :

[0088] (1)

[0089] Where: T is the total number of time periods corresponding to the time period; N is the number of microgrids in the hydrogen-electricity coupled multi-microgrid system; △t is the unit time; is the electricity purchase cost of the hydrogen-electricity coupled multi-microgrid system during period t; is the charging benefit of the hydrogen-electricity coupled multi-microgrid system during period t; is the hydrogen charging income of the hydrogen-electricity coupled multi-microgrid system during period t; is the purchased power and price of the hydrogen-electricity coupled microgrid system during period t; is the charging power and charging price of the i-th hydrogen-electricity coupled microgrid in period t; is the hydrogen charging power and hydrogen charging price of the i-th hydrogen-electricity coupled microgrid in period t;

[0090] (2) Power balance constraints:

[0091] (2)

[0092] Where: is the photovoltaic power generation power of the i-th hydrogen-electricity coupled microgrid in period t; is the hydrogen production power of the electrolyzer in the i-th hydrogen-electricity coupled microgrid during period t; is the photovoltaic power generation power and energy storage charging and discharging power of the i-th hydrogen-electricity coupled microgrid during period t (discharging is positive and charging is negative); is the station power load power of the i-th hydrogen-electricity coupled microgrid in period t.

[0093] (3) Operation constraints of photovoltaic power generation systems:

[0094] (3)

[0095] Where: 、 are the minimum and maximum photovoltaic output powers of the i-th hydrogen-electricity coupled microgrid in period t.

[0096] (4) Operational constraints of the electrolysis hydrogen production system

[0097] a. Operational constraints of electrolyzers:

[0098] (4)

[0099] Where: and are the lower and upper limits of the power consumed by the electrolyzer in the i-th hydrogen-electricity coupled microgrid during period t during normal operation;

[0100] b. Fuel cell operation constraints:

[0101] (5)

[0102] Where: is the operating power of the fuel cell in the i-th hydrogen-electric coupled microgrid during period t; and These are the lower and upper limits of the power consumed by a fuel cell during normal operation;

[0103] c. Hydrogen storage tank operation constraints:

[0104] (6)

[0105] Where: represents the amount of hydrogen in the hydrogen storage tank of the i-th hydrogen-electric coupled microgrid during period t; and Indicates the upper and lower limits of the hydrogen storage tank's storage capacity;

[0106] (5) Electrochemical energy storage operation constraints:

[0107] (7)

[0108] (8)

[0109] Where: and The upper and lower limits of the charging and discharging power of the energy storage system; The amount of hydrogen in the hydrogen storage tank of the i-th hydrogen-electric coupled microgrid during period t is the state of charge of the energy storage system; and The upper and lower limits of the state of charge of the energy storage system;

[0110] (6) Charging / hydrogen system operation constraints:

[0111] a. Charging / Hydrogen Load Constraints:

[0112] (9)

[0113] (10)

[0114] Where: is the initial charging load requirement; is the dispatchable charging load demand; It is the non-dispatchable charging load demand; is the initial hydrogen charging load requirement; It is the dispatchable hydrogen filling load demand; It is the non-dispatchable hydrogen filling load demand;

[0115] b. Charging / Hydrogen Pile Operation Constraints:

[0116] (11)

[0117] (12)

[0118] Where: is the rated power of the charging pile, that is, the charging power of the charging pile is within its rated power range; is the rated power of the hydrogen charging pile, that is, the hydrogen charging rate of the hydrogen charging pile is within its rated rate range.

[0119] S4. Initialize the operating environment of the energy management system intelligent agent. Specifically,

[0120] S41. Input the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG into the operating environment;

[0121] S42. Define the state signal space and action signal space, and set the penalty function and reward function. Specifically,

[0122] The state signal space S of the intelligent agent in the hydrogen-electricity coupled multi-microgrid system is defined as:

[0123] (13)

[0124] Where: are the photovoltaic power generation power, charging power, hydrogen charging power, hydrogen volume in the hydrogen storage tank, and hydrogen volume in the hydrogen storage tank, respectively, of the i-th hydrogen-electric coupled microgrid during period t, and the state of charge of the energy storage system;

[0125] The action signal space A of the intelligent agent in the hydrogen-electricity coupled multi-microgrid system is defined as:

[0126] (14)

[0127] Where: are the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power of the i-th hydrogen-electricity coupled microgrid in period t; is the electricity purchase price of the hydrogen-electricity coupled multi-microgrid system during period t;

[0128] Define the reward function R of deep reinforcement learning as

[0129] (15)

[0130] Where: C Rewards for the hydrogen-electricity coupled multi-microgrid system:

[0131] (16)

[0132] Where: T is the total number of time periods corresponding to the time period; N is the number of microgrids in the hydrogen-electricity coupled multi-microgrid system; is the unit time; are the purchased power and purchased price of the hydrogen-electricity coupled microgrid system during period t, respectively; are the charging power, charging price, hydrogen charging power, and hydrogen charging price of the i-th hydrogen-electricity coupled microgrid during period t;

[0133] D is the penalty function:

[0134] (17)

[0135] Where: Penalties for unbalanced electricity in the power system; is the penalty for over-discharge or over-charge of the energy storage system; λ is the penalty coefficient; is the unbalanced amount of power system at moment i; is the amount of over-discharge or over-charge of the energy storage system at moment i.

[0136] S5. Use the historical data of step S1 to train the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm to obtain a trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm.

[0137] S51. Initialize the Actor neural network, Critic neural network, and experience pool, set parameters, and initialize the current number of rounds.

[0138] In step S51, the parameters include the experience pool capacity D, the maximum number of training rounds , discount rate γ, learning rate α, current critic network weight parameter Q, target critic network weight 、Current Critic Network , Target Critic Network , target network parameter update interval C , greedy probability ε, training batch k, LSTM sequence length H .

[0139] S52, the state of coupling hydrogen electricity to multi-microgrid system s t Input the current policy network to get a t , perform the action a t , and calculate the reward r t and the next state s t+1 ;Will( s t , at , r t , s t+1 ) is stored in the experience replay pool, and the experience replay pool is used to train the Critic neural network and the Actor neural network.

[0140] In step S52, the experience pool is used to train the Critic neural network and the Actor neural network. Specifically, in each round, the noise is initialized to obtain an initial state s t , the current policy network maps the current state to the action according to the policy a t ; Execute action a t , get rewarded r t and environmental status s t+1 , and the data ( s t , a t , r t , s t+1 ) is stored in the experience replay pool D; when the data in the experience replay pool reaches the training threshold, a set number of multi-bit arrays ( s i , a i , r i , s i+1 ) is used to train the parameters of the Critic neural network and the Actor neural network.

[0141] S53, use the minimized loss function to update the Critic neural network, and the policy gradient to update the Actor neural network; specifically,

[0142] Actor neural network updates network parameters through policy gradient :

[0143]

[0144] Where: J is the expected return value; E is the expectation; β is the random strategy; ρ is the state transition probability distribution under the random strategy; Fitting function for Critic network evaluation; is the critic network parameter; is the fitting function of the state-action mapping relationship.

[0145] Critic neural network updates network parameters by minimizing the loss function L :

[0146]

[0147] Where: r t For rewards; E is the expectation; β is the random strategy; ρ is the state transition probability distribution under the random strategy; y t is the estimate of the Q function value.

[0148] The target network will fix the parameters in the network for a certain period of time, thereby eliminating model oscillations caused by the same parameters between the current network and the target network. The target network update mechanism usually adopts the soft update method. Soft update makes small adjustments to the target network parameters at each step, so that the target network parameters gradually approach the main network parameters. The formula for soft update is:

[0149]

[0150] Where: is the Critic target network parameter; is the Actor target network parameter; τ is the soft update coefficient, which is usually small, such as 0.001.

[0151] S54. Compare the current number of rounds with the set maximum number of rounds to see if they are consistent. If they are not consistent, increase the current number of rounds by one and go to step S52; otherwise, save the trained neural network parameters and end the training.

[0152] S6. Based on the current input state of the hydrogen-electricity coupled multi-microgrid system, the energy management strategy is obtained through the trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm.

[0153] This energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG adopts an energy management model for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG. The action of the hydrogen-electricity coupled microgrid is controlled not only by the current state of the microgrid, but also by the previous state of the microgrid. This method can make the action of the microgrid time-related, effectively avoid unnecessary energy management actions, effectively reduce the operating cost of the hydrogen-electricity coupled multi-microgrid, and improve the operating efficiency of the hydrogen-electricity coupled multi-microgrid.

[0154] This energy management method for hydrogen-electricity coupled multi-microgrid systems based on improved DDPG, in order to address the shortcomings of the traditional DDPG algorithm in energy management, optimizes the DDPG network structure by introducing LSTM to accelerate network training. From the perspective of coordinating the output of each unit in the hydrogen-electricity coupled multi-microgrid system while ensuring the safe and economical operation of the system, an energy management model for hydrogen-electricity coupled multi-microgrid systems based on improved DDPG is established with the goal of minimizing the energy cost of hydrogen-electricity coupled multi-microgrids, which can efficiently realize the energy management of hydrogen-electricity coupled multi-microgrids.

[0155] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. An energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG, characterized by: The following steps are included: S1. Obtain historical data of the hydrogen-electricity coupled multi-microgrid system including historical input states and historical actions; S2. Establish an energy management model for hydrogen-electricity coupled multi-microgrid system based on improved DDPG. The energy management model for hydrogen-electricity coupled multi-microgrid system based on improved DDPG includes Actor neural network and Critic neural network. The Actor neural network includes two strategy networks with the same structure, namely current strategy network and target strategy network. The current strategy network is used to input state s t And output action a t , the target policy network is used to input the next state and output the next action; the critic neural network includes two Q networks with the same structure, namely the current Q network and the target Q network. The current Q network is used to input the state s t and action a t And output about state a t With action a t The Q value of the target Q network is used to input the next state s t+1 and the next action a t+1 And output about the next state s t+1 and the next action a t+1 Q value; S3. Establish the objective function and constraints of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG; S4, initializing the operating environment of the energy management system intelligent agent; S5. Using the historical data of step S1, the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm is trained to obtain a trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm; S6. Based on the current input state of the hydrogen-electricity coupled multi-microgrid system, the energy management strategy is obtained through the trained energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG algorithm.

2. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG as claimed in claim 1, characterized in that: In step S1, the historical input status includes the photovoltaic power generation power, charging power, hydrogen charging power, hydrogen amount in the hydrogen storage tank, and the charge state of the energy storage system in each hydrogen-electricity coupled microgrid in each time period; the historical actions include the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power, and electricity purchase price in each hydrogen-electricity coupled microgrid in each time period.

3. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on improved DDPG according to claim 1, characterized in that: In step S2, each strategy network includes a first long short-term memory network, i.e., a first LSTM network, and two first fully connected layers. The first LSTM network processes the input state of the received hydrogen-electricity coupled microgrid, and then the two fully connected layers process it and output the action of the microgrid.

4. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG as claimed in claim 1, characterized in that: In step S2, each Q network includes a second LSTM network, a second fully connected layer, and a third fully connected layer. When the Critic neural network receives the input state and action of the hydrogen-electricity coupled multi-microgrid system, the input state is processed by the second LSTM network, and the action is processed by the second fully connected layer. The output results of the second LSTM network and the second fully connected layer are processed by the third fully connected layer to output the Q value of the state and action.

5. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG according to any one of claims 1 to 4, characterized in that: In step S3, the objective function of the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG is established. f : （1） Where: T is the total number of time periods corresponding to the time period; N is the number of microgrids in the hydrogen-electricity coupled multi-microgrid system; △t is the unit time; They are the electricity purchase cost, charging income, and hydrogen charging income of the hydrogen-electricity coupled multi-microgrid system in period t; are the purchased power and purchased price of the hydrogen-electricity coupled microgrid system during period t, respectively; are the charging power, charging price, hydrogen charging power, and hydrogen charging price of the i-th hydrogen-electricity coupled microgrid during period t; Constraints include: (1) Power balance constraints: （2） Where: is the photovoltaic power generation power of the i-th hydrogen-electricity coupled microgrid in period t; 、 、 is the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power, and station power load power in the i-th hydrogen-electricity coupled microgrid during period t; (2) Operation constraints of photovoltaic power generation systems: （3） Where: 、 are the minimum and maximum photovoltaic output power of the i-th hydrogen-electricity coupled microgrid during period t; (3) Operational constraints of the electrolysis hydrogen production system: a. Operational constraints of electrolyzers: （4） Where: and are the lower and upper limits of the power consumed by the electrolyzer in the i-th hydrogen-electricity coupled microgrid during period t during normal operation; b. Fuel cell operation constraints: （5） Where: is the operating power of the fuel cell in the i-th hydrogen-electric coupled microgrid during period t; and These are the lower and upper limits of the power consumed by a fuel cell during normal operation; c. Hydrogen storage tank operation constraints: （6） Where: represents the amount of hydrogen in the hydrogen storage tank of the i-th hydrogen-electric coupled microgrid during period t; and Indicates the upper and lower limits of the hydrogen storage tank's storage capacity; (4) Electrochemical energy storage operation constraints: （7） （8） Where: and The upper and lower limits of the charging and discharging power of the energy storage system; The amount of hydrogen in the hydrogen storage tank of the i-th hydrogen-electric coupled microgrid during period t is the state of charge of the energy storage system; and The upper and lower limits of the state of charge of the energy storage system; (5) Charging / hydrogen system operation constraints: a. Charging / Hydrogen Load Constraints: （9） （10） Where: is the initial charging load requirement; is the dispatchable charging load demand; It is the non-dispatchable charging load demand; is the initial hydrogen charging load requirement; It is the dispatchable hydrogen filling load demand; It is the non-dispatchable hydrogen filling load demand; b. Charging / Hydrogen Pile Operation Constraints: （11） （12） Where: is the rated power of the charging pile, is the rated power of the hydrogen charging pile.

6. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG according to any one of claims 1 to 4, characterized in that: In step S4, the operating environment of the energy management system agent is initialized, specifically, S41. Input the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG into the operating environment; S42. Define the state signal space and the action signal space, and set the penalty function and the reward function.

7. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on improved DDPG according to claim 6, characterized in that: Step S42 is specifically: The state signal space S of the intelligent agent in the hydrogen-electricity coupled multi-microgrid system is defined as: （13） Where: are the photovoltaic power generation power, charging power, hydrogen charging power, hydrogen volume in the hydrogen storage tank, and hydrogen volume in the hydrogen storage tank, respectively, of the i-th hydrogen-electric coupled microgrid during period t, and the state of charge of the energy storage system; The action signal space A of the intelligent agent in the hydrogen-electricity coupled multi-microgrid system is defined as: （14） Where: are the electrolyzer hydrogen production power, photovoltaic power generation power, energy storage charging and discharging power of the i-th hydrogen-electricity coupled microgrid in period t; is the electricity purchase price of the hydrogen-electricity coupled multi-microgrid system during period t; Define the reward function R of deep reinforcement learning as （15） Where: C Rewards for the hydrogen-electricity coupled multi-microgrid system: （16） Where: T is the total number of time periods corresponding to the time period; N is the number of microgrids in the hydrogen-electricity coupled multi-microgrid system; is the unit time; are the purchased power and purchased price of the hydrogen-electricity coupled microgrid system during period t, respectively; are the charging power, charging price, hydrogen charging power, and hydrogen charging price of the i-th hydrogen-electricity coupled microgrid during period t; D is the penalty function: （17） Where: Penalties for unbalanced electricity in the power system; is the penalty for over-discharge or over-charge of the energy storage system; λ is the penalty coefficient; is the unbalanced amount of power system at moment i; is the amount of over-discharge or over-charge of the energy storage system at moment i.

8. The energy management method for a hydrogen-electricity coupled multi-microgrid system based on an improved DDPG according to any one of claims 1 to 4, characterized in that: In step S5, the energy management model of the hydrogen-electricity coupled multi-microgrid system based on the improved DDPG is trained, specifically, S51, initialize the Actor neural network, Critic neural network and experience pool, set parameters, and initialize the current number of rounds; S52, the state of coupling hydrogen electricity to multi-microgrid system s t Input the current policy network to get a t , perform the action a t , and calculate the reward r t and the next state s t+1 ;Will( s t , a t , r t , s t+1 ) is stored in the experience pool, and the experience pool is used to train the Critic neural network and the Actor neural network; S53, use the minimized loss function to update the Critic neural network, and the policy gradient to update the Actor neural network; S54. Compare the current number of rounds with the set maximum number of rounds to see if they are consistent. If they are not consistent, increase the current number of rounds by one and go to step S52; otherwise, save the trained neural network parameters and end the training.

Images