Power allocation method for hydrogen energy storage units based on multi-agent deep reinforcement learning

Abstract

The present invention relates to the field of energy storage control technology, and in particular to a method for power allocation of hydrogen energy storage units based on multi-agent deep reinforcement learning, comprising the following steps: establishing a power output control strategy for hydrogen energy storage units based on a multi-agent deep deterministic policy gradient (MADDPG) algorithm, introducing charge-discharge conversion times and charge state modules, setting reward functions and constraints; establishing a hydrogen energy storage power allocation model; collecting historical day-ahead forecast data and actual power generation data of photovoltaic power stations, and training the hydrogen energy storage power allocation model; based on real-time photovoltaic output data, using the hydrogen energy storage power allocation model to obtain hydrogen energy storage power output instructions, and coordinate the output of each hydrogen energy storage unit. The present invention establishes a hydrogen energy storage adaptive power allocation model based on MADDPG, which is used for intelligent decision-making to compensate for power output control of photovoltaic power generation system prediction errors.

Description

Technical Field

[0001] The present invention relates to the field of energy storage control technology, and in particular to a hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning. Background Art

[0002] Due to the randomness and volatility of photovoltaic power generation and the limitations of photovoltaic output forecasting methods, there is a certain error between the photovoltaic power generation forecasted and reported to the power grid and the actual power generation data of the day. When there is a large deviation between the actual power generation and the forecasted power generation, it is necessary to purchase rotating reserve power generation from the superior power grid to meet the day-ahead reporting level. Considering that the electricity generated by photovoltaic power generation can be converted into hydrogen energy, and the hydrogen generated by photovoltaic power generation and water electrolysis is green and pollution-free, it has good application prospects. Therefore, the forecast error of photovoltaic power generation can be compensated by introducing a hydrogen energy storage system.

[0003] For large-scale photovoltaic power stations, the power of electricity generated is relatively large. If you want to compensate for its prediction error through hydrogen energy storage devices, the power and capacity requirements for hydrogen energy storage are also high. However, the rated power of electrolyzers and fuel cells is generally kW level. If you want to achieve a large power output, you generally need to configure multiple hydrogen energy storage units and coordinate the output between the units to meet actual needs. However, how to coordinate the output of each hydrogen energy storage unit to maximize economic benefits is a problem that needs to be solved at present. Traditional power allocation methods are mainly through linear programming, integer programming, nonlinear programming, dynamic programming, stochastic optimization and fuzzy logic control, etc. However, the above methods have problems such as low efficiency and lack of intelligence, and cannot better adapt to the uncertainty and dynamics of the power grid environment. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the existing technology and propose a power allocation method for hydrogen energy storage units based on multi-agent deep reinforcement learning to perform intra-day scheduling of hydrogen energy storage units to compensate for the day-ahead prediction error of the photovoltaic power generation system.

[0005] To achieve the above object, the present invention adopts the following specific technical solutions:

[0006] The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning provided by the present invention comprises the following steps:

[0007] S1. Establish a power output control strategy for hydrogen energy storage units based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, introduce the charge-discharge conversion times and state of charge modules, and set reward functions and constraints.

[0008] S2. Based on the power output control strategy of step S1, a hydrogen energy storage power allocation model is established;

[0009] S3, collecting historical day-ahead forecast data and actual power generation data of photovoltaic power plants, and training the hydrogen energy storage power allocation model of step S2;

[0010] S4. Based on real-time photovoltaic output data, the hydrogen energy storage power allocation model is used to obtain the hydrogen energy storage power output instruction and coordinate the output of each hydrogen energy storage unit.

[0011] Furthermore, the hydrogen energy storage unit includes a charging system and a discharging system. When the output of the fuel cell of the discharging system cannot meet the demand, the fuel cell of the charging system is used to supplement the response; when the output of the electrolyzer of the charging system cannot meet the demand, the electrolyzer of the discharging system is used to supplement the response; when the charging system or the discharging system reaches the state of charge (SOC) boundary, the functions of the two systems are exchanged;

[0012] In step S1, the charge-discharge conversion number and state of charge module considers the charge-discharge conversion number and SOC balance to improve the output power efficiency of the fuel cell and the electrolyzer. The expression is as follows:

[0013] ;

[0014] In the formula, The difference between the actual photovoltaic output and the predicted output; N is the number of hydrogen energy storage units that need to be operated, that is, the number of agents in the MADDPG algorithm; The optimal efficiency point power of the hydrogen energy storage unit is obtained by adding and subtracting Indicates that it is operating near the optimal efficiency point;

[0015] During the discharge process, the hydrogen energy storage unit with a higher SOC is discharged first; during the charging process, the hydrogen energy storage unit with a lower SOC is charged first;

[0016] ;

[0017] ;

[0018] In the formula, It is the reward part of SOC equilibrium in MADDPG algorithm. is the state of charge of hydrogen energy storage unit i at time t, It is the average SOC of all hydrogen energy storage units in all charging systems or discharging systems.

[0019] Furthermore, in step S1, the reward function includes spinning reserve costs and power generation income. The spinning reserve costs are generated by the deviation between the photovoltaic power generation system's day-ahead reported data and the actual power generation data. The power generation income is determined by the total amount of electricity generated by the photovoltaic system and the hydrogen energy storage system.

[0020] When the predicted value of PV power in the spinning reserve fee is greater than the actual value, the grid dispatching agency needs to purchase additional spinning reserve fees for the PV power station from the power plant that can provide spinning reserve, that is, to increase the positive spinning reserve for the PV power station to ensure that the sum of the spinning reserve output and the actual output reaches the predicted value of the day before; when the predicted value is less than the actual value, the grid dispatching agency needs to arrange additional negative spinning reserve for it;

[0021] The expression for spinning reserve cost is as follows:

[0022] ;

[0023] In the formula, Predict output value for photovoltaic power station, is the actual output value of the photovoltaic power station; , They are the compensation unit prices for positive and negative spinning reserve costs, in yuan / MWh; Starting from time t The cost of purchasing spinning reserve for the photovoltaic power station during the duration;

[0024] The total cost of spinning reserve in one day is:

[0025] ;

[0026] When the actual value of the combined output of photovoltaic and hydrogen energy storage is greater than the forecast value reported a day ago, the photovoltaic power station needs to pay the grid dispatching agency for the purchase of negative spinning reserve, the value of which is:

[0027] ;

[0028] ;

[0029] In the formula, is the energy storage output value at time t, is a Boolean constant. When the prediction error is still less than 0 after the hydrogen energy storage unit is compensated, the Boolean constant takes the value of 1;

[0030] When the actual value of the combined output of photovoltaic and hydrogen energy storage is less than the forecast value reported a day ago, the photovoltaic power station needs to pay the grid dispatching agency for the purchase of positive spinning reserve fees, the value of which is:

[0031] ;

[0032] ;

[0033] When the prediction error is still greater than 0 after the hydrogen energy storage unit is compensated, the Boolean constant takes 1;

[0034] In step S1, the expression of power generation revenue is as follows:

[0035] ;

[0036] In the formula, Time-of-use electricity price, is the actual output of the photovoltaic power generation system at time t, is the response power of the i-th hydrogen energy storage unit at time t;

[0037] In the power output control strategy of the hydrogen energy storage unit based on the MADDPG algorithm, the agent units are in a fully cooperative relationship, and each agent is set with the same reward function:

[0038] ;

[0039] In the formula, , , are parameters of uniform dimension.

[0040] Furthermore, in step S1, the constraints of the hydrogen energy storage unit are as follows:

[0041] ;

[0042] ;

[0043] ;

[0044] In the formula, The maximum charging and discharging power of hydrogen energy storage unit i; and are the charging power and discharging power of the i-th hydrogen energy storage unit at time t. At the same time, one of them must be 0, so they are restricted by a 0-1 variable, that is, and , are charging 0-1 variable and discharging 0-1 variable respectively;

[0045] The energy constraints of the hydrogen energy storage unit are as follows:

[0046] ;

[0047] ;

[0048] In the formula, and They are the upper and lower limits of energy storage SOC, is the installed capacity of the i-th hydrogen energy storage unit, is the remaining hydrogen energy storage capacity of the i-th hydrogen energy storage unit at time t;

[0049] Operation constraints:

[0050] In the actual response process, the hydrogen energy storage unit is in a fully cooperative state; if the photovoltaic output prediction error is greater than 0, the output of all hydrogen energy storage units must be greater than 0, and the total output cannot be over-adjusted, that is, the total power output of hydrogen energy storage is less than the prediction error. If the photovoltaic output prediction error is less than zero, the output of each hydrogen energy storage unit is negative, and the absolute value of the total output is less than the absolute value of the prediction error. Therefore, the constraints added to the intelligent agent during operation are as follows:

[0051] ;

[0052] The power output control strategy of the hydrogen energy storage unit based on the MADDPG algorithm also includes a state space and an action space. The state space refers to the environmental information perceived by the agent, and the action space is the decision variable of the hydrogen energy storage unit.

[0053] The state space of the i-th agent in period t can be expressed as:

[0054] ;

[0055] In the formula, is the photovoltaic output at time t, is the photovoltaic power output predicted at time t, is the charge state of the i-th hydrogen energy storage unit at time t;

[0056] The action space of each agent, i.e. the power output of the hydrogen energy storage unit at time t, can be expressed as:

[0057] ;

[0058] In the formula, is the power output of the i-th hydrogen energy storage unit at time t.

[0059] Furthermore, in step S2, a hydrogen energy storage power allocation model is established as follows:

[0060] In the MADDPG algorithm, each agent has its own policy network (Actor) and value network (Critic). The Actor network is used to determine actions, and the Critic network is used to evaluate the quality of the current policy. The policy parameters of each agent are:

[0061] ;

[0062] remember is the strategy set of all agents. For a random strategy, the policy gradient of the expected return of each agent i is:

[0063] ;

[0064] In the formula, are the policy network parameters of agent i, is the centralized action-value function, is the joint observation space, which represents the set of observation states of N agents. Each agent learns its own Function, so that each agent can define a different reward function to achieve cooperation or competition;

[0065] Expanding to deterministic strategies , based on the basic theory of the Deep Deterministic Policy Gradient (DDPG) algorithm, it is extended to multiple agents. For each agent i, its policy network update formula can be expressed as:

[0066] ;

[0067] In the formula, is the strategy function of agent i, which is based on the observation of the agent Produce Action , Represents the experience replay buffer. Represents sampling from the experience replay pool, which stores the current state, next state, current action, and current reward of each agent in each round of training, namely: , is the Critic network of agent i, which evaluates the and all agent actions Expected return under

[0068] The Actor network is updated by gradient descent, which mainly includes the following steps:

[0069] Calculate the policy gradient:

[0070] For each agent, the current state Output Actor network and generate corresponding actions ; Calculate Q value through Critic network ; Calculate the gradient of the action to the Q value ; Calculate the gradient of Actor network parameters .

[0071] Update Actor network parameters:

[0072] The parameters of the Actor network are updated using the Adam optimizer through gradient descent. The update of the Critic network uses the mean square error loss to minimize the difference between the target Q value and the Q value predicted by the value network (Critic network);

[0073] The gradient formula of the Critic network loss function is as follows:

[0074] ;

[0075] In the formula, is the value network parameter of agent i;

[0076] is the target Q value, which is calculated by the Bellman equation and is usually given by the reward And the value prediction of the next state:

[0077] ;

[0078] In the formula, is the discount factor for future rewards, represents the target policy network, represents the next state of the environment, Represents the actions of each agent;

[0079] By minimizing the Critic loss function, the parameters of the Critic network can be updated to make the predicted Q value closer to the target Q value. The network parameter update adopts the gradient descent method, and the specific steps include:

[0080] Calculate loss: Calculate the mean square error loss between the target Q value and the current Q value based on batch samples; Back propagation: Calculate the gradient of the loss relative to the Critic network parameters through the back propagation algorithm; Update parameters: Use the Adam optimizer to perform gradient update of the Critic network parameters.

[0081] Furthermore, in step S3, the hydrogen energy storage power allocation model is trained as follows:

[0082] S31. Construct an Actor-Critic deep reinforcement learning model. Create an experience replay pool by using the Actor network and Critic network of each unit in the hydrogen energy storage unit charging system and the Actor network and Critic network of each unit in the hydrogen energy storage unit discharging system. Define parameters such as the experience replay pool size, the number of rounds at the start of training, the maximum number of training rounds, and the batch size. During model training, continuously adjust the parameters according to the verification set results to select MADDPG parameters suitable for the hydrogen energy storage power distribution environment.

[0083] S32. Based on the day-ahead forecast data of the photovoltaic power generation system and actual power generation data during the day The prediction error at time t is obtained, and the environmental parameters at time t are calculated according to the response state of each hydrogen energy storage unit. ; The total environmental parameters include time t, prediction error at time t, and the charge state of each hydrogen energy storage unit at time t;

[0084] S33, the agent based on the current environment state Select Action , in the action selection network (Actor network), by Add noise to explore. Gaussian noise is used to encourage the agent to explore more action spaces. As the number of training rounds increases, the noise is reduced to reduce the exploration of actions to achieve stability in model training. According to the current state of the environment and action selection , constrain the actions through the hydrogen energy storage operating environment, including the power and energy boundary constraints of the hydrogen energy storage unit during operation, the constraints between the hydrogen energy storage output and the actual demand, etc., and return the actions of multiple intelligent agents after the constraints , Environmental status And the corresponding reward r; the agent updates its internal state based on the new information returned (new environment state, reward and constrained action), which is stored in the experience replay pool buffer for subsequent training and optimization; in the subsequent training process, small batches of data are sampled from the experience replay pool buffer to update the policy network (Actor network) and value network (Critic network) to optimize the decision-making strategy of the agent;

[0085] S34. The hydrogen energy storage power allocation MADDPG model is based on a fully cooperative multi-agent environment. It calculates the environmental rewards of multiple agents and the total reward to find the optimal total reward. It forms a tuple based on the environmental state at time t, the action of the agent, the total reward of the agent, and the state of the environment at time t+1. Stored in the experience replay pool;

[0086] S35. In the MADDPG algorithm, the policy function in the Actor-Critic network is updated through the loss function. Specifically, the policy loss (Actor Loss) is calculated based on the action value function (Q-value) of each agent, while the value loss (Critic Loss) is calculated based on the Bellman equation. The calculation of Actor Loss involves maximizing Q-value to improve the performance of the policy; while the calculation of Critic Loss optimizes the value function by minimizing the error between the predicted value and the target value, thereby ensuring the accuracy of the policy.

[0087] S36, before reaching the maximum number of training rounds, train the Actor and Critic networks through the two loss functions of Actor Loss and Critic Loss in S35, update the model parameters, so that the agent can learn effective strategies, and save the trained Actor-Critic network model after the training is completed for model testing;

[0088] S37, model testing, import the trained Actor-Critic network model; during the test process, ensure that the gradient update mechanism in the training phase is set to zero to prevent the gradient from affecting the network parameters during the test; perform a loop test, evaluate each test sample, and record the corresponding performance indicators to ensure the stability of the test process and the accuracy of the results, and verify the generalization ability of the model and the actual application effect.

[0089] Furthermore, in step S4, coordinating the output of each hydrogen energy storage unit includes the following steps:

[0090] S41. Initialize the multi-agent environment, including the state space, action space and reward function;

[0091] S42, create multiple agents and load the pre-trained Actor-Critic model;

[0092] S43, in the loop process, set the gradient update mechanism to zero, perform actions in the environment and calculate the corresponding rewards and the next moment state, and repeat the process until the termination condition is met;

[0093] S44. Visualize the results, evaluate the performance of the intelligent agent, observe the difference between the actual output of the hydrogen energy storage-photovoltaic system and the predicted data, and record the actual output of the hydrogen energy storage and the charge state of each unit energy storage.

[0094] The present invention can achieve the following technical effects:

[0095] The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning provided by the present invention introduces machine learning algorithms into the field of power grid optimization scheduling. The machine learning algorithm can learn strategies based on historical data in an offline state, and make decisions quickly based on real-time observation data after training. Deep reinforcement learning combines the advantages of reinforcement learning and deep learning, and is used for the problem of continuous spatial action of agents. The neural network combined with deep learning can solve the dimensionality curse problem faced by reinforcement learning; multi-agent deep reinforcement learning is used to solve the problem of continuous action decision-making of multiple agents, and coordinate the output of each hydrogen energy storage unit to compensate for photovoltaic prediction errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 Schematic diagram of a hydrogen energy storage unit power allocation method provided according to an embodiment of the present invention.

[0097] Figure 2 It is a block diagram of peak, flat and valley time-of-use electricity prices provided according to an embodiment of the present invention.

[0098] Figure 3 2 is a schematic diagram of MADDPG network training provided according to an embodiment of the present invention.

[0099] Figure 4 It is a pseudo code diagram of MADDPG algorithm training provided according to an embodiment of the present invention.

[0100] Figure 5 4 is a block diagram of hyperparameter settings of the MADDPG algorithm provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0101] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0102] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0103] The embodiment of the present invention provides a hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning, which is used to compensate for the day-ahead prediction error of the photovoltaic power generation system. Figure 1 shown.

[0104] S1. Establish a power output control strategy for hydrogen energy storage units based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, introduce the charge and discharge conversion times and state of charge modules, and set reward functions and constraints.

[0105] 1. Charge and discharge conversion times and charge state module

[0106] Considering the number of charge and discharge conversions and SOC balance, to prevent the hydrogen energy storage unit from frequent charge and discharge, that is, the electrolyzer and fuel cell are frequently started and stopped, the hydrogen energy storage unit is divided into a charging system and a discharging system, and the charging and discharging systems serve as backup for each other. When the output of the fuel cell of the discharge system cannot meet the demand, the fuel cell of the charging system is used to supplement the response; when the output of the electrolyzer of the charging system cannot meet the demand, the electrolyzer of the discharge system is used to supplement the response. When one of the charging system or the discharging system reaches the state of charge (SOC) boundary as a whole, the charging and discharging system functions are exchanged, that is, the original discharging system is converted to the charging system, and the original charging system is converted to the discharging system.

[0107] According to the operating characteristics of the electrolyzer and fuel cell, their efficiency curves increase first and then decrease with their operating power, that is, when the power is too high or too low, their efficiency is low, which accelerates the aging of the fuel cell and has a certain impact on the operating life of the electrolyzer. In order to make the output power of the fuel cell and electrolyzer near the optimal efficiency point, when the power output instruction is obtained, the power of the hydrogen energy storage system is allocated in advance, and the number of hydrogen energy storage units N that need to be started is determined according to the following principles.

[0108] (1);

[0109] In the formula, is the difference between the actual photovoltaic output and the predicted output; N is the number of hydrogen energy storage units that need to be operated, which is also the number of intelligent agents in the MADDPG algorithm; The optimal efficiency point power of the hydrogen energy storage unit is obtained by adding and subtracting It means that hydrogen energy storage operates near the optimal efficiency point.

[0110] During the discharge process, the hydrogen energy storage unit with a higher SOC is discharged first; during the charging process, the hydrogen energy storage unit with a lower SOC is charged first. SOC balance is reflected in the rewards of the MADDPG algorithm as follows:

[0111] (2);

[0112] In the formula, is the reward part of the SOC equilibrium strategy, is the charge state of hydrogen energy storage unit i at time t, is the average SOC of all hydrogen energy storage units in all charging systems or discharging systems, which can be calculated by formula (3).

[0113] (3);

[0114] 2. Reward Function

[0115] 1) Spinning reserve costs

[0116] Taking into account the deviation between the photovoltaic day-ahead reported data and the actual power generation data, the hydrogen energy storage system is used to compensate for the prediction error so that the photovoltaic power generation system can be connected to the grid according to the pre-reported grid data. The photovoltaic system and the hydrogen energy storage system are regarded as an independent economy, and the spinning reserve costs caused by the prediction deviation of the photovoltaic power generation system are considered.

[0117] Due to the existence of photovoltaic output prediction errors, the output value of the photovoltaic power generation system during actual operation will not be equal to the predicted value the day before. The traditional method is to report the predicted value directly to the dispatch center. When the photovoltaic power prediction value is greater than the actual value, the grid dispatching agency needs to purchase additional spinning reserve fees for the photovoltaic power generation system from power plants that can provide spinning reserve. That is, the dispatching agency adds positive spinning reserve to the photovoltaic power generation system to ensure that the sum of the spinning reserve output and the actual output reaches the predicted value the day before, so that the power supply and demand sides of the power system can be balanced, and the purchase fee needs to be paid by the photovoltaic power station; on the contrary, when the predicted value is less than the actual value, the grid needs to arrange additional negative spinning reserve for it, and the fee is also paid by the photovoltaic power station.

[0118] The spinning reserve charges are:

[0119] (4);

[0120] In the formula, Predict output value for photovoltaic power station, is the actual output value of the photovoltaic power station, , are the compensation unit prices for positive and negative spinning reserve costs, in RMB / MWh. Starting from time t The cost of purchasing spinning reserve for the photovoltaic power station within the duration.

[0121] Therefore, the total amount of spinning reserve fees charged by the PV dispatching agency to the PV power station in one day is:

[0122] (5);

[0123] When the combined output of photovoltaic and hydrogen energy storage is greater than the value reported on the previous day, the photovoltaic power station needs to pay the power grid dispatching center for the purchase of negative spinning reserve, the value of which is:

[0124] (6);

[0125] (7);

[0126] In the formula, is the energy storage output value at time t, It is a Boolean constant. When the prediction error is still less than 0 after the hydrogen energy storage system is compensated, the Boolean constant takes the value of 1.

[0127] When the combined output of photovoltaic and hydrogen energy storage is less than the reported value on the previous day, the total amount of fees that the photovoltaic power station needs to pay to the grid dispatching center for purchasing positive spinning reserve is:

[0128] (8);

[0129] (9);

[0130] The Boolean constant takes the value of 1 when the prediction error is still greater than 0 after compensation of the hydrogen energy storage system.

[0131] 2) Power generation income

[0132] Consider the peak and valley time-of-use electricity prices, that is, the power grid adopts different on-grid electricity prices in different electricity consumption periods within a day. The time-of-use electricity price mainly divides the 24 hours of a day into three periods: peak, flat and valley, and corresponds to three electricity prices respectively. The on-grid electricity price in peak periods adopts a higher peak electricity price, and the on-grid electricity price in valley periods adopts a lower valley electricity price.

[0133] The implementation of peak-valley time-of-use electricity prices is beneficial to the load rate and operational stability of the power system. It uses market levers to achieve the goal of peak shaving and valley filling. It is an economic means adopted in accordance with the market economy. The implementation of time-of-use electricity price policies can help improve the load curve and reduce the power supply pressure of the power grid.

[0134] Combined with the power generation and electricity consumption status of the power grid and the sales side, the 24 hours of a day are divided into three major periods: peak, flat and valley. The peak, flat and valley time-of-use electricity prices are as follows: Figure 2 shown.

[0135] Power generation income under time-of-use electricity price:

[0136] (10);

[0137] In the formula, Time-of-use electricity price, is the actual output of the photovoltaic power generation system at time t, is the response power of the i-th hydrogen energy storage unit at time t.

[0138] Considering that the agent units are in a fully cooperative relationship, the same reward function is set for each agent:

[0139] (11);

[0140] In the formula, , , are parameters of uniform dimension.

[0141] 3. Constraints

[0142] Hydrogen energy storage unit constraints:

[0143] (12);

[0144] (13);

[0145] (14);

[0146] In the formula, The maximum charging and discharging power of hydrogen energy storage unit i; and The charging power and discharging power of the i-th hydrogen energy storage unit at time t are respectively, at the same time, one of them must be 0, so it is restricted by a 0-1 variable, that is, and , are charging 0-1 variable and discharging 0-1 variable respectively.

[0147] Energy constraints of hydrogen energy storage units:

[0148] (15);

[0149] (16);

[0150] In the formula, and They are the upper and lower limits of energy storage SOC, is the installed capacity of the i-th hydrogen energy storage unit, is the remaining hydrogen energy storage capacity of the i-th hydrogen energy storage unit at time t.

[0151] Operation constraints:

[0152] In the actual response process, the hydrogen energy storage unit is in a fully cooperative state. If the photovoltaic output prediction error is greater than 0, the output of all hydrogen energy storage units must be greater than 0, and the total output cannot be overregulated, that is, the total power output of hydrogen energy storage is less than the prediction error. If the photovoltaic output prediction error is less than zero, the output of each hydrogen energy storage unit is negative, and the absolute value of the total output is less than the absolute value of the prediction error. Therefore, the constraints added to the intelligent agent during operation are as follows:

[0153] (17);

[0154] 4. State Space

[0155] State refers to the environmental information perceived by the agent. In the hydrogen energy storage scheduling application for compensating photovoltaic power generation systems, the environmental information observed by the agent includes photovoltaic power output, the predicted value at the corresponding time, the SOC state of the hydrogen energy storage unit, and the scheduling period t. Therefore, the state of the i-th agent in period t can be expressed as:

[0156] (18);

[0157] In the formula, is the photovoltaic output at time t, is the photovoltaic power output predicted at time t, is the charge state of the i-th hydrogen energy storage unit at time t.

[0158] 5. Action Space

[0159] Action is the decision variable of the hydrogen energy storage system, that is, the action of each agent is the power output of the hydrogen energy storage unit at time t, which can be expressed as:

[0160] (19);

[0161] In the formula, is the power output of the i-th hydrogen energy storage unit at time t.

[0162] S2. Based on the power output control strategy of step S1, a hydrogen energy storage power allocation model is established.

[0163] It includes multi-agent environment, Actor-Critic network, experience replay pool, target network, noise process, training and update mechanism, etc. The network is built based on the PyTorch framework, and GPU and CPU resources are combined for training.

[0164] The Multi-Agent Deep Deterministic Policy Gradient algorithm is a deep reinforcement learning algorithm used in a multi-agent environment. It is an extension of the Deep Deterministic Policy Gradient (DDPG) algorithm. In MADDPG, each agent has its own policy network (Actor) and value network (Critic), where the policy network is used to determine actions and the value network is used to evaluate the quality of the current policy.

[0165] Consider a game involving N agents, where the strategy parameters of each agent are:

[0166] (20);

[0167] remember is the strategy set of all agents. For a random strategy, the policy gradient of the expected return of each agent i is:

[0168] (twenty one);

[0169] In the formula, are the policy network parameters of agent i, is the centralized action-value function, is the joint observation space, which represents the set of observation states of N agents. Each agent learns its own function, so each agent can define a different reward function to achieve cooperation or competition.

[0170] Expanding to deterministic strategies , based on the basic theory of DDPG and extended to multiple agents, for each agent i, its Actor network update formula is based on the policy gradient method and can be expressed as:

[0171] (twenty two);

[0172] In the formula, is the strategy function of agent i, which is based on the observation of the agent Produce Action , Represents the experience replay buffer. Represents sampling from the experience replay pool, which stores the current state, next state, current action, and current reward of each agent in each round of training, namely: , is the Critic network of agent i, which evaluates the and all agent actions The expected return.

[0173] Specifically, the Actor network is updated by the gradient descent method, which mainly includes the following steps:

[0174] 1) Calculate the policy gradient:

[0175] For each agent, the current state Output Actor network and generate corresponding actions ; Calculate Q value through Critic network ; Calculate the gradient of the action to the Q value ; Calculate the gradient of Actor network parameters .

[0176] 2) Update Actor network parameters:

[0177] The parameters of the Actor network are updated using the Adam optimizer via gradient descent.

[0178] The update of the Critic network uses the mean square error loss to minimize the difference between the target Q value and the Q value predicted by the Critic network. The gradient formula of the Critic network loss function is as follows:

[0179] (twenty three);

[0180] In the formula, are the critic network parameters of agent i.

[0181] is the target Q value, which is calculated by the Bellman equation and is usually calculated by the reward And the value prediction of the next state:

[0182] (twenty four);

[0183] In the formula, is the discount factor for future rewards, Represents the target policy network. represents the next state of the environment, Indicates the actions of each agent in this state.

[0184] By minimizing the Critic loss function, the parameters of the Critic network can be updated to make the predicted Q value closer to the target Q value. The network parameter update adopts the gradient descent method, and the specific steps include:

[0185] 1) Calculate the loss: Calculate the mean square error loss between the target Q value and the current Q value based on the batch samples;

[0186] 2) Back propagation: The gradient of the loss relative to the Critic network parameters is calculated through the back propagation algorithm;

[0187] 3) Update parameters: Use the Adam optimizer to perform gradient updates on the parameters of the Critic network.

[0188] According to the above principle, the MADDPG network training in the hydrogen energy storage power allocation model is established. Figure 3 As shown, the MADDPG algorithm training pseudo code of its overall training process is as follows Figure 4 shown.

[0189] S3. Collect historical day-ahead forecast data and actual power generation data of photovoltaic power plants, and train the MADDPG-based hydrogen energy storage power allocation model of step S2.

[0190] The adaptive power allocation of hydrogen energy storage based on MADDPG mainly includes the following steps:

[0191] 1) Construct an Actor-Critic deep reinforcement learning model, which consists of the Actor and Critic networks of each unit of the hydrogen energy storage charging system and the Actor and Critic networks of each unit of the hydrogen energy storage discharging system. Create an experience replay pool, define parameters such as the experience replay pool size, the number of rounds at the start of training, the maximum number of training rounds, and the batch size, and continuously adjust the parameters according to the validation set results during model training to select the MADDPG parameters suitable for the hydrogen energy storage power allocation environment. Finally, consider the actual running memory of the MADDPG algorithm hyperparameter settings such as Figure 5 shown.

[0192] 2) Based on the day-ahead forecast data of the photovoltaic power generation system and actual power generation data during the day The prediction error at time t is obtained, and the environmental parameters at time t are calculated according to the response state of each hydrogen energy storage unit. The total environmental parameters include time t, the prediction error at time t, and the charge state of each hydrogen energy storage unit at time t.

[0193] 3) The agent determines the current state of the environment Select Action In the action selection network, noise is added to the action for exploration, and as the number of training rounds increases, the noise is reduced to reduce the exploration of the action to achieve the stability of model training. According to the current environment state and action selection to obtain the maximum reward, the action (i.e., the power output of the hydrogen energy storage unit) is constrained by power, capacity and operation through the hydrogen energy storage operation environment, and multiple intelligent agent actions, environment states and corresponding rewards after the constraints are returned.

[0194] 4) MADDPG is based on a fully cooperative multi-agent environment. It calculates the sum of the environmental rewards of multiple agents as the total reward, with the goal of finding the optimal total reward. The tuple is formed based on the environmental state at time t, the action of the agent, the total reward of the agent, and the state of the environment at time t+1. Stored in the experience replay pool.

[0195] 5) Update the policy function according to the policy value network (Actor-Critic network), and calculate the policy loss (ActorLoss) and value loss (Critic Loss). Actor Loss is calculated based on the action value function (Q-value) of each agent, and the goal is to maximize the Q-value to improve the policy performance. Critic Loss is calculated according to the Bellman equation, and the goal is to make the action value function output by the Critic as close to the target Q-value as possible. Use the Adam optimizer to update the Actor and Critic networks based on the calculated loss. The optimal decision of the agent is achieved through the action selection of the Actor network and the action evaluation of the Critic network.

[0196] 6) Before the maximum number of training rounds is reached, the Actor and Critic networks are continuously trained through the two loss functions in 5) to update the model parameters so that the agent can learn effective strategies. After the training is completed, the trained Actor-Critic model is saved for model testing.

[0197] 7) Model testing: In the testing phase, the pre-trained Actor-Critic network is imported. During the testing process, the gradient update mechanism in the training process is reset to zero to prevent the gradient from affecting the network parameters. The loop test is performed, each test result is evaluated, and the corresponding performance indicators are recorded: the total power output and prediction error of hydrogen energy storage, the output of each hydrogen energy storage unit, and the charge state of each hydrogen energy storage unit. Based on the above indicators, the effect of compensating for the prediction error of photovoltaic power generation is calculated, the cost of purchasing positive and negative spinning reserves from the power grid, and the power generation income of the photovoltaic-hydrogen energy storage system. The generalization ability and practical application effect of the hydrogen energy storage unit power allocation model based on MADDPG are verified through multiple sets of data.

[0198] S4. Based on real-time photovoltaic output data, the hydrogen energy storage power allocation model is used to obtain the hydrogen energy storage power output instruction and coordinate the output of each hydrogen energy storage unit.

[0199] The main steps include:

[0200] 1) Initialize the multi-agent environment: including the environment’s state space, action space, and reward function.

[0201] 2) Create multiple agents and load the trained Actor-Critic model.

[0202] 3) Similar to the MADDPG model test, the gradient update mechanism is set to zero during the cycle, actions are performed in the environment, and the rewards and next-moment states are calculated, and this process is repeated until the termination condition is met.

[0203] 4) Visualize the results, evaluate the performance of the intelligent agent, observe the difference between the actual output of the hydrogen energy storage-photovoltaic system and the predicted data, the actual output of the hydrogen energy storage, and the charge state of each unit energy storage, and calculate the cost of purchasing positive and negative spinning reserve from the power grid based on the above indicators, and the power generation income of the photovoltaic-hydrogen energy storage system.

[0204] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0205] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0206] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning, characterized in that: The steps include: S1. Establish a power output control strategy for hydrogen energy storage units based on a multi-agent deep deterministic policy gradient algorithm, introduce the charge-discharge conversion times and state of charge modules, and set reward functions and constraints. S2. Based on the power output control strategy of step S1, a hydrogen energy storage power allocation model is established; S3, collecting historical day-ahead forecast data and actual power generation data of photovoltaic power plants, and training the hydrogen energy storage power allocation model of step S2; S4. Based on real-time photovoltaic output data, the hydrogen energy storage power allocation model is used to obtain hydrogen energy storage power output instructions and coordinate the output of each hydrogen energy storage unit; The hydrogen energy storage unit includes a charging system and a discharging system. When the output of the fuel cell of the discharging system cannot meet the demand, the fuel cell of the charging system is used to supplement the response; when the output of the electrolyzer of the charging system cannot meet the demand, the electrolyzer of the discharging system is used to supplement the response; when the charging system or the discharging system reaches the state of charge boundary, the functions of the two systems are exchanged; the charge-discharge conversion number and state of charge module consider the charge-discharge conversion number and SOC balance to improve the output power efficiency of the fuel cell and the electrolyzer; In the power output control strategy of the hydrogen energy storage unit based on the MADDPG algorithm, the intelligent units are in a fully cooperative relationship, and each intelligent unit is set with the same reward function, which includes spinning reserve costs and power generation income. The spinning reserve costs are generated by the deviation between the data reported by the photovoltaic power generation system a day ago and the actual power generation data, and the power generation income is determined by the total power generated by the photovoltaic system and the hydrogen energy storage system. When the photovoltaic power forecast value is greater than the actual value of the spinning reserve cost, the grid dispatching agency needs to purchase additional spinning reserve costs for the photovoltaic power station from the power plant that can provide spinning reserve, that is, to increase the positive spinning reserve for the photovoltaic power station to ensure that the sum of the spinning reserve output and the actual output reaches the forecast value a day ago; when the forecast value is less than the actual value, the grid dispatching agency needs to arrange additional negative spinning reserve for it. The operating constraints are that during the response process, the hydrogen energy storage unit is in a fully cooperative state. If the photovoltaic output prediction error is greater than 0, the output of all hydrogen energy storage units must be greater than 0, and the total output cannot overshoot, that is, the total power output of hydrogen energy storage is less than the prediction error; if the photovoltaic output prediction error is less than 0, the output of each hydrogen energy storage unit is a negative value, and the absolute value of the total output is less than the absolute value of the prediction error.

2. The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: In step S1, the charge-discharge conversion times and the state-of-charge module expressions are as follows: Where P dif The difference between the actual photovoltaic output and the predicted output; N is the number of hydrogen energy storage units that need to be operated, that is, the number of agents in the MADDPG algorithm; P opt is the power at the optimal efficiency point of the hydrogen energy storage unit, and adding or subtracting δ indicates that it is operating near the optimal efficiency point; During the discharge process, the hydrogen energy storage unit with a higher SOC is discharged first; during the charging process, the hydrogen energy storage unit with a lower SOC is charged first; In the formula, R SOC is the reward part of SOC balance in the MADDPG algorithm. SOC(i,t) is the state of charge of hydrogen energy storage unit i at time t. mean (t) is the average SOC value of all hydrogen energy storage units in all charging systems or discharging systems; The expression for spinning reserve cost is as follows: Where P for is the predicted output value of the photovoltaic power station, P pv is the actual output value of the photovoltaic power station; α and β are the compensation unit prices of positive and negative spinning reserve costs, respectively, in yuan / MWh; C rota (t) is the cost of purchasing spinning reserve for the photovoltaic power station within Δt period starting from time t; The total cost of spinning reserve in one day is: When the actual value of the combined output of photovoltaic and hydrogen energy storage is greater than the forecast value reported a day ago, the photovoltaic power station needs to pay the grid dispatching agency for the purchase of negative spinning reserve, the value of which is: Where P HESS (t) is the energy storage output value at time t, Bool neg is a Boolean constant. When the prediction error is still less than 0 after the hydrogen energy storage unit is compensated, the Boolean constant takes the value of 1; When the actual value of the combined output of photovoltaic and hydrogen energy storage is less than the forecast value reported a day ago, the photovoltaic power station needs to pay the grid dispatching agency for the purchase of positive spinning reserve fees, the value of which is: When the prediction error is still greater than 0 after the hydrogen energy storage unit is compensated, the Boolean constant takes 1; The expression of power generation revenue is as follows: In the formula, C e (t) is the time-of-use electricity price, P pv (t) is the actual output of the photovoltaic power generation system at time t, P HESS (i,t) is the response power of the i-th hydrogen energy storage unit at time t; In the power output control strategy of the hydrogen energy storage unit based on the MADDPG algorithm, the agent units are in a fully cooperative relationship, and each agent is set with the same reward function: reward=-αR SOC -βC rota (t)+γC earn (t); Where α, β, and γ are parameters of the same dimension.

3. The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning according to claim 2 is characterized in that: In step S1, the constraints of the hydrogen energy storage unit are as follows: -P HESSn (i)≤P HESS (i,t)≤P HESSn (i); P HESS (i,t)=P ch (i,t)u ch (i,t)-P dch (i,t)u dch (i,t); u ch (i,t)+u dch (i,t)=1; Where P HESSn (i) is the maximum charging and discharging power of hydrogen energy storage unit i; P ch (i,t) and P dch (i, t) are the charging power and discharging power of the i-th hydrogen energy storage unit at time t. At the same time, one of them must be 0, so they are restricted by a 0-1 variable, that is, u ch (i,t) and u dch (i, t), are the charging 0-1 variable and the discharging 0-1 variable, respectively; The energy constraints of the hydrogen energy storage unit are as follows: k min S HESSn (i)≤S HESS (i,t)≤k max S HESSn (i); S HESS (i,t)+P ch (i,t)u ch (i,t)T-P dch (i,t)u dch (i,t)T=S ESS (i,t+1); In the formula, k max and k min are the upper and lower limits of energy storage SOC, S HESSn (i) is the installed capacity of the i-th hydrogen energy storage unit, S HESS (i, t) is the remaining hydrogen energy storage capacity of the i-th hydrogen energy storage unit at time t; The constraints of the agent during operation are as follows: The power output control strategy of the hydrogen energy storage unit based on the MADDPG algorithm also includes a state space and an action space. The state space is the environmental information perceived by the agent, and the action space is the decision variable of the hydrogen energy storage unit. The state space of the i-th agent in period t can be expressed as: s i ={P pv (t),P for (t),SOC(i,t),t}; Where P pv (t) is the photovoltaic output at time t, P for (t) is the predicted photovoltaic power output at time t, SOC(i,t) is the state of charge of the i-th hydrogen energy storage unit at time t; The action space of each agent, i.e. the power output of the hydrogen energy storage unit at time t, can be expressed as: a i ={P HESS (i,t)}; Where P HESS (i,t) is the power output of the i-th hydrogen energy storage unit at time t.

4. The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: In step S2, the hydrogen energy storage power allocation model is established as follows: In the MADDPG algorithm, each agent has its own strategy network and value network. The Actor network is used to determine actions, and the Critic network is used to evaluate the quality of the current strategy. The strategy parameters of each agent are: θ={θ1,…,θ n }; Let π={π1,…,π n } is the strategy set of all agents. For random strategies, the policy gradient of the expected return of each agent i is: In the formula, θ i are the policy network parameters of agent i, is the centralized action value function, x=(s1,…,s N ) is the joint observation space, which represents the set of observation states of N agents. Each agent learns its own Function, so that each agent can define a different reward function to achieve cooperation or competition; Expanding to deterministic strategies Based on the basic theory of deep deterministic policy gradient algorithm, it is extended to multiple agents. For each agent i, its policy network update formula can be expressed as: In the formula, μ i is the strategy function of agent i, which is based on the agent's observation s i Produce action a i , Represents the experience replay buffer. represents sampling from the experience replay pool, which stores the current state, next state, current action, and current reward of each agent in each round of training, namely: (x, x′, a1,…, a N ,r1,…,r N ), is the Critic network of agent i, which evaluates the behavior of agent i given the global state x and all agent actions a1,…,a N Expected return under The Actor network is updated by gradient descent, which mainly includes the following steps: Calculate the policy gradient: For each agent, the current state s i Output Actor network and generate corresponding action a i ; Calculate Q value through Critic network Calculate the gradient of the action with respect to the Q value Calculate the gradient of the Actor network parameters Update Actor network parameters: Update the parameters of the Actor network using the Adam optimizer through gradient descent. The update of the Critic network uses the mean square error loss to minimize the difference between the target Q value and the Q value predicted by the value network. The gradient formula of the Critic network loss function is as follows: In the formula, φ i is the value network parameter of agent i; y is the target Q value, which is calculated by the Bellman equation and is usually given by the reward r i And the value prediction of the next state: Where γ is the discount factor for future rewards, μ ′ represents the target policy network, x ′ represents the next state of the environment, a′1,...,a′ N Represents the actions of each agent; By minimizing the Critic loss function, the parameters of the Critic network can be updated so that the predicted Q value is closer to the target Q value. The network parameter update adopts the gradient descent method. The specific steps include: Calculate loss: Calculate the mean square error loss between the target Q value and the current Q value based on the batch samples; Back propagation: Calculate the gradient of the loss relative to the Critic network parameters through the back propagation algorithm; Update parameters: Use the Adam optimizer to calculate the gradient to update the parameters of the Critic network.

5. The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning according to claim 4 is characterized in that: In step S3, the hydrogen energy storage power allocation model is trained as follows: S31. Construct an Actor-Critic deep reinforcement learning model. Create an experience replay pool by using the Actor network and Critic network of each unit in the hydrogen energy storage unit charging system and the Actor network and Critic network of each unit in the hydrogen energy storage unit discharging system. Define the experience replay pool size, the number of rounds at the start of training, the maximum number of training rounds, and the batch size parameters. Adjust the parameters according to the verification set results during model training and select the MADDPG parameters for the hydrogen energy storage power distribution environment. S32, based on the photovoltaic power generation system day-ahead forecast data P for (t) and the actual power generation data P pv (t) obtain the prediction error at time t, and calculate the environmental parameter SOC(i,t) at time t according to the response state of each hydrogen energy storage unit; the total environmental parameter includes time t, the prediction error at time t, and the charge state of each hydrogen energy storage unit at time t; S33. The agent selects action a according to the current environment state x. In the Actor network, noise is added to action a for exploration. Gaussian noise is used to encourage the agent to explore more action space. As the number of training rounds increases, the noise is reduced to reduce the exploration of actions to achieve the stability of model training. According to the current environment state x and action selection a, the action is constrained by the hydrogen energy storage operating environment, including the power and energy boundary constraints of the hydrogen energy storage unit during operation, and the operating constraints between the hydrogen energy storage output and the actual demand. The constrained multiple agent actions a and environment state x are returned. ′ and the corresponding reward r; the agent updates its internal state according to the returned environment state, reward information and constrained action information, and stores the returned environment state, reward information and constrained action information in the experience replay pool buffer for subsequent training optimization; in subsequent training optimization, data is sampled from the experience replay pool buffer to update the policy network and value network to optimize the agent's decision-making strategy; S34. The hydrogen energy storage power allocation model is based on a fully cooperative multi-agent environment. The environmental rewards of multiple agents are calculated as the total reward to seek the optimal total reward. According to the environmental state at time t, the action of the agent, the total reward of the agent, and the state of the environment at time t+1, a tuple {x, a, r, x ′ }Stored in the experience replay pool; S35. In the MADDPG algorithm, the policy function in the policy-value network is updated through the loss function. The policy loss is calculated based on the action value function of each agent, and the value loss is calculated based on the Bellman equation. The calculation of Actor Loss involves maximizing the Q-value to improve the performance of the policy; the calculation of Critic Loss optimizes the value function by minimizing the error between the predicted value and the target value to ensure the accuracy of the policy. S36, before reaching the maximum number of training rounds, train the Actor and Critic networks through the two loss functions of Actor Loss and Critic Loss in S35, update the model parameters, so that the agent can learn effective strategies, and save the trained Actor-Critic network model after the training is completed for model testing; S37, model testing, import the trained Actor-Critic network model; during the test process, ensure that the gradient update mechanism in the training phase is set to zero to prevent the gradient from affecting the network parameters during the test; perform a loop test, evaluate each test sample, and record the corresponding performance indicators to ensure the stability of the test process and the accuracy of the results, and verify the generalization ability of the model and the actual application effect.

6. The hydrogen energy storage unit power allocation method based on multi-agent deep reinforcement learning according to claim 5 is characterized in that: In step S4, coordinating the output of each hydrogen energy storage unit includes the following steps: S41. Initialize the multi-agent environment, including the state space, action space and reward function; S42, create an agent and load the trained Actor-Critic model; S43, in the loop process, set the gradient update mechanism to zero, perform actions in the environment and calculate the corresponding rewards and the next moment state, and repeat the process until the termination condition is met; S44. Visualize the results, evaluate the performance of the intelligent agent, observe the difference between the actual output of the hydrogen energy storage-photovoltaic system and the predicted data, and record the actual output of the hydrogen energy storage and the charge state of each unit energy storage.

Images