A real-time optimization operation method of an electricity-hydrogen coupling island microgrid

Abstract

This invention relates to a real-time optimized operation method for an electric-hydrogen coupled island microgrid, belonging to the field of island microgrid optimized operation technology. It solves the problems of insufficient agent collaboration and difficulty in adapting to individual differences in complex power grid environments in existing reinforcement learning methods for controlling electric-hydrogen coupled systems. The method includes: acquiring historical data of the island microgrid to generate a first sample dataset; training an output prediction model based on the first sample dataset to obtain the optimal output prediction model, and then generating a second sample dataset; constructing a power grid multi-agent model; training the power grid multi-agent model based on the second sample dataset to obtain the optimal power grid multi-agent model; acquiring current historical and current data of the island microgrid, and then controlling each device based on the optimal output prediction model and the power grid multi-agent model to achieve real-time optimized operation of the island microgrid.

Description

Technical Field

[0001] This invention relates to the field of island microgrid optimization operation technology, and in particular to a real-time optimization operation method for an electric-hydrogen coupled island microgrid. Background Technology

[0002] In the context of building a new power system, the contradiction between the volatility of renewable energy output and the stability of the power system is becoming increasingly prominent. This is especially true in off-grid scenarios such as island microgrids, where the supply and demand balance problem of new energy sources, represented by wind power and photovoltaics, is more pronounced due to geographical conditions and resource constraints. Hydrogen energy, as a secondary energy carrier with both high energy density and zero carbon emissions, offers an innovative approach to smoothing intraday power fluctuations and improving system regulation capabilities by integrating it into the electricity-hydrogen coupling system (ECS) dispatch framework.

[0003] In current ECS research, deep reinforcement learning methods have been partially applied and have shown considerable performance. However, each agent usually shares the same network structure and parameters, which limits the agents' ability to cooperate and divide tasks. In particular, it is difficult to adapt to individual differences in complex power grid environments, resulting in a decline in control performance. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a real-time optimized operation method for an electric-hydrogen coupled island microgrid, in order to solve the problems of insufficient agent division of labor and cooperation capabilities and difficulty in adapting to individual differences in complex power grid environments in the process of controlling electric-hydrogen coupled systems by existing reinforcement learning methods.

[0005] This invention provides a real-time optimized operation method for an electric-hydrogen coupled island microgrid. The method comprises the following steps:

[0006] Historical data of the island microgrid was acquired and cleaned to generate the first sample dataset; the historical data included renewable energy output data, meteorological data, and power grid data.

[0007] Construct a power output prediction model; train the power output prediction model based on the first sample dataset to obtain the optimal power output prediction model;

[0008] Based on the optimal power output prediction model, the first sample dataset, and the cleaned historical data, a second sample dataset is generated.

[0009] A multi-agent model of the power grid is constructed; the multi-agent model of the power grid is trained based on the second sample dataset to obtain the optimal multi-agent model of the power grid; wherein, each agent is used to control the output or input power of each device in the island microgrid;

[0010] By acquiring historical and current data of the island microgrid, and then controlling each device based on the optimal power output prediction model and the power grid multi-agent model, the real-time optimized operation of the island microgrid can be achieved.

[0011] Furthermore, each first sample in the first sample dataset includes first feature data, second feature data, and a target variable. The first feature data is time series data of renewable energy output data and meteorological data with a set number of time steps. The second feature data is the 10-meter wind speed at the next time step of the corresponding first feature data. The target variable is the renewable energy output data at the next time step of the corresponding first feature data.

[0012] Furthermore, the output prediction model includes a relational feature extraction module, a time-series feature extraction module, and a time-series feature fusion module;

[0013] The relationship feature extraction module is used to generate data relationship features based on the input first feature data;

[0014] The time-series feature extraction module is used to generate multi-scale time-series features based on the received first feature data and data relationship features;

[0015] The time-series feature fusion module is used to generate target variables based on the received multi-scale time-series features and output them.

[0016] Furthermore, the loss function L used when training the output prediction model based on the first sample dataset is expressed as:

[0017] L = L MSE +L neg +L cur +L ene

[0018] In the formula, L MSE L neg L cur L ene These represent MSE loss, power non-negativity constraint, power curve smoothing constraint, and power upper limit constraint, respectively.

[0019] Furthermore, MSE loss L MSE Power nonnegativity constraint L neg Power curve smoothing constraint L cur Power upper limit constraint L ene They are represented as follows:

[0020]

[0021] In the formula, n represents the number of the first sample. Let represent the predicted values ​​of offshore wind power output and photovoltaic power output in the target variables at the next time step t, when the input to the power output prediction model is the first feature data in the i-th first sample. Let v represent the actual values ​​of the offshore wind power output and photovoltaic power output in the i-th first sample at the next time step t. t,i P represents the actual value of the 10-meter wind speed in the second feature data of the i-th first sample at the next time step t. wp,max P vp,max f represents the theoretical maximum power of offshore wind power and photovoltaic installations, respectively. the This represents the theoretical wind speed-power curve of the island microgrid, where max() indicates taking the maximum value.

[0022] Furthermore, each second sample in the second sample dataset includes predicted renewable energy output data, second feature data, and grid data at the same time step;

[0023] Furthermore, the multi-agent model of the power grid includes multiple agents, each agent corresponding to a device in the island microgrid; each agent includes an Actor network; wherein,

[0024] The Actor network generates and outputs actions based on input state observations; where state observations are the current time step and the predicted output power of offshore wind power and photovoltaic devices at the current time step, island load, grid-connected electricity price, hydrogen storage capacity, energy storage capacity, and energy storage power; actions are the output or input power of control equipment.

[0025] Furthermore, the power grid multi-agent model is trained in the following manner:

[0026] Construct multiple alternative Actor networks with different structures

[0027] For each agent, a master actor network and a master critic network are created, and the master actor network and master critic network are copied to generate the corresponding target actor network and target critic network; wherein, the master actor network is generated based on each candidate actor network;

[0028] Configure training and network parameters, and create an experience replay buffer;

[0029] Based on the second sample dataset, each agent is initially trained, and then each candidate Actor network is pre-trained. Based on the set selection criteria, the corresponding candidate Actor network is selected as the main Actor network. Then, based on the current main Actor network and the main Critic network, each agent is formally trained to obtain the trained main Actor network, which is then used as the optimal power grid multi-agent model.

[0030] Furthermore, three alternative Actor networks with different structures are constructed: a first alternative Actor network, a second alternative Actor network, and a third alternative Actor network.

[0031] The first candidate Actor network is a three-layer fully connected neural network, consisting of an input layer, a first hidden layer, a second hidden layer, and an output layer connected in sequence; wherein, the first and second hidden layers use ReLU as the activation function, and the output layer uses the hyperbolic tangent function as the activation function;

[0032] The second candidate Actor network is a four-layer fully connected neural network, consisting of an input layer, a third hidden layer, a fourth hidden layer, a fifth hidden layer, and an output layer connected in sequence; wherein, the third hidden layer, the fourth hidden layer, and the fifth hidden layer use ReLU as the activation function, and the output layer uses the hyperbolic tangent function as the activation function;

[0033] The third alternative Actor network is a hybrid network of convolutional and fully connected layers, comprising a first convolutional layer, a second convolutional layer, and a fully connected layer connected in sequence; wherein the first and second convolutional layers use ReLU as the activation function, and the fully connected layer uses the hyperbolic tangent function as the activation function.

[0034] Furthermore, the training processes for initial training, pre-training, and formal training are the same, but their termination conditions are different. The termination condition for initial training is that the amount of experience in the experience replay buffer reaches the set batch size; the termination condition for pre-training is that the number of training rounds reaches the set threshold for the number of training rounds in pre-training; and the termination condition for formal training is that the number of training rounds reaches the set threshold for the number of training rounds in formal training.

[0035] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0036] This invention provides a real-time optimization operation method for an electric-hydrogen coupled island microgrid. It obtains first sample data by acquiring historical data, trains and processes a prediction model to obtain the optimal output prediction model, and then generates a second sample set. Based on the second sample set, it trains a constructed multi-agent model of the power grid to obtain the optimal multi-agent model. Based on this, it performs real-time optimization operation of the island microgrid. By using the output prediction model to predict renewable energy output, it improves prediction accuracy. By dynamically evaluating the performance of candidate actor networks and adaptively selecting the optimal strategy, it effectively solves the problem of insufficient adaptability caused by shared agent parameters in traditional algorithms, enabling each device in the electric-hydrogen coupled system to optimize operation according to individual differences. The proposed operation method can effectively predict renewable energy output and optimize real-time scheduling strategies, improving the agent collaboration capabilities of each device, adapting to individual differences in complex power grid environments, and significantly improving the economic benefits of the electric-hydrogen coupled system while ensuring power system stability. Furthermore, it only needs to utilize historical data and device information to achieve continuous control through localized intelligent algorithms, reducing the dependence of traditional centralized control on communication and data processing.

[0037] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0039] Figure 1 This is a flowchart illustrating the real-time optimized operation method of the electric-hydrogen coupled island microgrid provided in Embodiment 1 of the present invention.

[0040] Figure 2 This is a schematic diagram of the structure of the electric-hydrogen coupled island microgrid provided in Embodiment 1 of the present invention;

[0041] Figure 3 This is a schematic diagram of the network structure of each candidate Actor network provided in Embodiment 1 of the present invention;

[0042] Figure 4 This is a schematic diagram of the simulation results in Example 1 provided in Embodiment 2 of the present invention;

[0043] Figure 5 This is a schematic diagram of the simulation results in Example 2 provided in Embodiment 2 of the present invention;

[0044] Figure 6 This is a schematic diagram of the simulation results provided in Example 3 according to Embodiment 2 of the present invention. Detailed Implementation

[0045] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0046] Example 1

[0047] A specific embodiment of the present invention discloses a real-time optimized operation method for an electric-hydrogen coupled island microgrid, such as... Figure 1 As shown.

[0048] S1. Obtain historical data of the island microgrid and perform data cleaning to generate the first sample dataset; the historical data includes renewable energy output data, meteorological data and power grid data.

[0049] Specifically, renewable energy output data includes the output power of offshore wind power and the output power of photovoltaic devices; meteorological data includes regional irradiance, 10-meter wind direction, 10-meter wind speed, temperature, humidity, pressure, and actual irradiance; grid data includes island load, grid-connected electricity price, hydrogen storage capacity, energy storage capacity, and energy storage power.

[0050] Specifically, historical data of the island microgrid is acquired at set time intervals. An outlier detection method based on interquartile range is used to identify and correct outliers in the historical data. At the same time, linear interpolation is used to fill in missing data in the historical data to maintain the continuity of the time series. The Z-score method is used to normalize the historical data to make the data standardized, thus completing the historical data cleaning.

[0051] In implementation, each first sample in the first sample dataset includes first feature data, second feature data, and target variable. The first feature data is time series data of renewable energy output data and meteorological data with a set number of time steps. The second feature data is the 10-meter wind speed of the corresponding first feature data in the next time step. The target variable is the renewable energy output data of the corresponding first feature data in the next time step.

[0052] Specifically, a sliding window is constructed based on a set number of time steps to sequentially obtain each first sample data. The number of time steps is determined based on a set time span and the time interval for acquiring data.

[0053] For example, in this embodiment, the time interval for acquiring data is 15 minutes, the time span is 24 hours, and the number of time steps is 96.

[0054] S2. Construct a power output prediction model; train the power output prediction model based on the first sample dataset to obtain the optimal power output prediction model.

[0055] In implementation, the power output prediction model includes a relational feature extraction module, a time-series feature extraction module, and a time-series feature fusion module;

[0056] The relationship feature extraction module is used to generate data relationship features based on the input first feature data;

[0057] The time-series feature extraction module is used to generate multi-scale time-series features based on the received first feature data and data relationship features;

[0058] The time-series feature fusion module is used to generate target variables based on the received multi-scale time-series features and output them.

[0059] In specific implementation, the relation feature extraction module includes an input layer, a variable embedding layer, an inverted self-attention layer, and a temporal feedforward network connected in sequence. The variable embedding layer maps the time series of each variable receiving the first feature data into a high-dimensional vector through an independent fully connected layer. The inverted self-attention layer calculates attention weights on the variable dimension of the high-dimensional vector and weightedly fuses the variable features. The temporal feedforward network independently applies a two-layer fully connected network containing activation functions and Dropout to each variable based on the weighted fused variable features to extract the data relation features. Each layer contains residual connections and layer normalization to stabilize training.

[0060] In specific implementation, the temporal feature extraction module includes multiple stacked residual blocks; each residual block includes a dilated causal convolutional layer, a normalization layer, an activation function, and a dropout layer connected in sequence. The input of the residual block and the output of the dilated causal convolutional layer are connected through residual connections. The dilated causal convolution achieves causality through left padding, captures long-term dependencies in the sequence, and mitigates gradient vanishing by utilizing residual connections.

[0061] Specifically, the left padding of the dilated causal convolutional layer is determined by the kernel size and the dilation factor; the dilation factor of each dilated causal convolutional layer increases exponentially to progressively capture multi-scale temporal dependencies. For example, the dilation factors are 1, 2, 4, 8, etc.

[0062] Specifically, when the dimensions of the residual connections do not match, they are adjusted using 1×1 convolutions.

[0063] In practical implementation, the temporal feature fusion module includes a forward GRU, a reverse GRU, a gating mechanism, and a fully connected layer. Each GRU controls the direction and memory method of temporal information flow through a gating mechanism, including update gates, reset gates, and candidate states. The forward GRU processes the input sequence in chronological order to capture the dependencies from history to the future. The reverse GRU processes the input sequence in reverse chronological order to capture the dependencies from the future to the past. Then, the contextual information is obtained by concatenating or averaging the bidirectional outputs, and the target variable is output through the fully connected layer. The bidirectional GRU structure learns the dynamic contextual features of the sequence, enhancing the modeling of temporal patterns.

[0064] Understandably, in this embodiment, the input data is first converted into a token sequence by the relation feature extraction module. The global dependency between meteorological factors and power generation is explicitly modeled through a reverse self-attention mechanism. Local temporal patterns are extracted using MLP in the time dimension, and the output relation features are then input together with the feature data into the temporal feature extraction module. Multi-scale temporal features are captured through a one-dimensional convolutional layer with deformable convolutional kernels, and the feature reuse and generalization capabilities are enhanced by inter-layer residual connections and Dropout layers. Subsequently, the multi-scale temporal features output by the temporal feature extraction module are input into the temporal feature fusion module. Forward and backward temporal information are fused through a gating mechanism, and finally, the prediction result is output through a fully connected layer.

[0065] During implementation, the power output prediction model is trained in the following ways:

[0066] S21. Divide the first sample dataset into a training set and a validation set, and set the optimizer, learning rate and loss function during the training process;

[0067] Specifically, the optimizer is the Adam optimizer; the learning rate is dynamically implemented through the CosineAnnealing learning rate scheduler.

[0068] Specifically, the loss function L is expressed as:

[0069] L = L MSE +L neg +L cur +L ene

[0070] In the formula, L MSE L neg L cur L ene These represent MSE loss, power non-negativity constraint, power curve smoothing constraint, and power upper limit constraint, respectively.

[0071] More specifically, MSE loss L MSE Power nonnegativity constraint Lneg Power curve smoothing constraint L cur Power upper limit constraint L ene They are represented as follows:

[0072]

[0073]

[0074] In the formula, n represents the number of the first sample. Let represent the predicted values ​​of offshore wind power output and photovoltaic power output in the target variables at the next time step t, when the input to the power output prediction model is the first feature data in the i-th first sample. Let v represent the actual values ​​of the offshore wind power output and photovoltaic power output in the i-th first sample at the next time step t. t,i P represents the actual value of the 10-meter wind speed in the second feature data of the i-th first sample at the next time step t. wp,max P vp,max f represents the theoretical maximum power of offshore wind power and photovoltaic installations, respectively. the This represents the theoretical wind speed-power curve of the island microgrid, where max() indicates taking the maximum value.

[0075] S22. The power output prediction model is trained using the training set. Based on the learning rate and optimizer, the network hyperparameters are adjusted using the backpropagation method to minimize the loss function, thus obtaining the trained power output prediction model.

[0076] Specifically, training stops when the loss function falls below a set first threshold. For example, the first threshold is set to 0.01.

[0077] S23. Validate the trained output prediction model based on the validation set. If the conditions are met, the trained output prediction model is taken as the optimal output prediction model; otherwise, return to step S22 and retrain.

[0078] Specifically, the condition is that the standard deviation between the predicted value and the actual value of the output prediction model is less than a set second threshold. For example, the second threshold is set to 0.01.

[0079] It is understandable that the power output prediction model constructed in this embodiment improves the prediction accuracy by predicting renewable energy power output. The nonlinear correlation between meteorological parameters and power generation is captured by the relation feature extraction module. At the same time, the prediction model jointly optimized by the time series feature extraction module and the time series feature fusion module can effectively capture the spatiotemporal characteristics of wind and solar power output, solve the problem of renewable energy volatility in island microgrids, and provide reliable input for subsequent optimized scheduling.

[0080] S3. Based on the optimal output prediction model, the first sample dataset, and the cleaned historical data, generate the second sample dataset.

[0081] Specifically, each second sample in the second sample dataset includes predicted renewable energy output data, second feature data, and grid data at the same time step.

[0082] Specifically, based on the first sample dataset and the power output prediction model, the predicted data of the target variables at each time step are obtained. The predicted data of the target variables at each time step are combined with the power grid data and the second feature data at the corresponding time step to form the second sample dataset.

[0083] S4. Construct a multi-agent model for the power grid; train the multi-agent model for the power grid based on the second sample dataset to obtain the optimal multi-agent model for the power grid; wherein, each agent is used to control the output or input power of each device in the island microgrid.

[0084] In implementation, the power grid multi-agent model includes multiple agents, each corresponding to a device in the island microgrid; each agent includes an Actor network; wherein,

[0085] The Actor network generates and outputs actions based on input state observations; where state observations are the current time step and the predicted output power of offshore wind power and photovoltaic devices at the current time step, island load, grid-connected electricity price, hydrogen storage capacity, energy storage capacity, and energy storage power; actions are the output or input power of control equipment.

[0086] Specifically, such as Figure 2 As shown, the devices in the electric-hydrogen coupled island microgrid in this embodiment include conventional energy storage, an electrolyzer, a hydrogen storage system, and a fuel cell; wherein, the intelligent agents of the conventional energy storage, hydrogen storage system, and fuel cell are responsible for controlling the output power of the corresponding devices; and the intelligent agent of the electrolyzer is responsible for controlling the input power of the device.

[0087] During implementation, the power grid multi-agent model is trained using the following methods:

[0088] S41. Construct multiple alternative Actor networks with different structures;

[0089] Specifically, in this embodiment, as Figure 3 As shown, three alternative Actor networks with different structures are constructed: a first alternative Actor network, a second alternative Actor network, and a third alternative Actor network.

[0090] More specifically, the first candidate Actor network is a three-layer fully connected neural network, including an input layer, a first hidden layer, a second hidden layer and an output layer connected in sequence; wherein, the first hidden layer and the second hidden layer use ReLU as the activation function, and the output layer uses the hyperbolic tangent function as the activation function.

[0091] Specifically, in the first candidate Actor network, the input layer receives state observations and has a dimension of dim(o); the first hidden layer has a dimension of hidden dim1, representing the number of neurons in that layer; the node structure from the input layer to the first hidden layer is dim(o). i The dimension of the second hidden layer is also hidden dim1, representing the number of neurons in that layer; the node structure from the first hidden layer to the second hidden layer is hidden dim1×hidden dim1; the dimension of the output layer is dim(a), representing the dimension of the output action; the node structure from the second hidden layer to the output layer is hidden dim1×dim(a).

[0092] For example, the value of hidden dim1 is set to 128.

[0093] More specifically, the second candidate Actor network is a four-layer fully connected neural network, including an input layer, a third hidden layer, a fourth hidden layer, a fifth hidden layer, and an output layer connected in sequence; wherein, the third hidden layer, the fourth hidden layer, and the fifth hidden layer use ReLU as the activation function, and the output layer uses the hyperbolic tangent function as the activation function.

[0094] Specifically, in the second candidate Actor network, the input layer receives state observations and has a dimension of dim(o); the dimension of the third hidden layer is hidden dim3, representing the number of neurons in this layer; the node structure from the input layer to the third hidden layer is dim(o). i The fourth hidden layer has a dimension of hidden dim3 × hidden dim4, representing the number of neurons in that layer; the node structure from the third to the fourth hidden layer is hidden dim3 × hidden dim4; the fifth hidden layer has a dimension of hidden dim5, representing the number of neurons in that layer; the node structure from the fourth to the fifth hidden layer is hidden dim4 × hidden dim5; the output layer has a dimension of dim(a), representing the dimension of the output action; the node structure from the fifth hidden layer to the output layer is hidden dim5 × dim(a).

[0095] For example, hidden dim3 is set to 64, hidden dim3 is set to 128, and hidden dim3 is set to 64.

[0096] More specifically, the third alternative Actor network is a hybrid network of convolutional and fully connected layers, comprising a first convolutional layer, a second convolutional layer, and a fully connected layer connected in sequence; wherein the first and second convolutional layers use ReLU as the activation function, and the fully connected layer uses the hyperbolic tangent function as the activation function.

[0097] Specifically, the first convolutional layer receives state observations with dimension dim(o), the first and second convolutional layers perform convolution operations, and the fully connected layer outputs actions with dimension dim(a).

[0098] For example, the first convolutional layer has a kernel of 3, and the second convolutional layer has a kernel of 3.

[0099] It is understood that in this embodiment, the first alternative Actor network is a general structure, the second alternative Actor network is applicable to complex tasks, and the third alternative Actor network is applicable to tasks with relevance.

[0100] S42. Create a master Actor network and a master Critic network for each agent, and copy the master Actor network and master Critic network to generate the corresponding target Actor network and target Critic network; wherein, the master Actor network is generated based on each candidate Actor network.

[0101] Specifically, initially, the network structure of the main Actor network is arbitrarily selected from the candidate Actor networks.

[0102] Specifically, the main Critic network is a deep neural network, for example, a fully connected network or a convolutional neural network.

[0103] S43. Configure training parameters and network parameters, and create an experience replay buffer; where training parameters include discount factor, policy noise, target network update rate, number of training rounds and number of training round steps, and network parameters include optimizer, hyperparameters and learning rate.

[0104] Specifically, the discount factor γ = 0.99 is used to adjust the weight of future rewards, the policy noise σ = 0.1 is used for action exploration, and the target network update rate τ = 0.005 is used to control the network update speed.

[0105] Specifically, the optimizer for each network is configured as Adam, and the learning rate for each network is set to 10. -4 .

[0106] Specifically, the buffer capacity of the experience playback buffer is set to 10. 6Its batch size is set to 256 for storing and sampling training data.

[0107] Specifically, in this embodiment, training includes pre-training and formal training. In pre-training, the threshold for the number of training rounds is set to 50, and the threshold for the number of iterations in a round is set to 96. In formal training, the threshold for the number of training rounds is set to 3000, and the threshold for the number of iterations in a round is set to 96.

[0108] S44. Based on the second sample dataset, perform initial training on each agent, then pre-train each candidate Actor network, and select the corresponding candidate Actor network as the main Actor network based on the set selection criteria. Then, perform formal training on each agent based on the current main Actor network and main Critic network to obtain the trained main Actor network, and use it as the optimal power grid multi-agent model.

[0109] It is understandable that by constructing Actor networks with different structures for training and selection, the multi-agent collaboration capability is enhanced. That is, by dynamically evaluating the performance of candidate Actor networks and adaptively selecting the optimal strategy, the problem of insufficient adaptability caused by the sharing of agent parameters in traditional algorithms is effectively solved, enabling each device in the electro-hydrogen coupling system to operate optimally according to individual differences.

[0110] In practice, the pre-training of each candidate Actor network involves pre-training each candidate Actor network as a separate main Actor network.

[0111] In practice, the initial training is based on the initial settings of the main Actor network and the main Critic network. If the current main Actor network and main Critic network have been trained but the termination condition has not been met, then one of the alternative Actor networks is randomly selected as the main Actor network to continue training until the termination condition is met.

[0112] In practice, the training processes for initial training, pre-training, and formal training are the same, but their termination conditions are different. The termination condition for initial training is that the amount of experience in the experience playback buffer reaches the set batch size. The termination condition for pre-training is that the number of training rounds reaches the set threshold for the number of training rounds in pre-training. The termination condition for formal training is that the number of training rounds reaches the set threshold for the number of training rounds in formal training.

[0113] In practice, the initial training, pre-training, and formal training processes are the same, including the following steps:

[0114] a1. Based on the second sample dataset and the current initial time step, reset the state observations and total round reward, and train for the current training round:

[0115] a11. Obtain the state observations of each agent at the current iteration time step, and then generate the actions of each agent in the current iteration.

[0116] Specifically, the action of each agent is represented as follows:

[0117]

[0118] In the formula, a i,k This represents the action value of agent i in the current iteration k. This indicates that the main Actor network of agent i has state observation o. i,k and parameters are The initial action value generated below, o i,k This represents the state observation of agent i in the current iteration k. N represents the weight parameters of the main Actor network, and N(0,σ) represents a random number that follows a Gaussian distribution with a standard deviation equal to the policy noise σ.

[0119] a12. Based on the actions of each agent in the current iteration, obtain the immediate reward for each agent in the current iteration.

[0120] Specifically, the immediate reward for each agent in the current iteration is the same, which is the immediate reward for the current iteration, expressed as:

[0121]

[0122] In the formula, r i,t r represents the immediate reward of agent i at time step t in the current iteration. t This represents the immediate reward at time step t corresponding to the current iteration. This represents the overall profit of the island microgrid. These represent penalties for hydrogen storage devices and conventional energy storage, respectively.

[0123] More specifically, the penalty for conventional energy storage Represented as:

[0124]

[0125] in,

[0126] In the formula, This represents the energy storage capacity at time step t, i.e., the amount of electricity held by conventional energy storage. This represents the energy storage capacity at time step t-1. These represent the maximum and minimum energy storage capacities of conventional energy storage, respectively. η represents the SOC penalty coefficient for conventional energy storage. ESThis is the charge / discharge coefficient for conventional energy storage, where Δt represents a fixed value. The fixed value Δt is set based on the data acquisition time interval and is used to calculate electrical energy; for example, if the data acquisition time interval is 15 minutes, Δt = 0.25.

[0127] More specifically, penalties for hydrogen storage equipment Represented as:

[0128]

[0129] in,

[0130] In the formula, This indicates the capacity of the hydrogen storage device at time step t, i.e., the amount of hydrogen remaining in the hydrogen storage device. This indicates the capacity of the hydrogen storage device at time step t-1; These represent the maximum and minimum values ​​of hydrogen that the hydrogen storage device can hold, respectively. This represents the SOC penalty coefficient for hydrogen storage devices; These represent the amount of hydrogen input and output to the hydrogen storage device at time step t, respectively, which are given by the intelligent agent controlling the hydrogen storage device. These represent the hydrogen storage and hydrogen release loss factors of the hydrogen storage device, respectively.

[0131] More specifically, the overall profit f of island microgrids total Represented as:

[0132]

[0133] In the formula, This represents the energy market profit at time step t. This represents the profit in the carbon trading market at time step t.

[0134] Among them, the energy market profit at time step t Represented as:

[0135]

[0136] In the formula, C represents the total energy market revenue at time step t. en Indicates the economic cost of operation. This indicates the grid connection price at time step t. This represents the power output by the island microgrid at time step t. This represents the price at which hydrogen is sold at time step t; This represents the amount of hydrogen used for sale at time step t, which is given by the intelligent agent controlling the hydrogen storage equipment. This represents the cost of generating electricity at time step t. k represents the electricity purchase cost at time step t. pv k wp k ES k PE k represents the unit power generation cost of photovoltaic, wind power, and hydrogen fuel cells, respectively. EL k S k ES These represent the unit operating costs of the electrolyzer, hydrogen storage equipment, and conventional energy storage equipment, respectively. These represent the predicted power output of the photovoltaic system and the offshore wind power generation at time step t, respectively. This represents the output power of the fuel cell at time step t, which is given by the intelligent agent controlling the fuel cell device; These represent the input power of the electrolyzer and the output power of the conventional energy storage at time step t, respectively, which are given by the intelligent agent controlling the electrolyzer and the conventional energy storage device. This represents the missing load power at time step t.

[0137] More specifically, the profit of island microgrids in the carbon trading market at time step t. Represented as:

[0138]

[0139]

[0140] In the formula, This represents the total revenue of the carbon trading market. ρ represents the cost of carbon emissions. ccer Indicates the selling price of CCE, ζ gc Indicates the marginal carbon emission factor of the power grid. Indicates the marginal carbon emission factor of gray hydrogen production. This represents the amount of hydrogen produced at time step t; ζ EL Indicates the electrolysis conversion factor; η EL Indicates the efficiency parameter of the electrolyzer; c represents the benchmark price of the carbon quota, E CQ Let E represent the initial carbon allowance for the island microgrid, q represent the range of carbon emissions, α represent the rate of increase in the price of the carbon allowance, and E L This indicates the actual carbon emissions.

[0141] a13. Sampling is performed in the experience replay buffer. Based on the sampled experience, the main Actor network and main Critic network of each agent are updated. Soft updates are then used to update the target Actor network and target Critic network of each agent.

[0142] Specifically, when sampling in the experience replay buffer, the sampling probability of each experience is expressed as:

[0143]

[0144] In the formula, W(j) represents the empirical sampling probability of j; p j p n represents the priorities of experiences j and n respectively; N represents the total number of experiences in the experience replay buffer; α represents a hyperparameter that controls the degree of influence of priorities. When α = 0, sampling becomes uniform sampling; when α increases, higher-priority experiences are sampled more frequently.

[0145] Specifically, gradient descent is used to update the main Critic network, and the Critic network loss function L... critic Represented as:

[0146]

[0147] In the formula, T represents the total number of time steps, t0 represents the initial time step, and ω i,t V(o) represents the sampling weight of the experience of agent i at time step t. i,t ), V(o i,t+1 ) represent the main Critic network with input and state observation 0 respectively. i,t The state observation is o i,t+1 The state value output at time, where γ represents the discount factor.

[0148] Wherein, the importance sampling weight ω of the experience of agent i at time step t is... i,t Represented as:

[0149]

[0150] In the formula, W(j i,t ) represents the experience j of agent i at time step t. i,t The sampling probability; β represents a hyperparameter used to control the degree of influence of importance sampling, which is gradually increased during training, from β=0 (no importance sampling) to β=1 (complete importance sampling).

[0151] Specifically, the main Actor network is updated based on the generalized dominance estimation, and the Actor network loss function L... actor Represented as:

[0152]

[0153] in, δ i,t =r i,t +γV(o i,t+1 )-V(o i,t );

[0154] In the formula, δ i,t λ represents the TD error of agent i at time step t; λ represents the trade-off parameter, preferably 0.95. represents the dominance value estimated by the generalized dominance estimate at time step t and time step t+1, respectively.

[0155] Specifically, soft updates are used to update the target Actor network and the target Critic network for each agent, as shown below:

[0156]

[0157] In the formula, Let represent the parameters of the main Actor network and the main Critic network of agent i, respectively. Let represent the parameters of the target Actor network and the target Critic network of agent i, respectively.

[0158] a14. Determine whether the current iteration count has reached the set threshold. If it has, end the current training round and obtain the total reward for the current round. If it has not, use the state observations of each agent in the next time step in the second sample dataset as the state observations of each agent in the next iteration, return to step a11 to continue execution, and obtain the experience of each agent in the current iteration and store it in the experience replay buffer.

[0159] Specifically, the total reward for the current round, episode_reward, is the sum of the instantaneous rewards of all agents across all iterations in the current training round, expressed as:

[0160]

[0161] In the formula, I represents the total number of agents, K represents the total number of iterations, and r i,k This represents the instantaneous reward for agent i at iteration number k.

[0162] It should be noted that each agent was trained using the second sample dataset, meaning that the state observations of each agent were the same.

[0163] Specifically, the experience of each agent in the current iteration is represented as (o i,t ,a i,t ,r i,t ,o i,t+1 ), where o i,t a i,t r i,t Let o represent the state observation, action value, and immediate reward of agent i at time step t corresponding to the current iteration, respectively. i,t+1 This represents the state observation of agent i at time step t in the next iteration.

[0164] Preferably, each experience in the experience replay buffer also includes the priority of each experience, and the priority of each experience is expressed as follows:

[0165] p j =|δ j |+ν

[0166] In the formula, p j δ represents the priority of experience j. j Let ν represent the TD error corresponding to experience j, and let ν represent a non-zero minimum constant, for example, ν = 0.001.

[0167] It should be noted that the priority of each experience is dynamically adjusted based on the TD error during the training process.

[0168] a2. Determine if the termination condition is met. If it is, stop training; otherwise, reset the initial time step and return to step a1 to execute the next training round.

[0169] Optionally, the initial time step can be reset to randomly select a time step from the time series of the second sample dataset as the initial time step for the next training round.

[0170] Optionally, the initial time step can be reset to the initial time step incremented by 1 and used as the initial time step for the next training round.

[0171] Specifically, the state observations of the current initial time step are reset based on the state observations corresponding to the current initial time step in the second sample dataset; the total round reward is reset to 0; and the initial value of the initial time step is set to the first time step of the second sample dataset.

[0172] Specifically, the selection criterion for each candidate Actor network after pre-training is to use the candidate Actor network corresponding to the maximum average cumulative reward of each training round as the main Actor network for formal training.

[0173] Understandably, this embodiment proposes a reward signal mechanism based on the dynamic characteristics of the electro-hydrogen coupling system and a multi-agent collaborative optimization strategy. This mechanism generates scheduling strategies through an Actor network, strategy values ​​through a Critic network, and updates the network with reward signals from a pricing strategy. For example, when a wind turbine outputs 5MW in its current state, the environmental model calculates the overall operational efficiency of the system at this power level based on renewable energy output data predicted by the output prediction model, combined with the operating status of equipment such as electrolyzers and hydrogen storage systems. Then, based on parameters such as current electricity prices and hydrogen conversion efficiency, it calculates the operating revenue of each device (including electricity sales revenue and hydrogen production revenue) and the overall system revenue (total revenue minus equipment operation and maintenance costs). When an agent's output power exceeds the equipment's safety limit, the system's overall revenue decreases due to increased adjustment costs. When the output power is too conservative, the system incurs economic losses due to insufficient energy utilization. The reward signal is obtained by balancing the local revenue of each agent with the overall system revenue, guiding agents to achieve optimal synergy between economic efficiency and low carbon emissions while ensuring equipment safety.

[0174] S5. Obtain historical and current data of the island microgrid, and then control each device based on the optimal power output prediction model and the power grid multi-agent model to achieve real-time optimized operation of the island microgrid.

[0175] Specifically, based on historical data of the current island microgrid, the first characteristic data is obtained. Then, the output prediction model with the optimal input value of the first characteristic data is used to obtain the current target variable. The action values ​​of each agent are obtained by combining the current target variable, the second characteristic data, and the grid data input values ​​to form the optimal grid multi-agent model, thus obtaining the output power value of each device. Based on this, the optimized operation of the island microgrid is achieved. Rolling optimization scheduling is performed on the island microgrid within a selected medium-term period to achieve economic and low-carbon target optimization within the system cycle.

[0176] Compared with existing technologies, this embodiment provides a real-time optimization operation method for an electric-hydrogen coupled island microgrid. It obtains first sample data by acquiring historical data, trains and processes a prediction model to obtain the optimal output prediction model, and then generates a second sample set. Based on the second sample set, it trains a constructed power grid multi-agent model to obtain the optimal power grid multi-agent model. Based on this, it performs real-time optimization operation of the island microgrid. By using the output prediction model to predict renewable energy output, it improves prediction accuracy. By dynamically evaluating the performance of candidate actor networks and adaptively selecting the optimal strategy, it effectively solves the problem of insufficient adaptability caused by shared agent parameters in traditional algorithms, enabling each device in the electric-hydrogen coupled system to optimize operation according to individual differences. The proposed operation method can effectively predict renewable energy output and optimize real-time scheduling strategies, improving the agent collaboration capabilities of each device, adapting to individual differences in complex power grid environments, and significantly improving the economic benefits of the electric-hydrogen coupled system while ensuring power system stability. Furthermore, it only needs to utilize historical data and device information to achieve continuous control through localized intelligent algorithms, reducing the dependence of traditional centralized control on communication and data processing.

[0177] Example 2

[0178] To verify the effectiveness of the method proposed in Example 1, a specific Example 2 is provided, in which the method proposed in Example 1 is simulated under three conditions.

[0179] Example 1: Without considering enhancing the resilience of the external power grid, the system arbitrarily connects to the external power grid during peak and off-peak periods without participating in demand response regulation. In this case, the ECS operates only in normal mode and does not optimize the peak-to-valley difference of the power grid, which may lead to local overload during peak periods or insufficient reserve during off-peak periods.

[0180] Example 2: The system participates in peak-shaving demand response, reducing external power purchases during peak load periods to alleviate local overload problems. In this mode, the ECS assists the grid in reducing operational pressure by lowering peak loads and obtains corresponding demand response incentives.

[0181] Example 3: The system participates in valley filling demand response, proactively increasing load during periods of low grid reserve to smooth out peak-valley differences. This strategy helps improve grid regulation capabilities, increase energy utilization during off-peak hours, and achieves economic benefits through demand response rewards.

[0182] For each instance, the scenario of participating in both the energy market and the carbon market simultaneously was simulated, and the results are as follows: Figure 4 , Figure 5 and Figure 6 As shown.

[0183] It can be seen that, under the same conditions, in Example 1, each device adopted a differentiated power output strategy when facing different states. For example... Figure 4 As shown, the renewable energy power output increases over time, and after 12:00, it can basically cover the island's load. Conventional energy storage devices, in order to reduce electricity purchase costs, concentrate their discharge activity when renewable energy output is insufficient, especially during peak hours when the system is short-handed, while charging activity concentrates when renewable energy is abundant and electricity prices are low. The strategy for hydrogen energy storage is similar to that of conventional energy storage devices. When the system is short-handed, it mainly reduces the system load by decreasing the input to the electrolyzer. Simultaneously, hydrogen storage devices choose to sell hydrogen during off-peak hours and during normal periods when electricity prices are low to increase system revenue. Furthermore, to increase system electricity sales revenue, both types of energy storage choose to concentrate discharge during peak hours when renewable energy is abundant.

[0184] In Example 2, such as Figure 5 As shown, selling electricity during peak hours will generate demand response incentives, therefore both conventional energy storage and hydrogen energy storage choose to concentrate their output during peak hours, especially during the morning peak hours when there is a power shortage. In addition, since the revenue from selling electricity during peak hours is increased, hydrogen storage devices correspondingly reduce the amount of hydrogen sold, and instead put the hydrogen into fuel cells to increase output.

[0185] In Example 3, such as Figure 6 As shown, all devices choose to discharge during off-peak hours to assist the external power grid in increasing load and achieving valley filling. The scheduling results for each example demonstrate that the proposed real-time optimization operation method has high generalization ability and can adapt to different scheduling scenarios.

[0186] As can be seen, the method proposed in Example 1 achieves synergistic optimization of economy and low carbon emissions. Simulation results show that, compared with traditional scheduling methods, the proposed algorithm significantly reduces operating costs and carbon emissions while ensuring stable system operation, thus verifying the comprehensive optimization effect of the method in off-grid microgrids.

[0187] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0188] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time optimized operation method for an electric-hydrogen coupled island microgrid, characterized in that, Includes the following steps: Historical data of the island microgrid is acquired and cleaned to generate a first sample dataset. The historical data includes renewable energy output data, meteorological data, and power grid data. Each first sample in the first sample dataset includes first feature data, second feature data, and a target variable. The first feature data is time series data of renewable energy output data and meteorological data with a set number of time steps. The second feature data is the 10-meter wind speed at the next time step of the corresponding first feature data. The target variable is the renewable energy output data at the next time step of the corresponding first feature data. Construct a power output prediction model; train the power output prediction model based on the first sample dataset to obtain the optimal power output prediction model; a loss function at training of the output prediction model based on the first sample dataset is represented as: ； In the formula, , , , The MSE loss, power nonnegativity constraint, power curve smoothing constraint, and power upper limit constraint are respectively represented as: ； ； ； ； In the formula, Indicates the number of the first sample. , These represent the inputs to the power output prediction model as the first... In the first sample, the first feature data is the target variable, which includes the output power of offshore wind power and the output power of photovoltaic devices, in the next time step. The predicted value, , They represent the first In the first sample, the target variables, offshore wind power output and photovoltaic power output, will be displayed in the next time step. The actual value, Indicates the first In the second feature data of the first sample, at the next time step The actual value of the 10-meter wind speed. , These represent the theoretical maximum power outputs of offshore wind power and photovoltaic installations, respectively. This represents the theoretical wind speed-power curve of the island microgrid, and max() indicates taking the maximum value; Based on the optimal power output prediction model, the first sample dataset, and the cleaned historical data, a second sample dataset is generated. A multi-agent model of the power grid is constructed; the multi-agent model of the power grid is trained based on the second sample dataset to obtain the optimal multi-agent model of the power grid; wherein, each agent is used to control the output or input power of each device in the island microgrid; By acquiring historical and current data of the island microgrid, and then controlling each device based on the optimal power output prediction model and the power grid multi-agent model, the real-time optimized operation of the island microgrid can be achieved.

2. The real-time optimized operation method for an electric-hydrogen coupled island microgrid according to claim 1, characterized in that, The power output prediction model includes a relation feature extraction module, a time series feature extraction module, and a time series feature fusion module; The relationship feature extraction module is used to generate data relationship features based on the input first feature data; The time-series feature extraction module is used to generate multi-scale time-series features based on the received first feature data and data relationship features; The time-series feature fusion module is used to generate target variables based on the received multi-scale time-series features and output them. 3.The real-time optimal operation method of the electro-hydro coupled island micro-grid of claim 1, wherein, Each second sample in the second sample dataset includes predicted renewable energy output data, second feature data, and grid data at the same time step. 4.The method of claim 1, wherein, The power grid multi-agent model includes multiple agents, each corresponding to a device in the island microgrid; each agent includes an Actor network; wherein... The Actor network generates and outputs actions based on input state observations; where state observations are the current time step and the predicted output power of offshore wind power and photovoltaic devices at the current time step, island load, grid-connected electricity price, hydrogen storage capacity, energy storage capacity, and energy storage power; actions are the output or input power of control equipment.

5. The method of claim 4, wherein, The power grid multi-agent model was trained using the following method: Construct multiple alternative Actor networks with different structures For each agent, a master actor network and a master critic network are created, and the master actor network and master critic network are copied to generate the corresponding target actor network and target critic network; wherein, the master actor network is generated based on each candidate actor network; Configure training and network parameters, and create an experience replay buffer; Based on the second sample dataset, each agent is initially trained, and then each candidate Actor network is pre-trained. Based on the set selection criteria, the corresponding candidate Actor network is selected as the main Actor network. Then, based on the current main Actor network and the main Critic network, each agent is formally trained to obtain the trained main Actor network, which is then used as the optimal power grid multi-agent model.

6. The real-time optimized operation method for an electric-hydrogen coupled island microgrid according to claim 5, characterized in that, Construct three alternative Actor networks with different structures: a first alternative Actor network, a second alternative Actor network, and a third alternative Actor network. The first candidate Actor network is a three-layer fully connected neural network, consisting of an input layer, a first hidden layer, a second hidden layer, and an output layer connected in sequence; wherein, the first and second hidden layers use ReLU as the activation function, and the output layer uses the hyperbolic tangent function as the activation function; The second candidate Actor network is a four-layer fully connected neural network, consisting of an input layer, a third hidden layer, a fourth hidden layer, a fifth hidden layer, and an output layer connected in sequence; wherein, the third hidden layer, the fourth hidden layer, and the fifth hidden layer use ReLU as the activation function, and the output layer uses the hyperbolic tangent function as the activation function; The third alternative Actor network is a hybrid network of convolutional and fully connected layers, comprising a first convolutional layer, a second convolutional layer, and a fully connected layer connected in sequence; wherein the first and second convolutional layers use ReLU as the activation function, and the fully connected layer uses the hyperbolic tangent function as the activation function. 7.The real-time optimal operation method of the e-h hybrid island microgrid of claim 5, wherein, The training process for initial training, pre-training, and formal training is the same, but their termination conditions are different. The termination condition for initial training is that the amount of experience in the experience replay buffer reaches the set batch size. The termination condition for pre-training is that the number of training rounds reaches the set threshold for the number of training rounds in pre-training. The termination condition for formal training is that the number of training rounds reaches the set threshold for the number of training rounds in formal training.

Images