Multi-energy storage collaborative control method based on bi-lstm and ppo machine learning

Abstract

The application discloses a multi-energy storage collaborative control method based on BiLSTM and PPO machine learning, which comprises the following steps: S1, establishing an information physical system model of an energy storage system in an island microgrid, including a physical layer, a communication layer and an information layer; S2, in the information layer, constructing a battery health state prediction model based on a bidirectional long short-term memory network, which is used for real-time prediction of the health state of an energy storage battery; S3, in the information layer, constructing a collaborative control model of the energy storage system based on a proximal policy optimization algorithm (PPO), based on the output of the constructed battery health state prediction model, adjusting the decision-making process of the collaborative control model of the energy storage system, and generating power distribution instructions of the energy storage battery and a super capacitor; and S4, issuing the power distribution instructions to the physical layer through the communication layer, and controlling the power output of the energy storage battery and the super capacitor. The method can accurately predict the battery health state and optimize the power distribution of the energy storage system according to the battery health state, thereby prolonging the service life of the energy storage system.

Description

Technical Field

[0001] This invention relates to the field of energy storage control technology for islanded microgrids, specifically to a multi-energy storage collaborative control method based on BiLSTM and PPO machine learning. Background Technology

[0002] An islanded microgrid is a small-scale power generation and distribution system capable of operating independently, typically comprising distributed power sources, energy storage systems, and loads. In an islanded microgrid, energy storage systems play a crucial role in balancing power, stabilizing voltage, and frequency. Commonly used energy storage devices include lithium-ion batteries and supercapacitors. Lithium-ion batteries have high energy density, making them suitable for providing continuous power, but their cycle life is significantly affected by charge-discharge strategies. Supercapacitors, on the other hand, have high power density and long cycle life, making them suitable for handling transient power fluctuations.

[0003] To extend the lifespan of isolated microgrid energy storage systems, especially for expensive lithium batteries, it is necessary to optimize the charging and discharging strategies of these systems. Traditional control methods are often based on fixed rules or simple filtering allocation, neglecting battery health and failing to adapt to complex operating environments. In recent years, artificial intelligence technology has provided new insights into the optimized control of energy storage systems.

[0004] Long Short-Term Memory (LSTM) networks and their variant Bidirectional Long Short-Term Memory (BiLSTM) in deep learning can effectively process time-series data and are often used for battery state of health (SOH) prediction. Proximal Policy Optimization (PPO) algorithms in reinforcement learning can learn optimal control strategies through interaction with the environment. However, there is currently no research combining BiLSTM and PPO for extending the lifetime of energy storage systems in islanded microgrids. Summary of the Invention

[0005] To address the problem that existing energy storage system control methods do not consider battery health status and cannot adaptively optimize, this invention provides a multi-energy storage collaborative control method based on BiLSTM and PPO machine learning. This method accurately predicts battery health status and optimizes the power allocation of the energy storage system accordingly, thereby extending the lifespan of the energy storage system.

[0006] The technical solution adopted in this invention is as follows:

[0007] A multi-energy storage collaborative control method based on BiLSTM and PPO machine learning includes the following steps: S1: Establish a cyber-physical system model for energy storage systems in isolated microgrids, including the physical layer, communication layer, and information layer; S2: In the information layer, a battery health status prediction model based on a bidirectional long short-term memory network is constructed to predict the health status of energy storage batteries in real time. S3: In the information layer, a collaborative control model for the energy storage system based on the near-end policy optimization algorithm (PPO) is constructed. Based on the output of the constructed battery health state prediction model, the decision-making process of the collaborative control model for the energy storage system is adjusted to generate power allocation instructions for the energy storage battery and supercapacitor. S4: Sends power distribution commands to the physical layer through the communication layer to control the power output of the energy storage battery and supercapacitor.

[0008] In S1, the physical layer includes a lithium battery pack, a supercapacitor pack, a bidirectional DC / DC converter, a DC / AC inverter, current / voltage / temperature sensors, and a local controller. As the physical entity of the system, it is responsible for the storage and release of electrical energy, executes power scheduling commands from the information layer, and collects and uploads operating data such as voltage, current, temperature, and state of charge (SOC) of the energy storage unit to the communication layer in real time. The communication layer aggregates the sensor data collected by the physical layer and sends it to the information layer, and sends the power allocation command generated by the information layer to the local controller of the physical layer. The information layer cleans and normalizes the historical operational data uploaded from the communication layer to form a dataset. Using a trained bidirectional long short-term memory network, it predicts the health status of the lithium battery and supercapacitor in real time, providing key state variables for optimized control. Based on the current state space and power demand, the optimal action space, i.e., the energy storage power allocation command, is calculated through a near-end strategy optimization algorithm.

[0009] In step S2, the battery health state prediction model takes the historical operating data sequence of the energy storage battery as input, including voltage / current statistics, temperature gradient, cycle count, and charge / discharge efficiency; the output is the predicted SOH value of the energy storage battery; specifically, it includes the following steps: S2.1: Collect historical operating data of energy storage batteries and supercapacitors. The historical operating data includes battery voltage, current, temperature, number of cycles and charge / discharge efficiency. Organize the collected data into a dataset. S2.2: The battery health status prediction model adopts BiLSTM, and its network structure includes an input layer, a bidirectional LSTM layer, a Dropout layer, a fully connected layer, and an output layer. S2.3: Input the dataset prepared in S2.1 into BiLSTM to train BiLSTM. The trained BiLSTM can then be used to predict the SOH value of the energy storage battery in real time.

[0010] In S3, the state space of the energy storage system collaborative control model includes the voltage, current, temperature, state of charge, and health of the energy storage battery and supercapacitor, as well as the power demand of the microgrid. The state space of the energy storage system collaborative control model includes the power allocation ratio of the energy storage battery and the supercapacitor. Design a composite reward function, which includes a reward item based on health status, a reward item based on system efficiency, and a penalty item based on security constraints; Specifically, it includes: 3.1: Define the state space of the energy storage system cooperative control model at time t. for: (1); In equation (1), , Let represent the terminal voltages of the lithium battery and the supercapacitor at time t, respectively. , These represent the output currents of the lithium battery and the supercapacitor at time t, respectively. , These represent the internal temperatures of the lithium battery and the supercapacitor at time t, respectively. , These represent the state of charge of the lithium battery and the supercapacitor at time t, respectively. , These represent the health status of the lithium battery and the supercapacitor at time t, respectively. This represents the total power demand of the microgrid at time t; This represents the frequency component dominated by power demand at time t; Indicates the current time step; 3.2: Define the action space of the energy storage system collaborative control model for: (2); In equation (2), This indicates the power command issued to the lithium battery at time t; This indicates the power command issued to the supercapacitor at time t.

[0011] Satisfying power balance: (3); 3.3: The advantage function estimation in the Proximity Policy Optimization (PPO) algorithm adopts the generalized advantage estimation method: (4); In equation (4), This represents the estimated advantage function value of the action at time t; This is a discount factor, ranging from 0 to 1, representing the future reward decay rate; The adjustment parameter for generalized advantage estimation, with a value ranging from 0 to 1; The time difference error is defined by equation (5); This represents the total time steps of a complete training round; Indicates the current time step; Indicates the number of training sessions.

[0012] (5); In formula (5): This refers to the time difference error; For instant rewards, The state value function output by the critic network; and Let represent the state spaces at time t and time t+1, respectively.

[0013] 3.4: The policy update mechanism of the Proximal Policy Optimization (PPO) algorithm is implemented through two parts: the policy optimization pruning objective function and the value function loss. The pruning objective function is expressed as: (6); In formula (6): This represents the pruning loss function of the policy network; Let represent the probability ratio, where For the new strategy, For the old strategy, This is the pruning function, typically taking a value between 0.1 and 0.3, used to limit the magnitude of policy updates; This represents the expectation for time step t; This represents the estimated value of the advantage function; This represents the clipping function, which compares the probability to... Limited to the interval [ ， ]Inside.

[0014] The value function loss can be expressed by the following equation (7): (7); In equation (7): This represents the average error loss of the critic network; This represents the state value function output by the commentator network. ,in, Let represent the discount factor γ raised to the power of l, used to calculate the decay weight of the reward in the l-th step in the future; This represents the target value for accumulating discount rewards starting from time t.

[0015] 3.5: The output port of the Proximal Policy Optimization (PPO) algorithm integrates two parts: a safety constraint and a power adaptive allocation strategy. The action pruning for the safety constraint can be expressed as: (8); In equation (8): This represents the action vector after safe clipping; , Let represent the minimum and maximum allowable power of each energy storage unit at time t, respectively; The frequency-based adaptive power allocation strategy can be expressed by the following equation: (9); (10); In the above formula, This is the low-frequency threshold, typically ranging from 0.1Hz to 1Hz. This is a high-frequency threshold, typically ranging from 10Hz to 100Hz; The adaptive allocation coefficients are calculated in real time by the policy network based on the battery health status, and the values ​​range from 0 to 1.

[0016] In S3, the composite reward function Designed as follows: (11); In equation (11): Rewards are based on battery health status; Rewards are based on the system's operational status; Penalties for violating safety regulations; , , These are the weighting coefficients.

[0017] In step S4, the decision-making process for adjusting the collaborative control model of the energy storage system includes: The weight of the health status reward item in the reward function is dynamically adjusted based on the predicted battery health status value, and the power allocation ratio of the energy storage battery is reduced when the health status is below the threshold.

[0018] This invention discloses a multi-energy storage collaborative control method based on BiLSTM and PPO machine learning, with the following technical effects: 1) This invention enables real-time prediction of battery health status; 2) This invention is based on a deep reinforcement learning algorithm, which solves the problem of multi-objective trade-off optimization between lifetime and efficiency, and improves the economy of microgrid systems; 3) This invention improves system security by embedding safety action clipping and safety penalty items into the PPO algorithm; 4) This invention breaks away from the limitations of traditional optimization control methods that rely on precise mathematical models and manual parameter adjustment, and can automatically adapt to different operating conditions, thereby improving the flexibility of the microgrid. Attached Figure Description

[0019] The present invention will be further described below with reference to the accompanying drawings and examples; Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is the overall control block diagram.

[0021] Figure 3 This is a schematic diagram illustrating the implementation principle of an LSTM neural network.

[0022] Figure 4 This is a block diagram of the BiLSTM neural network structure.

[0023] Figure 5 This is a schematic diagram illustrating the implementation principle of the PPO algorithm. Detailed Implementation

[0024] The multi-energy storage collaborative control algorithm based on long short-term memory network and proximal policy optimization machine learning is as follows: (1) Collect historical data on lithium batteries and supercapacitors; (2) To address the problem that the state of health (SOH) value of energy storage components is difficult to calculate accurately in real time, a long short-term memory network (BiLSTM) is designed and trained to predict the SOH value of energy storage components based on the regularity of battery historical data; (3) Design a near-end strategy optimization algorithm (PPO) to achieve coordinated control of lithium batteries and supercapacitors at different time scales based on the SOH value predicted by BiLSTM. (4) The control commands finally generated by the PPO algorithm are sent to the energy storage system. The algorithm proposed in this invention can maximize economic benefits and smooth power fluctuations while meeting the requirements of grid dispatch. At the same time, it can extend the life of energy storage components through strategy adjustment, which is of great significance for improving the economy of microgrids and promoting the grid connection of high proportion of new energy sources.

[0025] like Figure 1 As shown, this invention discloses a multi-energy storage cooperative control algorithm based on BiLSTM+PPO machine learning, the specific steps of which are as follows: Step 1: To address the issues of short lifespan and poor economic efficiency of energy storage systems in microgrids, a multi-energy storage collaborative control algorithm is designed, incorporating bidirectional long short-term memory network prediction and near-end strategy optimization; for example... Figure 2 As shown, historical operating data is used to train a BiLSTM, which is then used to predict lithium battery performance in real time. With supercapacitors The PPO then integrates the State of Harmony (SOH) values ​​of both energy storage systems with the power demand on the load side to generate an energy storage power output command, which is fed forward to the energy storage side. (This refers to the photovoltaic side of the microgrid system.) Through the maximum power point tracking algorithm Drive the load side together with the energy storage side Work.

[0026] The detailed structural diagram of the Long Short-Term Memory (LSTM) network is as follows: Figure 3 As shown, the specific steps include the following: 1) Collect historical operating data of energy storage modules, preprocess the historical operating dataset, and filter the influencing features that are strongly correlated with the predicted SOH value, such as battery voltage, current, temperature, cycle number and charge / discharge efficiency, remove outliers and fill in missing values.

[0027] 2) Reduce the voltage, current and other features to the same dimension (0~1) to simplify the computational complexity. Divide the data into time windows, and let every ten consecutive time steps be a sample. The input features of each sample are the multivariate time series data within the window, which are combined into a vector form.

[0028] 3) Construct an LSTM prediction model: input layer, LSTM layer, Dropout layer, fully connected layer, and output layer. The input layer accepts time-series data in the form of (time step, number of features); each LSTM unit in the LSTM layer regulates the information flow through forget gates, input gates, and output gates, memorizing and constructing a dynamic model of SOH decay; the fully connected layer maps the hidden states output by the LSTM to the predicted SOH value; the Dropout layer avoids overfitting of the neural network by randomly discarding neurons. The output layer is a single-neuron output corresponding to the SOH percentage.

[0029] 4) Finally, the data is divided into training set, validation set and test set and input into the LSTM model to be trained. The training set is used to iteratively update the weights, the validation set is used to monitor whether overfitting occurs, and the test set is used to verify the model performance with three indicators: root mean square error, mean absolute percentage error and coefficient of determination, i.e. whether it can accurately predict the SOH value.

[0030] Step 2: Couple the LSTM units together to obtain a bidirectional long short-term memory network, such as... Figure 4 As shown, the specific implementation process is as follows: 1) Processing the forward LSTM: Input is time-series data from beginning to end, processing direction is from the beginning to the end of the sequence, and the forward hidden state at each time step. It contains historical information up to the current moment; 2) Processing backward LSTM: Input time-series data from back to front, processing direction from end of sequence to beginning, backward hidden state at each time step. It contains information about the future, from the past to the present; 3) At each time step, concatenate the forward and backward hidden states: ; After fusion It also includes contextual information from the past and future, providing richer feature representations.

[0031] Step 3: Use the predicted SOH value of the energy storage system and the power demand of the load as inputs to the PPO algorithm, and output the energy storage power output command, such as... Figure 5 As shown, the specific implementation process includes the following steps: 1) Define the state space and action space, where the state space consists of the input lithium battery and supercapacitor parameters and the load power requirement: (2); The action space primarily stores the energy storage output commands for the lithium battery and supercapacitor, respectively. ; 2) Design a composite reward function for the PPO algorithm. The reward function is designed by integrating the health of the energy storage side, the power demand of the load side, and the system security. ; By designing the weighting coefficients of the three factors, the aim is to penalize high-power charging and discharging when the State of Harmony (SOH) is low, thereby ensuring the load power requirements and system safety and reliability.

[0032] 3): Construct the policy network (Actor) and value network (Critic). Initialize the environment and obtain the initial state. Select actions using the current Actor network. Performing this action will earn you a reward. With the new state ,storage To the experience pool; use the Critic network to calculate the state value, and the time difference error is obtained by the following formula: ; The generalized dominance estimation method is used to obtain the estimate of the dominance function at the current time step. The generalized dominance estimation method is expressed by the following equation: (4); 4) Calculate the probability ratio between the old and new strategies: ; The policy update mechanism of the PPO algorithm is implemented through two parts: a policy optimization pruning objective function and a value function loss. The pruning objective function is expressed as: (6); The value function loss can be expressed by the following formula: (7); 5) The output port of the PPO algorithm is integrated from two parts: safety constraints and adaptive power allocation strategy. The action pruning of the safety constraints can be expressed as: (8); The frequency-based adaptive power allocation strategy can be expressed by the following equation: (9); (10).

[0033] To verify the feasibility of the proposed method, this invention compares the proposed method with the traditional droop control algorithm, and compares the performance indicators of islanded microgrids under the two methods. The results are shown in Table 1 below.

[0034]

[0035] As shown in Table 1, while maintaining a 100% frequency qualification rate, the SOH (State of Harm) degradation of the lithium battery was reduced by approximately 11.4%. This effectively delayed battery aging while ensuring system frequency stability, verifying the superiority of the strategy in multi-objective collaborative optimization. Under the premise of allowing for small power balance errors, comprehensive optimization of energy storage life and system operating efficiency was achieved, demonstrating the adaptive adjustment capability of the data-driven control strategy under complex operating conditions.

[0036] In summary, the hybrid energy storage coordinated control strategy based on SOH prediction and near-end strategy optimization proposed in this invention effectively solves the problem of coordinating the optimization of energy storage lifetime and system stability in islanded microgrids through the intelligent control architecture of BiLSTM+PPO algorithm.

Claims

1. A multi-energy storage cooperative control method based on BiLSTM and PPO machine learning, characterized in that... Includes the following steps: S1: Establish a cyber-physical system model for energy storage systems in isolated microgrids, including the physical layer, communication layer, and information layer; S2: In the information layer, a battery health status prediction model based on a bidirectional long short-term memory network is constructed to predict the health status of energy storage batteries in real time. S3: In the information layer, a collaborative control model for the energy storage system based on the near-end policy optimization algorithm (PPO) is constructed. Based on the output of the constructed battery health state prediction model, the decision-making process of the collaborative control model for the energy storage system is adjusted to generate power allocation instructions for the energy storage battery and supercapacitor. S4: Sends power distribution commands to the physical layer through the communication layer to control the power output of the energy storage battery and supercapacitor.

2. The multi-energy storage collaborative control method based on BiLSTM and PPO machine learning according to claim 1, characterized in that: In S1, the physical layer includes a lithium battery pack, a supercapacitor pack, a bidirectional DC / DC converter, a DC / AC inverter, current / voltage / temperature sensors, and a local controller. As the physical entity of the system, it is responsible for the storage and release of electrical energy, executes power scheduling commands from the information layer, and collects and uploads operating data such as voltage, current, temperature, and state of charge (SOC) of the energy storage unit to the communication layer in real time.

3. The multi-energy storage collaborative control method based on BiLSTM and PPO machine learning according to claim 2, characterized in that: The communication layer aggregates the sensor data collected by the physical layer and sends it to the information layer, and sends the power allocation command generated by the information layer to the local controller of the physical layer.

4. The multi-energy storage collaborative control method based on BiLSTM and PPO machine learning according to claim 3, characterized in that: The information layer cleans and normalizes the historical operation data uploaded by the communication layer to form a dataset. Using a trained bidirectional long short-term memory network, it predicts the health status of lithium batteries and supercapacitors in real time. Based on the current state space and power demand, it calculates the optimal action space, i.e., the energy storage power allocation command, through a near-end strategy optimization algorithm.

5. The multi-energy storage collaborative control method based on BiLSTM and PPO machine learning according to claim 4, characterized in that: In step S2, the battery health state prediction model takes the historical operating data sequence of the energy storage battery as input, including voltage / current statistics, temperature gradient, number of cycles, and charge / discharge efficiency; the output is the predicted SOH value of the energy storage battery.

6. The multi-energy storage collaborative control method based on BiLSTM and PPO machine learning according to claim 5, characterized in that: S2 includes the following steps: S2.1: Collect historical operating data of energy storage batteries and supercapacitors. The historical operating data includes battery voltage, current, temperature, number of cycles and charge / discharge efficiency. Organize the collected data into a dataset. S2.2: The battery health status prediction model adopts BiLSTM, and its network structure includes an input layer, a bidirectional LSTM layer, a Dropout layer, a fully connected layer, and an output layer. S2.3: Input the dataset prepared in S2.1 into BiLSTM to train BiLSTM. The trained BiLSTM can then be used to predict the SOH value of the energy storage battery in real time.

7. The multi-energy storage cooperative control method based on BiLSTM and PPO machine learning according to claim 6, characterized in that: In S3, the state space of the energy storage system collaborative control model includes the voltage, current, temperature, state of charge, and health of the energy storage battery and supercapacitor, as well as the power demand of the microgrid. The state space of the energy storage system collaborative control model includes the power allocation ratio of the energy storage battery and the supercapacitor. Design a composite reward function, which includes a reward based on health status, a reward based on system efficiency, and a penalty based on security constraints.

8. The multi-energy storage cooperative control method based on BiLSTM and PPO machine learning according to claim 7, characterized in that: S3 specifically includes: 3.1: Define the state space of the energy storage system cooperative control model at time t. for: （1）； In equation (1), , Let represent the terminal voltages of the lithium battery and the supercapacitor at time t, respectively. , These represent the output currents of the lithium battery and the supercapacitor at time t, respectively. , These represent the internal temperatures of the lithium battery and the supercapacitor at time t, respectively. , These represent the state of charge of the lithium battery and the supercapacitor at time t, respectively. , These represent the health status of the lithium battery and the supercapacitor at time t, respectively. This represents the total power demand of the microgrid at time t; This represents the frequency component dominated by power demand at time t; Indicates the current time step; 3.2: Define the action space of the energy storage system collaborative control model for: （2）； In equation (2), This indicates the power command issued to the lithium battery at time t; This indicates the power command issued to the supercapacitor at time t; Satisfying power balance: （3）； 3.3: The advantage function estimation in the Proximity Policy Optimization (PPO) algorithm adopts the generalized advantage estimation method: （4）； In equation (4), This represents the estimated advantage function value of the action at time t; This is a discount factor, ranging from 0 to 1, representing the future reward decay rate; The adjustment parameter for generalized advantage estimation, with a value ranging from 0 to 1; The time difference error is defined by equation (5); This represents the total time steps of a complete training round; Indicates the current time step; Indicates the number of training sessions; （5）； In equation (5): This refers to the time difference error; For instant rewards, The state value function output by the critic network; and Let represent the state spaces at time t and t+1, respectively; 3.4: The policy update mechanism of the Proximal Policy Optimization (PPO) algorithm is implemented through two parts: the policy optimization pruning objective function and the value function loss. The pruning objective function is expressed as: （6）； In formula (6): This represents the pruning loss function of the policy network; Let represent the probability ratio, where For the new strategy, For the old strategy, This is a pruning function used to limit the magnitude of policy updates; This represents the expectation for time step t; This represents the estimated value of the advantage function; This represents the clipping function, which compares the probability to... Limited to the interval [ ， ]Inside; The value function loss is expressed by the following equation (7): （7）； In equation (7): This represents the average error loss of the critic network; This represents the state-value function output by the critic network; ,in, Let represent the discount factor γ raised to the power of l, used to calculate the decay weight of the reward in the l-th step in the future; This represents the target value for accumulating discount rewards starting from time t; 3.5: The output port of the Proximal Policy Optimization (PPO) algorithm integrates two parts: a safety constraint and a power adaptive allocation strategy. The action pruning for the safety constraint is expressed as: （8）； In equation (8): This represents the action vector after safe clipping; , Let represent the minimum and maximum allowable power of each energy storage unit at time t, respectively; The frequency-based adaptive power allocation strategy is expressed by the following equation: （9）； （10）； In the above formula, This is the low-frequency threshold, typically ranging from 0.1Hz to 1Hz. High-frequency threshold; The adaptive allocation coefficients are calculated in real time by the policy network based on the battery health status, and the values ​​range from 0 to 1.

9. The multi-energy storage cooperative control method based on BiLSTM and PPO machine learning according to claim 8, characterized in that: In S3, the composite reward function Designed as follows: （11）； In equation (11): Rewards are based on battery health status; Rewards are based on the system's operational status; Penalties for violating safety regulations; , , These are the weighting coefficients.

10. The multi-energy storage cooperative control method based on BiLSTM and PPO machine learning according to claim 8, characterized in that: In S4, the decision-making process for adjusting the collaborative control model of the energy storage system includes: dynamically adjusting the weight of the health status reward item in the reward function based on the predicted value of the battery health status, and reducing the power allocation ratio of the energy storage battery when the health status is below the threshold.

Images