Network-configuration type hybrid energy storage collaborative control method based on deep reinforcement learning and MPC

Abstract

The application discloses a network-constructing type hybrid energy storage collaborative control method based on deep reinforcement learning and MPC, constructs a network-constructing type hybrid energy storage system, builds a deep reinforcement learning intelligent agent, takes the real-time operation state of the power grid and the operation state of each unit of the hybrid energy storage system as input, and trains the intelligent agent with the optimal control target of the network-constructing type hybrid energy storage system as a reward function; a controller is designed, the controller is based on optimized controller parameters and a multi-objective optimization objective function, collects real-time operation data of the power grid and operation data of each unit of the network-constructing type hybrid energy storage system, inputs the data into the trained intelligent agent, obtains the optimized controller parameters, and performs real-time power distribution and charge-discharge control on the lithium iron phosphate battery energy storage unit and the super capacitor energy storage unit of the network-constructing type hybrid energy storage system, so that the collaborative operation of the network-constructing type hybrid energy storage system and the power grid is realized. The safe operation of the energy storage unit can be ensured, the service life is prolonged, and the operation and maintenance cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of agricultural information processing technology, and in particular to a network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC. Background Technology

[0002] Hybrid energy storage is gradually replacing single energy storage systems due to its ability to balance energy storage capacity and power response speed. However, several bottlenecks remain in practical applications. Existing hybrid energy storage systems often combine lithium iron phosphate batteries with supercapacitors, but the lack of scientific collaborative control strategies prevents these two components from fully realizing their advantages, frequently resulting in issues such as unreasonable power allocation and response lag. Traditional control methods rely on precise mathematical models and fixed parameters for the MPC controller, making it difficult to adapt to the complex and ever-changing operating scenarios of the power grid. Meanwhile, single DRL control lacks stability and is prone to decision-making biases. Furthermore, existing systems suffer from low data processing accuracy and poor agent adaptability, failing to achieve precise coordination between energy storage units and the power grid. This not only affects the stability of grid operation but also exacerbates energy storage unit losses and shortens their lifespan, making it difficult to meet the high standards required for grid-connected operation. Summary of the Invention

[0003] The purpose of this invention is to solve the above problems by designing a network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC.

[0004] To achieve the above objectives, the technical solution of the present invention further includes the following steps in the above-mentioned network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC: Construct a grid-type hybrid energy storage system, wherein the hybrid energy storage system includes at least a lithium iron phosphate battery energy storage unit and a supercapacitor energy storage unit; A deep reinforcement learning (DRL) agent is constructed, with the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and the optimal control objective of the grid-type hybrid energy storage system as the reward function for training the DRL agent. Design an MPC controller, based on a mathematical model of a hybrid energy storage system, and construct a multi-objective optimization objective function by combining the optimized MPC controller parameters output by the DRL agent; Real-time power grid operation data and operation data of each unit in the grid-type hybrid energy storage system are collected and input into the trained DRL agent to obtain optimized MPC controller parameters; The MPC controller, based on optimized MPC controller parameters and multi-objective optimization objective function, performs real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage unit and supercapacitor energy storage unit of the grid-type hybrid energy storage system, thereby realizing the coordinated operation of the grid-type hybrid energy storage system and the power grid.

[0005] Furthermore, in the aforementioned collaborative control method for network-based hybrid energy storage based on deep reinforcement learning and MPC, the construction of the network-based hybrid energy storage system includes at least lithium iron phosphate battery energy storage units and supercapacitor energy storage units, comprising: Among them, the lithium iron phosphate battery energy storage unit is used to provide continuous energy support, and the supercapacitor energy storage unit is used to achieve millisecond-level fast power response, adapting to the primary frequency regulation scenario of the power grid. The system adopts a centralized DC bus topology. The lithium iron phosphate battery units are connected to the DC bus through a bidirectional DC / DC converter, and the supercapacitor units are connected in parallel to the same DC bus through a bidirectional DC / DC converter. The DC bus is connected to the AC grid through a grid-connected inverter, realizing bidirectional AC-DC energy conversion. Configure an EMS energy management system to coordinate the operating status of the two energy storage units, collect and upload system operation data, and receive upper-level control commands.

[0006] Furthermore, in the aforementioned collaborative control method for grid-based hybrid energy storage based on deep reinforcement learning and MPC, the construction of the deep reinforcement learning DRL agent, using the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and training the DRL agent with the optimal control objective of the grid-based hybrid energy storage system as the reward function, includes: An intelligent agent is built using the PPO reinforcement learning algorithm. The state input layer receives multi-dimensional operational data, the feature extraction layer uses a CNN convolutional neural network to extract key features from the data, and the decision output layer outputs the optimized key parameters of the MPC controller. The input parameters include the real-time operating status of the power grid and the operating status of each unit in the hybrid energy storage system. The real-time operating status of the power grid includes the grid frequency, bus voltage, active power, reactive power and load fluctuation. The operating status of the hybrid energy storage system includes the SOC, charging and discharging power and battery temperature of the lithium iron phosphate battery, the SOC, terminal voltage and charging and discharging power of the supercapacitor, and the operating status of each converter.

[0007] Furthermore, in the aforementioned collaborative control method for grid-based hybrid energy storage based on deep reinforcement learning and MPC, the construction of the deep reinforcement learning DRL agent, using the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and training the DRL agent with the optimal control objective of the grid-based hybrid energy storage system as the reward function, includes: With the optimal control objective of the grid-type hybrid energy storage system as the core, a multi-objective weighted reward function is constructed, with the expression R=ω1×R1+ω2×R2+ω3×R3+ω4×R4, where ω1-ω4 are weight coefficients that are dynamically adjusted according to the control priority; R1 is the grid frequency stability reward, R2 is the energy storage unit SOC maintenance reward, R3 is the system charging and discharging loss reward, and R4 is the system operation stability reward.

[0008] Furthermore, in the aforementioned network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC, the design of the MPC controller involves constructing a multi-objective optimization objective function based on the mathematical model of the hybrid energy storage system and the optimized MPC controller parameters output by the DRL agent, including: The MPC controller includes a prediction module, an optimization module, and an execution module. The prediction module is based on a mathematical model of a hybrid energy storage system and combines real-time grid operation data and the status of energy storage units to predict the system operation status at Np future moments. The optimization module solves for the optimal control sequence based on a multi-objective optimization objective function and the optimization parameters output by the DRL agent; the execution module applies the optimized control sequence to the converters of each energy storage unit to achieve charge and discharge control.

[0009] Furthermore, in the aforementioned collaborative control method for grid-based hybrid energy storage based on deep reinforcement learning and MPC, the process of collecting real-time grid operation data and operation data of each unit in the grid-based hybrid energy storage system, and inputting them into the trained DRL agent to obtain optimized MPC controller parameters, includes: Collect grid frequency, voltage, active power and reactive power parameters, and sequentially perform outlier deletion, missing value supplementation and normalization on the collected data to obtain preprocessed parameters; The preprocessed real-time data is input into the trained DRL agent, which then extracts features and makes policy decisions to output optimized MPC controller parameters.

[0010] Furthermore, in the aforementioned collaborative control method for grid-connected hybrid energy storage based on deep reinforcement learning and MPC, the MPC controller performs real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage unit and supercapacitor energy storage unit of the grid-connected hybrid energy storage system based on optimizing MPC controller parameters and a multi-objective optimization objective function, thereby realizing the collaborative operation of the grid-connected hybrid energy storage system with the power grid, including: The MPC controller calculates the optimal charging and discharging power allocation scheme for the next Nc time points based on the optimized parameters and prediction model. When the power grid experiences sudden high-frequency load changes, the supercapacitor responds quickly and outputs power to smooth out high-frequency fluctuations; when the power grid experiences low-frequency load changes, the lithium iron phosphate battery slowly adjusts its charging and discharging power to maintain the system's energy balance.

[0011] Furthermore, a system for implementing a network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC is provided, the system comprising the following modules: An energy storage system establishment module is used to construct a grid-type hybrid energy storage system, wherein the hybrid energy storage system includes at least a lithium iron phosphate battery energy storage unit and a supercapacitor energy storage unit. The DRL agent training module is used to build a deep reinforcement learning DRL agent. The DRL agent is trained with the real-time operation status of the power grid and the operation status of each unit of the hybrid energy storage system as inputs, and the optimal control objective of the grid-type hybrid energy storage system as the reward function. The MPC controller design module is used to design MPC controllers. Based on the mathematical model of the hybrid energy storage system, it constructs a multi-objective optimization objective function by combining the optimized MPC controller parameters output by the DRL agent. The controller parameter optimization module is used to collect real-time power grid operation data and operation data of each unit of the grid-type hybrid energy storage system, and input them into the trained DRL agent to obtain optimized MPC controller parameters. The collaborative operation control module is used by the MPC controller to perform real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage units and supercapacitor energy storage units of the grid-type hybrid energy storage system based on optimized MPC controller parameters and multi-objective optimization objective function, so as to realize the collaborative operation of the grid-type hybrid energy storage system with the power grid.

[0012] Furthermore, a system for implementing a network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC is provided, wherein the controller parameter optimization module includes the following sub-modules: The data acquisition submodule is used to acquire grid frequency, voltage, active power and reactive power parameters. The acquired data is then processed by deleting outliers, filling in missing values ​​and normalizing the data to obtain preprocessed parameters. The output submodule is used to input the preprocessed real-time data into the trained DRL agent. The agent then outputs the optimized MPC controller parameters through feature extraction and policy decision-making.

[0013] Furthermore, a system for implementing a network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC is provided, wherein the collaborative operation control module includes the following sub-modules: The computational sub-MPC controller calculates the optimal charging and discharging power allocation scheme for the next Nc time steps based on the optimized parameters and prediction model. When the power grid experiences high-frequency load changes, the supercapacitor responds quickly and outputs power to smooth out high-frequency fluctuations; when the power grid experiences low-frequency load changes, the lithium iron phosphate battery slowly adjusts the charging and discharging power to maintain the system's energy balance.

[0014] Its beneficial effects lie in the fact that by adopting a hybrid architecture of lithium iron phosphate batteries and supercapacitors, coupled with a scientific power allocation strategy, it effectively mitigates high- and low-frequency load fluctuations in the power grid and improves grid operational stability, balancing energy storage and rapid response requirements. The DRL agent can dynamically optimize MPC controller parameters, overcoming the shortcomings of traditional MPC parameters being fixed and lacking adaptability. It can handle complex grid scenarios without relying on precise mathematical models, improving control flexibility and accuracy. Real-time data acquisition and preprocessing, along with an online agent update mechanism, ensure the real-time nature and accuracy of control decisions, reducing system charging and discharging losses. A robust fault emergency handling and feedback adjustment mechanism ensures the safe operation of the energy storage unit, extends its service life, and reduces operation and maintenance costs. This method achieves efficient collaboration between the hybrid energy storage system and the power grid, significantly improving the control performance of grid-connected energy storage. Attached Figure Description

[0015] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention.

[0016] Figure 1 This is a schematic diagram of the first embodiment of the network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC in this invention. Figure 2 This is a schematic diagram of the second embodiment of the network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC in this invention. Figure 3 This is a schematic diagram of the first embodiment of the network-based hybrid energy storage collaborative control system based on deep reinforcement learning and MPC in this invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0018] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms "one," "an," and "this" used herein may also include the plural forms. It should be further understood that the terminology used in this specification includes the presence of features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1As shown, a network-based hybrid energy storage collaborative control method based on deep reinforcement learning and MPC is presented. This network-based hybrid energy storage collaborative control method includes the following steps: Step 101: Construct a grid-type hybrid energy storage system, which includes at least lithium iron phosphate battery energy storage units and supercapacitor energy storage units; Specifically, in this embodiment, the lithium iron phosphate battery energy storage unit is used to provide continuous energy support, and the supercapacitor energy storage unit is used to achieve millisecond-level fast power response, adapting to the primary frequency regulation scenario of the power grid; a centralized DC bus topology is adopted, with the lithium iron phosphate battery unit connected to the DC bus through a bidirectional DC / DC converter, and the supercapacitor unit connected in parallel to the same DC bus through a bidirectional DC / DC converter. The DC bus is connected to the AC grid through a grid-connected inverter, realizing bidirectional AC-DC energy conversion; an EMS energy management system is configured to coordinate the operating status of the two energy storage units, collect and upload system operating data, and receive upper-level control commands.

[0020] The lithium iron phosphate battery energy storage unit uses high cycle life and high energy density models. The rated capacity is determined according to the grid load fluctuation characteristics and energy storage requirements, and 100-500kWh is recommended. The rated voltage is 380V / 750V, the charge / discharge rate is controlled at 0.5C-2C, the cycle life is not less than 3000 cycles, and the operating temperature range is -20℃~60℃. It is equipped with a battery management system (BMS) to monitor the voltage, current, temperature, and SOC, State of Charge, and charging status of individual cells in real time, and to achieve overcharge, over-discharge, and over-temperature protection. The supercapacitor energy storage unit uses double-layer supercapacitors with a rated capacity of 50-200F. The rated voltage is matched with the lithium battery unit. The charge / discharge rate can reach 10C-50C, the response time is ≤10ms, and the cycle life is not less than 100,000 cycles. It is mainly used to smooth high-frequency power fluctuations in the grid and make up for the power response lag of lithium batteries.

[0021] Employing a centralized DC bus topology, lithium iron phosphate battery units are connected to the DC bus via bidirectional DC / DC converters, while supercapacitor units are connected in parallel to the same DC bus via bidirectional DC / DC converters. The DC bus is then connected to the AC grid via a grid-connected inverter, achieving bidirectional AC / DC energy conversion. An energy management system (EMS) is also configured to coordinate the operational status of the two energy storage units, collect and upload system operation data, receive upper-level control commands, and ensure stable system operation.

[0022] Each energy storage unit, converter, BMS, and EMS communicates via CAN bus or Ethernet with a communication rate of no less than 1Mbps to ensure real-time transmission of operating data with a delay of ≤50ms. The system establishes a communication link with the power grid dispatch center to support bidirectional interaction of real-time power grid operating status and control commands, meeting the requirements of grid-based operation for dispatch response.

[0023] Step 102: Build a deep reinforcement learning (DRL) agent, using the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and use the optimal control objective of the grid-type hybrid energy storage system as the reward function to train the DRL agent; Specifically, in this embodiment, the PPO reinforcement learning algorithm is used to build an intelligent agent. The state input layer receives multi-dimensional operating data, the feature extraction layer uses a CNN convolutional neural network to extract key features from the data, and the decision output layer outputs optimized key parameters of the MPC controller. The input parameters include the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system. The real-time operating status of the power grid includes the grid frequency, bus voltage, grid active power, reactive power, and load fluctuation. The operating status of the hybrid energy storage system includes the SOC, charging and discharging power, and battery temperature of the lithium iron phosphate battery, the SOC, terminal voltage, and charging and discharging power of the supercapacitor, as well as the operating status of each converter.

[0024] With the optimal control objective of the grid-type hybrid energy storage system as the core, a multi-objective weighted reward function is constructed, with the expression R=ω1×R1+ω2×R2+ω3×R3+ω4×R4, where ω1-ω4 are weight coefficients that are dynamically adjusted according to the control priority; R1 is the grid frequency stability reward, R2 is the energy storage unit SOC maintenance reward, R3 is the system charging and discharging loss reward, and R4 is the system operation stability reward.

[0025] A three-layer network structure of state input layer, feature extraction layer, and decision output layer is adopted. The PPO algorithm is used to build the agent, which balances training stability and convergence speed. The state input layer receives multi-dimensional operation data. The feature extraction layer uses convolutional neural network, CNN or long short-term memory network, LSTM to extract key features from the data, such as the trend of power grid frequency fluctuation and the change law of energy storage unit SOC. The decision output layer outputs the optimized key parameters of MPC controller to ensure the rationality and real-time performance of the output parameters.

[0026] Input parameters include the real-time operating status of the power grid and the operating status of each unit in the hybrid energy storage system. The real-time power grid operating status includes grid frequency (50Hz±0.2Hz), bus voltage (380V / 10kV), active power, reactive power, and load fluctuations. The hybrid energy storage system operating status includes the lithium iron phosphate battery SOC (20%-80%), charge / discharge power, and battery temperature; the supercapacitor SOC (10%-90%), terminal voltage, and charge / discharge power; and the operating status of each converter (normal / fault). Output parameters are the optimization parameters of the MPC controller, including prediction time domain (Np), control time domain (Nc), weighting coefficients of each objective function, and power allocation thresholds.

[0027] With the optimal control objective of the grid-type hybrid energy storage system as the core, a multi-objective weighted reward function is constructed, specifically expressed as: R = ω1×R1 + ω2×R2 + ω3×R3 + ω4×R4, where ω1-ω4 are weight coefficients, summing to 1, and dynamically adjusted according to control priority; R1 is the grid frequency stability reward, with a larger reward value for smaller frequency deviations; R2 is the energy storage unit SOC maintenance reward, with the maximum reward value when the SOC is within a reasonable range, and the reward value decreases when it deviates from the range; R3 is the system charging and discharging loss reward, with a larger reward value for smaller losses; and R4 is the system operation stability reward, with a constant reward value when there is no fault, and the reward value is reset to zero when a fault occurs.

[0028] Historical data on typical power grid operation scenarios, such as peak load, off-peak load, load surges, and power grid faults, were collected to construct a training dataset with a sample size of no less than 100,000 sets. The input parameters were normalized and mapped to the [0,1] interval to avoid the impact of data size differences on training results. Training parameters were set with a learning rate of 0.001-0.01, 1000-5000 iterations, and a batch processing sample size of 64-128 sets per iteration. An early stopping strategy was adopted, stopping training when the validation set reward value showed no improvement for 50 consecutive iterations to ensure agent convergence. After training, offline testing was conducted to verify the accuracy and rationality of the agent's output parameters, with a test pass rate of no less than 95%.

[0029] Step 103: Design the MPC controller. Based on the mathematical model of the hybrid energy storage system, construct a multi-objective optimization objective function by combining the optimized MPC controller parameters output by the DRL agent. Specifically, in this embodiment, the MPC controller includes a prediction module, an optimization module, and an execution module. The prediction module predicts the system operating state at Np future time points based on the mathematical model of the hybrid energy storage system and by combining real-time grid operation data and the state of the energy storage units. The optimization module solves for the optimal control sequence based on a multi-objective optimization objective function and by combining the optimization parameters output by the DRL agent. The execution module applies the optimized control sequence to the converters of each energy storage unit to achieve charge and discharge control.

[0030] Mathematical models for lithium iron phosphate batteries, supercapacitors, and converters were established separately. For lithium iron phosphate batteries, a first-order RC equivalent model was used to describe the relationship between state of charge (SOC) and charging / discharging current and voltage. Model parameters, including internal resistance and capacitance, were determined through a combination of offline testing and online identification. For supercapacitors, an equivalent resistance-capacitor model was used to focus on describing their rapid charging / discharging characteristics and SOC variation. For the bidirectional DC / DC converter, a state-space model was used to describe the conversion relationship between input and output voltage and current, considering switching losses and efficiency characteristics. Based on these unit models, an overall mathematical model of the hybrid energy storage system was obtained, providing a foundation for the prediction and optimization of the MPC controller.

[0031] The MPC controller adopts a predictive-optimization-control closed-loop structure, including a prediction module, an optimization module, and an execution module. The prediction module, based on a mathematical model of the hybrid energy storage system and combined with real-time grid operation data and the energy storage unit status, predicts the system's operating state in the time domain, such as SOC, charging / discharging power, and grid frequency, for the next Np time points. The optimization module, based on a multi-objective optimization objective function and combined with optimization parameters output by the DRL agent, such as Np, Nc, and weighting coefficients, solves for the optimal control sequence. The execution module applies the optimized control sequence to the converters of each energy storage unit to achieve charging / discharging control, while simultaneously collecting feedback data in real time to update the prediction model, forming a closed-loop control.

[0032] Based on the core control objectives of the grid-type hybrid energy storage system—grid frequency stability, energy storage unit SOC balance, minimum system loss, and operational safety—a multi-objective optimization objective function is constructed, specifically in the form: minJ=ω1×J1+ω2×J2+ω3×J3+ω4×J4. Here, J1 is the sum of squares of the grid frequency deviation, minimizing frequency fluctuations to ensure the frequency remains at 50Hz±0.2Hz; J2 is the sum of squares of the energy storage unit SOC deviation, ensuring the lithium battery SOC remains between 20%-80% and the supercapacitor SOC between 10%-90%; J3 is the system charge / discharge loss, including battery loss, supercapacitor loss, and converter loss; and J4 is the rate of change of the control variable, minimizing sudden changes in the control variable to avoid impacting the system. The weight coefficients ω1-ω4 of each objective function are output in real-time by the DRL agent and dynamically adjusted according to the grid operating status to ensure the optimality of the control objectives under different scenarios.

[0033] For the multi-objective optimization objective function, quadratic programming and the QP algorithm are selected for solution. This algorithm has high computational efficiency and fast convergence speed, which can meet the real-time control requirements of the MPC controller. The solution time is ≤10ms. At the same time, constraints are set, including energy storage unit charging and discharging power constraints, SOC constraints, and converter output voltage / current constraints, to ensure that the optimization results meet the requirements of safe system operation.

[0034] Step 104: Collect real-time power grid operation data and operation data of each unit of the grid-type hybrid energy storage system, and input them into the trained DRL agent to obtain optimized MPC controller parameters; Specifically, in this embodiment, the grid frequency, voltage, active power, and reactive power parameters are collected. The collected data are then subjected to outlier deletion, missing value supplementation, and normalization to obtain preprocessed parameters. The preprocessed real-time data is then input into the trained DRL agent. The agent extracts features and makes policy decisions to output optimized MPC controller parameters.

[0035] A PMU (Phasor Measurement Unit) is deployed at the power grid bus to collect parameters such as grid frequency, voltage, active power, and reactive power at a frequency of 10-50Hz. Current sensors, voltage sensors, and temperature sensors are deployed at the lithium iron phosphate battery units and supercapacitor units to collect the charging and discharging current, terminal voltage, and temperature of each unit at a frequency of 10Hz. Status monitoring modules are deployed at each bidirectional DC / DC converter and grid-connected inverter to collect the converter's operating status, switching status, and loss data at a frequency of 5Hz. All acquisition devices are connected to the EMS (Electronic Management System) to achieve centralized data aggregation and preprocessing.

[0036] The collected real-time data is preprocessed to remove outliers. The 3σ criterion is used to remove data that deviates from the normal range and missing values. Linear interpolation is used to supplement the missing values ​​and normalize the data to the [0,1] interval to eliminate the impact of data magnitude differences on the DRL agent's decision-making. At the same time, the data is smoothed by using the moving average method to filter high-frequency noise and ensure the stability and accuracy of the data. The delay after preprocessing is ≤50ms.

[0037] The preprocessed real-time data is input into the trained DRL agent. The agent extracts features and makes policy decisions, and outputs optimized MPC controller parameters, including prediction time-domain Np (5-20 steps), control time-domain Nc (2-10 steps), weight coefficients ω1-ω4 of the multi-objective optimization objective function, and energy storage unit power allocation threshold. After the parameters are output, the EMS system performs a rationality check to ensure that the parameters are within the preset range. If the check passes, the parameters are transmitted to the MPC controller. If the check fails, the agent decision-making is retried until reasonable parameters are output.

[0038] Establish an online update mechanism for the DRL agent. Every 24 hours, the agent is fine-tuned and trained using real-time operating data collected that day, with 100-200 iterations. The network parameters are updated to adapt to changes in the power grid operating status, ensuring the decision-making accuracy and adaptability of the agent and avoiding parameter optimization failure due to changes in the power grid operating scenario.

[0039] Step 105: Based on the optimized MPC controller parameters and multi-objective optimization objective function, the MPC controller performs real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage unit and supercapacitor energy storage unit of the grid-type hybrid energy storage system, so as to realize the coordinated operation of the grid-type hybrid energy storage system and the power grid.

[0040] Specifically, in this embodiment, the MPC controller calculates the optimal charging and discharging power allocation scheme for the next Nc time points based on the optimized parameters and prediction model; when the power grid experiences a sudden high-frequency load change, the supercapacitor responds quickly and outputs power to smooth out high-frequency fluctuations; when the power grid experiences a low-frequency load change, the lithium iron phosphate battery slowly adjusts the charging and discharging power to maintain the system's energy balance.

[0041] Based on optimized parameters and a prediction model, the MPC controller calculates the optimal charging and discharging power allocation scheme for the next Nc time points, following the principle that high-frequency fluctuations are handled by supercapacitors and low-frequency fluctuations by lithium iron phosphate batteries. The specific allocation logic is as follows: when the power grid experiences a high-frequency load change with a fluctuation frequency ≥1Hz, the supercapacitor responds quickly, outputting or absorbing power to smooth out the high-frequency fluctuations; when the power grid experiences a low-frequency load change with a fluctuation frequency <1Hz, the lithium iron phosphate battery slowly adjusts its charging and discharging power to maintain the system's energy balance; simultaneously, considering the SOC status of the energy storage unit, when the lithium battery SOC is below 20% or above 80%, and the supercapacitor SOC is below 10% or above 90%, the power allocation ratio is dynamically adjusted to prioritize the safe operation of the energy storage unit.

[0042] The MPC controller transmits optimized charge / discharge power commands to the bidirectional DC / DC converters of each energy storage unit via a communication link. The converters adjust their switching states according to the commands to achieve precise control of the charge / discharge power. For lithium iron phosphate battery units, a constant current-constant voltage (CC-CV) charge / discharge control strategy is adopted. During charging, a constant current is first used to charge to the rated voltage, and then the voltage is maintained at a constant level to avoid overcharging. During discharging, a constant current is used to ensure stable discharge. For supercapacitor units, a constant current charge / discharge control strategy is adopted, which quickly adjusts the charge / discharge current according to the power command with a response time of ≤10ms, while controlling the charge / discharge voltage to not exceed the rated voltage to avoid damage to the equipment.

[0043] During the control process, the EMS system collects real-time operating status of each unit in the hybrid energy storage system, real-time grid operating data, and the control effect of the MPC controller, and feeds this data back to the DRL agent and the MPC controller. If the grid frequency deviation exceeds the allowable range, the SOC of the energy storage unit deviates from the reasonable range, or a system fault occurs, such as a converter fault or sensor fault, the DRL agent immediately re-optimizes the MPC parameters, and the MPC controller adjusts the power allocation and charging / discharging commands until the system returns to stability. At the same time, the system maintains coordination with the grid dispatch center, receives dispatch commands such as peak shaving and valley filling, and frequency regulation commands, and adjusts the control strategy to achieve coordinated operation of the hybrid energy storage system and the grid, thereby improving the stability and reliability of grid operation.

[0044] An emergency fault module is set up so that when a system fault is detected, such as battery overheating, supercapacitor overvoltage, or converter failure, the connection between the faulty unit and the system is immediately cut off, its charging and discharging operation is stopped, and the power output of another energy storage unit is adjusted to compensate for the functional loss of the faulty unit and ensure the overall stable operation of the system. After the fault is cleared, the system automatically returns to the normal control mode, the DRL agent re-optimizes the parameters, and the MPC controller adjusts the control strategy to achieve a smooth transition of the system.

[0045] Its beneficial effects lie in the fact that by adopting a hybrid architecture of lithium iron phosphate batteries and supercapacitors, coupled with a scientific power allocation strategy, it effectively mitigates high- and low-frequency load fluctuations in the power grid and improves grid operational stability, balancing energy storage and rapid response requirements. The DRL agent can dynamically optimize MPC controller parameters, overcoming the shortcomings of traditional MPC parameters being fixed and lacking adaptability. It can handle complex grid scenarios without relying on precise mathematical models, improving control flexibility and accuracy. Real-time data acquisition and preprocessing, along with an online agent update mechanism, ensure the real-time nature and accuracy of control decisions, reducing system charging and discharging losses. A robust fault emergency handling and feedback adjustment mechanism ensures the safe operation of the energy storage unit, extends its service life, and reduces operation and maintenance costs. This method achieves efficient collaboration between the hybrid energy storage system and the power grid, significantly improving the control performance of grid-connected energy storage.

[0046] Please see Figure 2 In the collaborative control method for network-based hybrid energy storage based on deep reinforcement learning and MPC, a network-based hybrid energy storage system is constructed. The hybrid energy storage system includes at least lithium iron phosphate battery energy storage units and supercapacitor energy storage units, comprising the following steps. Step 201: The lithium iron phosphate battery energy storage unit is used to provide continuous energy support, and the supercapacitor energy storage unit is used to achieve millisecond-level fast power response to adapt to the primary frequency regulation scenario of the power grid. Step 202: Adopting a centralized DC bus topology, the lithium iron phosphate battery units are connected to the DC bus through a bidirectional DC / DC converter, and the supercapacitor units are connected in parallel to the same DC bus through a bidirectional DC / DC converter. The DC bus is connected to the AC grid through a grid-connected inverter to realize bidirectional AC-DC energy conversion. Step 203: Configure the EMS energy management system to coordinate the operating status of the two energy storage units, collect and upload system operation data, and receive upper-level control commands.

[0047] Please see Figure 3 In the collaborative control method for grid-based hybrid energy storage based on deep reinforcement learning and MPC, a deep reinforcement learning DRL agent is built. The DRL agent is trained using the real-time operating status of the power grid and the operating status of each unit in the hybrid energy storage system as inputs, and the optimal control objective of the grid-based hybrid energy storage system as the reward function. The training includes the following steps. Step 301: Build an intelligent agent using the PPO reinforcement learning algorithm. The state input layer receives multi-dimensional running data, the feature extraction layer uses a CNN convolutional neural network to extract key features from the data, and the decision output layer outputs the optimized key parameters of the MPC controller. Step 302: Input parameters include the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system. The real-time operating status of the power grid includes the grid frequency, bus voltage, active power, reactive power and load fluctuation. The operating status of the hybrid energy storage system includes the SOC, charging and discharging power and battery temperature of the lithium iron phosphate battery, the SOC, terminal voltage and charging and discharging power of the supercapacitor, and the operating status of each converter.

[0048] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended technical solutions and equivalents.

Claims

1. A network construction type hybrid energy storage collaborative control method based on deep reinforcement learning and MPC, characterized in that, The grid-type hybrid energy storage collaborative control method includes the following steps: Construct a grid-type hybrid energy storage system, wherein the hybrid energy storage system includes at least a lithium iron phosphate battery energy storage unit and a supercapacitor energy storage unit; A deep reinforcement learning (DRL) agent is constructed, with the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and the optimal control objective of the grid-type hybrid energy storage system as the reward function for training the DRL agent. Design an MPC controller, based on a mathematical model of a hybrid energy storage system, and construct a multi-objective optimization objective function by combining the optimized MPC controller parameters output by the DRL agent; Real-time power grid operation data and operation data of each unit in the grid-type hybrid energy storage system are collected and input into the trained DRL agent to obtain optimized MPC controller parameters; The MPC controller, based on optimized MPC controller parameters and multi-objective optimization objective function, performs real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage unit and supercapacitor energy storage unit of the grid-type hybrid energy storage system, thereby realizing the coordinated operation of the grid-type hybrid energy storage system and the power grid.

2. The hybrid energy storage coordinated control method based on deep reinforcement learning and MPC for network construction type according to claim 1, wherein, The constructed grid-type hybrid energy storage system includes at least lithium iron phosphate battery energy storage units and supercapacitor energy storage units, comprising: Among them, the lithium iron phosphate battery energy storage unit is used to provide continuous energy support, and the supercapacitor energy storage unit is used to achieve millisecond-level fast power response, adapting to the primary frequency regulation scenario of the power grid. The system adopts a centralized DC bus topology. The lithium iron phosphate battery units are connected to the DC bus through a bidirectional DC / DC converter, and the supercapacitor units are connected in parallel to the same DC bus through a bidirectional DC / DC converter. The DC bus is connected to the AC grid through a grid-connected inverter, realizing bidirectional AC-DC energy conversion. Configure an EMS energy management system to coordinate the operating status of the two energy storage units, collect and upload system operation data, and receive upper-level control commands.

3. The hybrid energy storage coordinated control method based on deep reinforcement learning and MPC for network construction type according to claim 1, wherein, The construction of the deep reinforcement learning (DRL) agent, using the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and using the optimal control objective of the grid-type hybrid energy storage system as the reward function, includes: An intelligent agent is built using the PPO reinforcement learning algorithm. The state input layer receives multi-dimensional operational data, the feature extraction layer uses a CNN convolutional neural network to extract key features from the data, and the decision output layer outputs the optimized key parameters of the MPC controller. The input parameters include the real-time operating status of the power grid and the operating status of each unit in the hybrid energy storage system. The real-time operating status of the power grid includes the grid frequency, bus voltage, active power, reactive power and load fluctuation. The operating status of the hybrid energy storage system includes the SOC, charging and discharging power and battery temperature of the lithium iron phosphate battery, the SOC, terminal voltage and charging and discharging power of the supercapacitor, and the operating status of each converter.

4. The hybrid energy storage coordinated control method based on deep reinforcement learning and MPC for network construction type according to claim 1, wherein, The construction of the deep reinforcement learning (DRL) agent, using the real-time operating status of the power grid and the operating status of each unit of the hybrid energy storage system as inputs, and using the optimal control objective of the grid-type hybrid energy storage system as the reward function, includes: With the optimal control objective of the grid-type hybrid energy storage system as the core, a multi-objective weighted reward function is constructed, with the expression R=ω1×R1+ω2×R2+ω3×R3+ω4×R4, where ω1-ω4 are weight coefficients that are dynamically adjusted according to the control priority; R1 is the grid frequency stability reward, R2 is the energy storage unit SOC maintenance reward, R3 is the system charging and discharging loss reward, and R4 is the system operation stability reward.

5. The hybrid energy storage coordinated control method based on deep reinforcement learning and MPC for network construction type according to claim 1, wherein, The designed MPC controller, based on the mathematical model of the hybrid energy storage system, constructs a multi-objective optimization objective function by combining the optimized MPC controller parameters output by the DRL agent, including: The MPC controller includes a prediction module, an optimization module, and an execution module. The prediction module is based on a mathematical model of a hybrid energy storage system and combines real-time grid operation data and the status of energy storage units to predict the system operation status at Np future moments. The optimization module solves for the optimal control sequence based on a multi-objective optimization objective function and the optimization parameters output by the DRL agent; the execution module applies the optimized control sequence to the converters of each energy storage unit to achieve charge and discharge control.

6. The hybrid energy storage coordinated control method based on deep reinforcement learning and MPC for network construction type according to claim 1, wherein, The collected real-time power grid operation data and the operation data of each unit in the grid-type hybrid energy storage system are input into the trained DRL agent to obtain optimized MPC controller parameters, including: Collect grid frequency, voltage, active power and reactive power parameters, and sequentially perform outlier deletion, missing value supplementation and normalization on the collected data to obtain preprocessed parameters; The preprocessed real-time data is input into the trained DRL agent, which then extracts features and makes policy decisions to output optimized MPC controller parameters.

7. The hybrid energy storage coordinated control method based on deep reinforcement learning and MPC for network construction type according to claim 1, wherein, The MPC controller, based on optimized MPC controller parameters and a multi-objective optimization objective function, performs real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage units and supercapacitor energy storage units of the grid-type hybrid energy storage system, realizing the coordinated operation of the grid-type hybrid energy storage system with the power grid, including: The MPC controller calculates the optimal charging and discharging power allocation scheme for the next Nc time points based on the optimized parameters and prediction model. When the power grid experiences sudden high-frequency load changes, the supercapacitor responds quickly and outputs power to smooth out high-frequency fluctuations; when the power grid experiences low-frequency load changes, the lithium iron phosphate battery slowly adjusts its charging and discharging power to maintain the system's energy balance.

8. A system for implementing the hybrid energy storage collaborative control method based on deep reinforcement learning and MPC for network construction according to claim 1, characterized in that, The system includes the following modules: An energy storage system establishment module is used to construct a grid-type hybrid energy storage system, wherein the hybrid energy storage system includes at least a lithium iron phosphate battery energy storage unit and a supercapacitor energy storage unit. The DRL agent training module is used to build a deep reinforcement learning DRL agent. The DRL agent is trained with the real-time operation status of the power grid and the operation status of each unit of the hybrid energy storage system as inputs, and the optimal control objective of the grid-type hybrid energy storage system as the reward function. The MPC controller design module is used to design MPC controllers. Based on the mathematical model of the hybrid energy storage system, it constructs a multi-objective optimization objective function by combining the optimized MPC controller parameters output by the DRL agent. The controller parameter optimization module is used to collect real-time power grid operation data and operation data of each unit of the grid-type hybrid energy storage system, and input them into the trained DRL agent to obtain optimized MPC controller parameters. The collaborative operation control module is used by the MPC controller to perform real-time power allocation and charge / discharge control on the lithium iron phosphate battery energy storage units and supercapacitor energy storage units of the grid-type hybrid energy storage system based on optimized MPC controller parameters and multi-objective optimization objective function, so as to realize the collaborative operation of the grid-type hybrid energy storage system with the power grid.

9. The system for implementing the hybrid energy storage collaborative control method based on deep reinforcement learning and MPC for network construction according to claim 8, wherein The controller parameter optimization module includes the following sub-modules: The data acquisition submodule is used to acquire grid frequency, voltage, active power and reactive power parameters. The acquired data is then processed by deleting outliers, filling in missing values ​​and normalizing the data to obtain preprocessed parameters. The output submodule is used to input the preprocessed real-time data into the trained DRL agent. The agent then outputs the optimized MPC controller parameters through feature extraction and policy decision-making.

10. A system for implementing the hybrid energy storage collaborative control method based on deep reinforcement learning and MPC for network construction according to claim 8, characterized in that, The collaborative operation control module includes the following sub-modules: The computational sub-MPC controller calculates the optimal charging and discharging power allocation scheme for the next Nc time steps based on the optimized parameters and prediction model. When the power grid experiences high-frequency load changes, the supercapacitor responds quickly and outputs power to smooth out high-frequency fluctuations; when the power grid experiences low-frequency load changes, the lithium iron phosphate battery slowly adjusts the charging and discharging power to maintain the system's energy balance.

Images