Energy storage frequency modulation control method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of power control, and discloses an energy storage frequency modulation control method and device, electronic equipment and a storage medium. Comprising the following steps: predicting system state variables based on an MPC mechanism model and an LSTM-based load prediction model; based on the system state variable and the load disturbance state variable, outputting a basic power instruction at the current moment through an MPC-based model prediction controller; based on the system state variable and the load disturbance state variable, outputting a power compensation instruction through a reinforcement learning agent based on DDPG; superposing the basic power instruction and the power compensation instruction to generate a target control instruction and acting the target control instruction on a control object; the output state of the control object base is acquired, and the output state is used as a feedback signal. According to the invention, the response speed, the control precision and the operation robustness of the energy storage frequency modulation system to complex power grid working conditions are obviously improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of power control, in particular to a method and device for frequency control of energy storage, an electronic device and a storage medium. BACKGROUND

[0002] In modern power systems, large-scale grid connection of new energy and wide application of power electronic devices make the grid frequency stability face severe challenges, further promoting the implementation and application of energy storage frequency control, multi-source collaborative optimization, dynamic response modeling and other technologies, providing protection for the safe and stable operation of power systems.

[0003] Due to the complexity and uncertainty of the operating environment of the power system and the limitations of the response characteristics of traditional frequency regulation resources, in the frequency regulation process, regulation delay, frequency deviation and other situations will inevitably occur, resulting in unsatisfactory frequency control effect. These regulation processes with dynamic deviation cannot directly meet the demand for fast and accurate frequency regulation of high-proportion new energy power systems. SUMMARY

[0004] In view of the above problems, the embodiments of the present application provide a method and device for frequency control of energy storage, an electronic device and a storage medium, which aims to solve the above problems or at least partially solve the above problems.

[0005] In the first aspect, the embodiments of the present application provide a method for frequency control of energy storage, which comprises: In each control cycle, predict the system state variable based on the MPC mechanism model and the load prediction model based on LSTM; Based on the system state variable and the load disturbance state variable, predict the optimal power distribution instruction in the future control time domain through the model prediction controller based on MPC, and output the basic power instruction at the current time; Based on the system state variable and the load disturbance state variable, output the power compensation instruction through the reinforcement learning agent based on DDPG; Superimpose the basic power instruction output by the model prediction controller and the power compensation instruction output by the reinforcement learning agent module to generate a target control instruction and act on the control object; Obtain the output state of the control object based on the target control instruction, take the output state as a feedback signal, and send it to the MPC mechanism model and the load prediction model based on LSTM for feedback correction, and to the reinforcement learning agent module for policy update.

[0006] In the second aspect, the embodiments of the present application also provide a device for frequency control of energy storage, which comprises: a prediction module, configured to predict system state variables based on a mechanism model of MPC and a load prediction model based on LSTM in each control cycle; a first output module, configured to predict optimal power distribution instructions in a future control time domain based on the system state variables and the load disturbance state variables by a model predictive controller based on MPC, and output basic power instructions at a current time; a second output module, configured to output power compensation instructions by a reinforcement learning agent based on DDPG based on the system state variables and the load disturbance state variables; a control module, configured to superimpose the basic power instructions output by the model predictive controller and the power compensation instructions output by the reinforcement learning agent to generate target control instructions and act on a control object; a feedback module, configured to obtain an output state of the control object based on the target control instructions, and send the output state as a feedback signal to the mechanism model based on MPC and the load prediction model based on LSTM for feedback correction, and to the reinforcement learning agent module for policy update.

[0007] In a third aspect, an electronic device is provided, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the steps of the first aspect.

[0008] In a fourth aspect, a computer readable storage medium is provided, which stores one or more programs that, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the steps of the first aspect.

[0009] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: By coordinating the prediction of system state variables through the MPC-based mechanism model and the LSTM-based load prediction model, the physical logic rationality of the state prediction is ensured by relying on the MPC mechanism model, and the prediction accuracy of nonlinear and time-varying load disturbances is improved by leveraging the ability of the LSTM model to capture the load time-series characteristics. On this basis, the basic power command generated by the MPC model predictive controller can be effectively incorporated into the system constraints to ensure the safety and stability of the control. The power compensation command output by the reinforcement learning agent based on DDPG can dynamically adapt to model errors and sudden disturbances under complex operating conditions. The target control command formed by the superposition of the two has both basic stability and dynamic adaptability. At the same time, the output state of the controlled object corrects the prediction deviation of the MPC mechanism model and the LSTM load prediction model in real time through the feedback mechanism, and drives the reinforcement learning agent to continuously optimize the strategy, forming a complete closed-loop control link. This integrated architecture effectively combines the constraint processing capabilities of MPC with the adaptive decision-making advantages of reinforcement learning, significantly improving the response speed, control accuracy, and operational robustness of energy storage frequency regulation systems to complex grid conditions. It can more efficiently smooth frequency fluctuations, maintain the stable state of charge of energy storage, and thus improve the overall performance of grid frequency regulation. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This diagram illustrates the overall framework of the energy storage frequency regulation control method provided in the embodiments of this application. Figure 2 A flowchart of the energy storage frequency regulation control method provided in an embodiment of this application is shown; Figure 3 A schematic diagram of the power grid frequency regulation model provided in the embodiments of this application is shown; Figure 4 A schematic diagram of the architecture of a reinforcement learning agent based on DDPG provided in an embodiment of this application is shown; Figure 5 A schematic diagram of frequency deviation provided in an embodiment of this application is shown; Figure 6 This paper presents a schematic diagram of the power output of each power generation unit under a step disturbance provided in an embodiment of this application. Figure 7 This paper illustrates a schematic diagram of the system frequency deviation under randomly generated continuous disturbances provided in an embodiment of this application. Figure 8 This invention provides a schematic diagram illustrating the output of each power generation unit under continuous disturbance according to an embodiment of this application. Figure 9 This paper illustrates a schematic diagram of the change in the state of charge of energy storage under continuous disturbances, as provided in an embodiment of this application. Figure 10 A structural diagram of the energy storage frequency modulation control device provided in an embodiment of this application is shown; Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the term "comprising" and its variations should be interpreted as open-ended terms meaning "including but not limited to."

[0013] Energy storage frequency regulation control technology coordinates the cooperation between energy storage systems and traditional generating units through advanced algorithms, dynamically optimizes power allocation strategies, minimizes frequency deviation and reduces regulation costs, and is a key link in the frequency stability control of new power systems.

[0014] The specific steps of the existing energy storage joint frequency regulation algorithm are as follows: 1) Construct a steady-state model for thermal power units or a model for new energy power generation such as wind power and photovoltaic power, and set power constraints for the units; 2) Establish a basic model of the energy storage power station, taking into account the energy storage state of charge and charging / discharging power limitations; 3) AGC commands are allocated to each frequency modulation resource using a proportional allocation or frequency division control method; 4) The generator set and the energy storage power station adjust according to the instructions they receive.

[0015] Existing thermal power-energy storage combined frequency regulation control technology has the following key technical shortcomings when addressing the frequency regulation requirements of new power systems: 1. Traditional model predictive control (MPC) methods rely too heavily on the accuracy of the system model. Existing methods often use linearized models with fixed parameters to handle highly nonlinear frequency regulation processes. When the system encounters fluctuations in renewable energy output or sudden load changes, this model dependence can cause significant deviations between the regulation commands and the actual output.

[0016] 2. Existing control strategies have inherent limitations in handling the nonlinear characteristics of systems. The dynamic output process of thermal power units involves multiple components connected in series, such as governors and reheaters, and its response characteristics exhibit complex time-varying nonlinear features. Although traditional linearization methods can improve the efficiency of model solving, they introduce significant computational errors and cannot accurately reflect the true dynamics of the system.

[0017] 3. Multiple uncertainties exist in frequency regulation scenarios. On the one hand, AGC command prediction errors accumulate as the prediction time domain lengthens; on the other hand, the response characteristics of thermal power units are significantly affected by the load rate. Conventional frequency regulation control systems lack adaptive optimization mechanisms, and existing fixed-parameter optimization algorithms lack online learning capabilities, making it difficult to adjust control strategies in a timely manner to adapt to real-time changes in system states. Controllers with fixed parameters cannot autonomously adjust their strategies. This rigid control architecture leads to both a decline in frequency regulation performance and unnecessary losses of energy storage resources. In scenarios with a high proportion of renewable energy integration, the increased system uncertainty exacerbates this problem.

[0018] Based on this, this invention proposes an energy storage frequency regulation control method. It collaboratively predicts system state variables using an MPC-based mechanistic model and an LSTM-based load forecasting model. The MPC mechanistic model ensures the physical logic rationality of the state prediction, while the LSTM model's ability to capture load time-series characteristics improves the prediction accuracy for nonlinear and time-varying load disturbances. Furthermore, the basic power command generated by the MPC model predicts the controller and effectively incorporates system constraints, ensuring control safety and stability. Meanwhile, the power compensation command output by the DDPG-based reinforcement learning agent dynamically adapts to model errors and sudden disturbances under complex operating conditions. The target control command formed by the superposition of these two methods possesses both basic stability and dynamic adaptability. Simultaneously, the output state of the controlled object corrects the prediction deviations of the MPC mechanistic model and the LSTM load forecasting model in real time through a feedback mechanism, driving the reinforcement learning agent to continuously optimize the strategy, forming a complete closed-loop control link. This integrated architecture effectively combines the constraint processing capabilities of MPC with the adaptive decision-making advantages of reinforcement learning, significantly improving the response speed, control accuracy, and operational robustness of energy storage frequency regulation systems to complex grid conditions. It can more efficiently smooth frequency fluctuations, maintain the stable state of charge of energy storage, and thus improve the overall performance of grid frequency regulation.

[0019] The present application will now be described in detail through specific embodiments.

[0020] Figure 1 The diagram illustrates the overall framework of an energy storage frequency regulation control method according to an embodiment of this application. It mainly includes: a hybrid prediction model (a mechanism model based on MPC and a load state prediction model based on LSTM), a model prediction controller, a reinforcement learning agent, and a feedback phase. The MPC-based mechanism model derives state prediction results based on the system's physical / mathematical mechanisms (such as a state-space model), while the LSTM-based data-driven state prediction submodule outputs state prediction results by learning patterns from historical time-series data. Together, these two components constitute the hybrid prediction model, enabling the prediction of system state variables.

[0021] After fusing the load disturbance prediction results with the output of the state prediction module, the data is transmitted in two paths: one path is sent to the model predictive controller (MPC): this controller is configured with a prediction time domain (covering the control time domain) including times k, k+1, ..., k+m. Based on the input load disturbance and state prediction information, it infers the future state and control sequence of the system within the prediction time domain, generating the basic power command for the MPC control output. The other path is sent to the reinforcement learning agent: this agent generates a control policy through a policy function and evaluates the policy's merits using a reward function, outputting a power compensation command.

[0022] The basic power command of the model predictive controller is superimposed with the power compensation command of the reinforcement learning agent and fed into the rolling optimization module to generate the optimal control command for the current moment. The controlled object receives the optimal control command and executes the control action, while interacting with the external environment to generate the actual output state.

[0023] The actual output state of the controlled object is sent as a feedback signal to the feedback correction module. After processing, it is fed back to the MPC-based mechanism model and the LSTM-based load state prediction model to correct the subsequent state prediction results and improve the prediction accuracy. It is also fed back to the reinforcement learning agent module to update its policy function and reward function, so that the reinforcement learning policy can adapt to the actual operation of the system, and finally form a complete closed loop process of prediction-decision-control-feedback-correction.

[0024] This architecture leverages the rolling optimization capabilities of MPC while enhancing the system's adaptability to nonlinear and uncertain operating conditions through reinforcement learning, thus addressing the issues of traditional control methods' reliance on accurate models and insufficient robustness.

[0025] Figure 2 This diagram illustrates a flow chart of an energy storage frequency regulation control method according to an embodiment of this application. Figure 2 It can be seen that the method may include steps S201 to S205: Step S201: In each control cycle, the system state variables are predicted by the MPC-based mechanistic model and the LSTM-based load forecasting model.

[0026] Step S202: Based on the system state variables and load disturbance state variables, the optimal power allocation command in the future control time domain is predicted by the model predictive controller based on MPC, and the basic power command at the current moment is output.

[0027] Step S203: Based on the system state variables and load disturbance state variables, output power compensation commands through a reinforcement learning agent based on DDPG.

[0028] Step S204: The basic power command output by the model prediction controller is superimposed with the power compensation command output by the reinforcement learning agent module to generate a target control command and apply it to the controlled object.

[0029] Step S205: Obtain the output state of the controlled object based on the target control command, and use the output state as a feedback signal to send to the MPC-based mechanism model and the LSTM-based load prediction model for feedback correction, and to the reinforcement learning agent module for policy update.

[0030] from Figure 1 As can be seen from the method shown, this application effectively combines the constraint processing capability of MPC with the adaptive decision-making advantage of reinforcement learning through this fusion architecture, which significantly improves the response speed, control accuracy and operational robustness of the energy storage frequency regulation system to complex grid conditions. It can more efficiently smooth frequency fluctuations and maintain the stable state of charge of energy storage, thereby improving the overall performance of grid frequency regulation.

[0031] In some embodiments of this application, the output vector of the MPC-based mechanistic model includes grid frequency deviation and the combined state of charge of hybrid energy storage.

[0032] Among them, the grid frequency deviation is determined based on the power output of thermal power units, the power output of battery energy storage, the power output of supercapacitors, and system load disturbances; the power output of hybrid energy storage is determined based on the state of charge of battery energy storage and the state of charge of super battery energy storage system.

[0033] Specifically, firstly, a grid frequency regulation model incorporating hybrid energy storage is established, such as... Figure 3 As shown. Based on the dynamic characteristics of the power grid, the relationship between frequency deviation and power in a combined frequency regulation system with hybrid energy storage can be described as follows:

[0034] In the formula, For the power output of thermal power units, and These represent the power outputs of supercapacitors and battery energy storage, respectively. For system load disturbance, Let represent the change in system frequency deviation, and D and H represent the generator load damping coefficient and inertial time constant, respectively.

[0035] A mathematical model is established for the joint frequency modulation system, and the system equations are shown in the figure.

[0036]

[0037] In the formula, This is the power command corresponding to the opening degree of the turbine regulating valve; This is the unit power regulation coefficient for thermal power units; This refers to the power grid frequency deviation. The transfer function of the speed controller; This refers to the mechanical power of the generator; , These are the transfer functions for the steam turbine and the generator, respectively. , These are the power reference commands for battery energy storage and supercapacitor systems, respectively. and These are the transfer functions for battery energy storage and supercapacitor energy storage systems, respectively, used to describe the dynamic characteristics of power point tracking in energy storage systems. and These refer to the state of charge of battery energy storage systems and supercapacitor energy storage systems, respectively. and These are the rated capacities of battery energy storage and supercapacitor energy storage, respectively. The sampling time interval; The overall state of charge for hybrid energy storage; , The weighting coefficients for calculating the overall state of charge of hybrid energy storage. This indicates the deviation of system load disturbance.

[0038] The standard form of the discrete state equations is:

[0039] The system output vector includes the combined state of charge of the grid frequency deviation and the hybrid energy storage:

[0040] in, , Here, A, B, and C are the system input and output vectors, respectively; A, B, and C are the model parameters, representing the input matrix, control matrix, and output matrix, respectively; and D is the disturbance matrix. Simultaneously, a comprehensive dynamic model of the energy storage state of charge is embedded. Used to constrain charging and discharging power and prevent the battery's state of charge. and the state of charge of supercapacitors An over-limit event has occurred.

[0041] In this embodiment, the MPC-based mechanism model constructs a prediction model based on the physical or mathematical mechanisms of the system (such as state-space models, dynamic equations, etc.), ensuring the physical logic correctness and interpretability of the state prediction, and making the prediction results strictly conform to the essential dynamic laws of the system. At the same time, it can explicitly incorporate system constraints (such as upper and lower limits of the state, control input boundaries, etc.), providing state expectations that conform to physical constraints for the rolling optimization of the model predictive controller, ensuring the safety and feasibility of the control strategy. In scenarios where the system mechanism is clear and the operating conditions and model match well, the prediction accuracy is high and the robustness is strong. It can accurately characterize the basic dynamics of the system, provide reliable input for the optimization decision-making process of MPC, and thus improve the stability and control accuracy of the entire control algorithm.

[0042] In some embodiments of this application, the MPC-based model predictive controller is optimized based on a multi-objective function, which includes: a frequency deviation minimization term; a state-of-charge deviation minimization term; and an economic cost minimization term, whereby the economic cost includes the coal consumption cost of thermal power units, environmental governance costs, energy storage loss costs, and operation and maintenance costs.

[0043] Specifically, the multi-objective function is set as follows:

[0044] In the formula, Weighted matrix for the target; ΔY(k) The output vector change includes the degree of frequency deviation and the degree of overall state of charge deviation, with reference values ​​of 0 and 0.5, respectively. C hess This refers to the cost of system frequency regulation.

[0045]

[0046] In the formula, To predict the state of charge deviation of hybrid energy storage in the time domain.

[0047] Hybrid energy storage economic cost indicators include unit frequency regulation costs Energy storage frequency regulation cost and primary frequency modulation subsidy :

[0048] Frequency regulation costs of generating units include coal consumption of thermal power units. and environmental governance costs .

[0049]

[0050] The coal consumption of thermal power units is:

[0051] In the formula, , These are the coal consumption level of thermal power units and the average price of coal, respectively. This refers to the power generation of thermal power units.

[0052]

[0053] In the formula, For thermal power units in Frequency modulation output value at any given moment.

[0054] The cost of environmental remediation is:

[0055] In the formula, , SO2 and NO are produced during thermal power generation, respectively. x The amount of gas discharged per unit of electricity; , For SO2 and NO x The corresponding unit gas treatment cost.

[0056] Energy storage frequency regulation cost =Cost of lost electricity from supercapacitor and battery energy storage + Energy storage operation and maintenance costs Among these, the cost of lost electricity from supercapacitors and battery energy storage. As shown below:

[0057] In the formula, p el Cost per unit of electricity loss; P ess (t) represents the real-time output power of the energy storage; the charge and discharge efficiencies η of supercapacitor energy storage and battery energy storage are 0.95 and 0.90, respectively.

[0058] Energy storage operation and maintenance costs As shown below:

[0059] In the formula: Maintenance cost per unit capacity; Maintenance cost per unit power; rated power of the energy storage system; This refers to the rated capacity of the energy storage system.

[0060] The subsidies for energy storage participating in primary frequency regulation in a thermal power-HESS system are as follows:

[0061] In the formula, The unit price of revenue for energy storage participating in primary frequency regulation.

[0062] Furthermore, the constraints of the model predictive controller based on MPC include thermal power unit ramping constraints, energy storage charging and discharging power constraints, and energy storage state of charge constraints.

[0063] Specifically, the traditional unit ramp-up constraint can be simplified as follows:

[0064] In the formula, This represents the current ramp value of the thermal power unit. , These are the upper and lower limits of the unit's output value, respectively. R represents the unit's base capacity; R represents the unit's ramp-up limit.

[0065] The energy storage charging and discharging power constraints are as follows:

[0066] In the formula, , Determined based on real-time state of charge The upper and lower limits of energy storage charging and discharging power at any given time; For two types of energy storage Time prediction The respective power at each moment; for Time prediction The increment of energy storage power at any given moment.

[0067] The state of charge constraints are as follows:

[0068] In the formula, and Battery and flywheel energy storage respectively Always Predicted state of charge at time t.

[0069] In this embodiment, by setting multi-objective functions and constraints, the basic power command output by MPC is both optimal (satisfying multi-objective balance) and feasible (not violating system constraints), providing a reliable basic layer control guarantee for subsequent superposition with reinforcement learning compensation commands.

[0070] In some embodiments of this application, the DDPG-based reinforcement learning agent includes a policy network and a value network. The policy network, acting as a policy function, generates continuous power compensation instructions based on the system state mapping; the value network, acting as an evaluator of the reward function, evaluates the long-term value of actions and guides the policy optimization of the policy network.

[0071] Specifically, the architecture of the DDPG-based reinforcement learning agent is as follows: Figure 4 As shown, the Actor network, also known as the policy network, receives state information, learns output actions, and obtains rewards through environmental feedback. In energy storage frequency regulation applications, it takes the grid state (frequency deviation, state of charge, etc.) as input and outputs continuous energy storage charging and discharging power commands, realizing the mapping from state to action. The Critic network, also known as the value network, is used to evaluate the value of the Actor network's output actions and guide policy optimization. The target Actor network and target Critic network replicate the parameters of the Actor and Critic networks, respectively, to calculate the target Q value and stabilize the training process. Action is the deterministic action output by the policy network in the given state; reward is the reward value at step i; and N is the number of transition samples randomly sampled from the experience replay pool. It combines the experience replay mechanism of DQN and the AC network architecture, solving the convergence problem of neural networks when approximating function values.

[0072] During training, the agent generates experience data (state, action, reward, next state) by interacting with the power grid environment, stores it in the experience replay pool, and randomly samples to update network parameters, avoiding the influence of sample correlation on training stability.

[0073] Furthermore, to adapt to the complex dynamic characteristics of the power grid, a hierarchical state encoder is introduced.

[0074] The hierarchical state encoder employs a CNN-LSTM hybrid network. The input vector of the hierarchical state encoder includes the current frequency deviation, frequency change rate, energy storage state of charge, state of charge deviation, and load allocation command prediction. After being encoded by the hierarchical state encoder, it is input into the policy network and value network of the DDPG-based reinforcement learning agent.

[0075] Specifically, the CNN layer extracts spatial features, such as the frequency deviation distribution of different nodes and regional power fluctuation differences, and captures local power grid characteristics through convolution operations; LSTM layers capture temporal dynamics, such as the trend of changes in the state of charge and the time-series characteristics of frequency deviations, and memorize long-term dependencies through gating mechanisms.

[0076] The input vector includes: current frequency deviation, frequency change rate, energy storage state of charge, state of charge deviation, and load allocation command prediction value. After encoding, it is used as input to the policy network and value network to improve the accuracy of state perception.

[0077] Furthermore, the reinforcement learning agent based on DDPG employs a multi-objective reward function, which includes a frequency stability reward function and a charge state maintenance reward function.

[0078] Design a composite reward function R=αR1+βR2, and rationally select the reward weight coefficients according to the control objective.

[0079] Frequency stability reward function R1:

[0080] in, As a reward for frequency stability; ε=1e-5 to prevent division by zero in the denominator; when Δf>0.01pu, an exponential penalty is applied, where k is the penalty coefficient, which is -10.

[0081] The reward function R2 for maintaining the state of charge:

[0082] In the formula, The weighting is dynamic and adjusted based on the battery health status (SOH). Rewards for maintaining the state of charge; Penalty for exceeding the state of charge limit; η is the attenuation coefficient; Describes the maximum allowable output / input power of the energy storage unit.

[0083] In this embodiment, the multi-objective reward function of DDPG can correspond to the multi-objective function of MPC, making the power compensation command of reinforcement learning more aligned with the global optimization objective of MPC. When the two work together, they can more efficiently balance model constraints and dynamic adaptation, further improving the overall control performance.

[0084] In some embodiments of this application, the DDPG-based reinforcement learning agent inputs the frequency deviation and state of charge of the next preset number of control cycles predicted by the MPC-based mechanism model and the LSTM-based load prediction model into the value network, and optimizes its network parameters by minimizing the prediction error.

[0085] Specifically, a composite prediction model is introduced, consisting of an LSTM-based load forecasting module and a mechanism-based state-space prediction model, to predict frequency deviation and SOC changes over the next 3-5 cycles. This enables the Critic network to assess the long-term impact of actions and improves the accuracy of the reward function. The composite prediction model is jointly trained with DDPG, and parameters are optimized by minimizing prediction errors, enhancing the algorithm's ability to predict power grid dynamics.

[0086] Furthermore, the reinforcement learning agent based on DDPG employs a priority-based experience replay mechanism, assigning high sampling probabilities to high-reward or large-error samples that experience sudden increases in frequency deviation or exceed load limits, and using these samples for DDPG network updates. For example, when frequency deviation suddenly increases or the SOC exceeds limits, the corresponding samples are given higher priority, ensuring that the agent focuses on learning coping strategies under extreme conditions.

[0087] In some embodiments of this application, the DDPG-based reinforcement learning agent is trained offline based on the following process: Initialize the parameters of the policy network, value network, and target network; Generate simulation conditions to simulate the dynamic fluctuations of the primary frequency regulation of the power grid; Control the intelligent agent to perform actions in the simulation environment, collect experience data and store it in the playback pool; Every 100 steps, 128 data points are sampled from the replay pool, and the value network and policy network are updated using gradient descent. Every 1000 steps, the target network and the main network are synchronized via a soft update.

[0088] Furthermore, the DDPG-based reinforcement learning agent is fine-tuned online using the following process: Every preset control cycle, new measured data is sent back to the playback pool for incremental training, enabling the agent to adapt to the drift of actual power grid characteristics.

[0089] The energy storage frequency regulation method proposed in this application is demonstrated below by combining experimental data: A system based on the MATLAB / Simulink platform is constructed as follows... Figure 1 , 3 The system model of energy storage-assisted frequency regulation shown in Figure -4 is used to train the data-driven model based on the generated simulation conditions (load variation range is 5% of the baseline value). The system power baseline value is set to 1000MW and the frequency baseline value is 50HZ. The following frequencies and system output units are all standardized. Figure 5 The frequency deviation is 0.02 pu under a step response. Figure 6 In the figure, (a) represents the total output of each system under a step disturbance; Figure 6 (b) represents the output of each thermal power unit under a step disturbance; Figure 6 (c) represents the output of each energy storage unit under a step disturbance; Figure 7 The system frequency deviation under randomly generated continuous disturbances; Figure 8 In the figure, (a) represents the total output of each system under continuous disturbance; Figure 8 (b) represents the output of each system's thermal power unit under continuous disturbance; Figure 8 (c) represents the output of each energy storage unit under continuous disturbance; Figure 9This refers to the change in the state of charge of energy storage under continuous disturbances. For example... Figures 5-9 As shown, energy storage participating in frequency regulation auxiliary services effectively reduces the frequency regulation pressure of traditional thermal power units, and the MPC-DDPG method can make fuller use of the speed of energy storage units, maximize the output of energy storage units and perform frequency regulation within a reasonable range, thereby further reducing the system frequency deviation.

[0090] In summary, the energy storage frequency regulation method proposed in this application has the following technical effects: 1) Improved model adaptability and control accuracy To address the issue of traditional MPC's excessive reliance on model accuracy, this application introduces a reinforcement learning mechanism to enable the control system to possess dynamic adaptability. Simulation data shows that the frequency regulation deviation of the system under this application is effectively reduced compared to the traditional MPC algorithm. This improvement stems from the fact that the reinforcement learning policy network can compensate for model prediction errors in real time, effectively solving the mismatch problem of traditional linearized models by adaptively adjusting control parameters.

[0091] 2) Enhanced ability to handle nonlinear characteristics To address the control challenges posed by the nonlinear characteristics of the system, this application innovatively constructs a two-layer optimization structure of "MPC framework + reinforcement learning compensation". Experiments show that in complex frequency regulation scenarios involving the dynamic characteristics of thermal power units, the proposed algorithm shortens the regulation response time and reduces dynamic frequency deviation. This is attributed to the Actor-Critic network of reinforcement learning, which can effectively learn the nonlinear characteristics of the system and guide the MPC optimization process through the value function. In summary, this method demonstrates superior consistency in tracking the dynamic response of the unit.

[0092] 3) Improved robustness under uncertain environments To address the multiple uncertainties inherent in frequency modulation scenarios, the improved DDPG algorithm in this application significantly enhances the system's anti-interference capability by introducing an environment prediction module. This advantage stems from the reinforcement learning experience replay mechanism's ability to continuously accumulate optimization strategies under different operating conditions, forming an adaptive knowledge base.

[0093] In some embodiments of this application, an energy storage frequency regulation control device is provided, which corresponds one-to-one with the energy storage frequency regulation control method in the above embodiments. For example... Figure 10 As shown, the energy storage frequency regulation control device includes a prediction module 101, a first output module 102, a second output module 103, a control module 104, and a feedback module 105.

[0094] The prediction module 101 is used to predict system state variables in each control cycle using an MPC-based mechanism model and an LSTM-based load prediction model. The first output module 102 is used to predict the optimal power allocation command in the future control time domain based on system state variables and load disturbance state variables through a model predictive controller based on MPC, and output the basic power command at the current moment. The second output module 103 is used to output power compensation commands based on system state variables and load disturbance state variables through a reinforcement learning agent based on DDPG. The control module 104 is used to superimpose the basic power command output by the model prediction controller with the power compensation command output by the reinforcement learning agent module to generate a target control command and apply it to the controlled object. Feedback module 105 is used to obtain the output state of the controlled object based on the target control command, and send the output state as a feedback signal to the MPC-based mechanism model and the LSTM-based load prediction model for feedback correction, and to the reinforcement learning agent module for policy update.

[0095] In some embodiments of this application, in the above-described apparatus, the output vector of the MPC-based mechanism model includes grid frequency deviation and the combined state of charge of hybrid energy storage; wherein, the grid frequency deviation is determined based on the power output of thermal power units, the power output of battery energy storage, the power output of supercapacitors, and system load disturbances; the power output of hybrid energy storage is determined based on the state of charge of battery energy storage and the state of charge of the super battery energy storage system.

[0096] In some embodiments of this application, in the above-described device, the MPC-based model predictive controller is optimized based on a multi-objective function, which includes: a frequency deviation minimization term; a state-of-charge deviation minimization term; and an economic cost minimization term, wherein the economic cost includes coal consumption cost of thermal power units, environmental governance cost, energy storage loss cost, and operation and maintenance cost. The constraints of the MPC-based model predictive controller include thermal power unit ramping constraints, energy storage charging and discharging power constraints, and energy storage state of charge constraints.

[0097] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent includes a policy network and a value network; the policy network serves as a policy function, used to generate continuous power compensation instructions based on the system state mapping; the value network serves as an evaluator of the reward function, used to evaluate the long-term value of actions and guide the policy optimization of the policy network.

[0098] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent includes a hierarchical state encoder, which employs a CNN-LSTM hybrid network. The input vector of the hierarchical state encoder includes the current frequency deviation, frequency change rate, energy storage state of charge, state of charge deviation, and load allocation command prediction value. After being encoded by the hierarchical state encoder, the vector is input to the policy network and value network of the DDPG-based reinforcement learning agent.

[0099] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent employs a multi-objective reward function, which includes a frequency stability reward function and a state-of-charge maintenance reward function.

[0100] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent, through the frequency deviation and state of charge of the future preset number of control cycles predicted by the MPC-based mechanistic model and the LSTM-based load prediction model, inputs them into the value network, thereby optimizing its network parameters by minimizing the prediction error.

[0101] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent employs a priority experience replay mechanism to assign high sampling probabilities to high-reward or large-error samples with sudden increases in frequency deviation or overload conditions, and uses them for DDPG network updates.

[0102] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent is trained offline based on the following process: Initialize the parameters of the policy network, value network, and target network; Generate simulation conditions to simulate the dynamic fluctuations of the primary frequency regulation of the power grid; Control the intelligent agent to perform actions in the simulation environment, collect experience data and store it in the playback pool; Every 100 steps, 128 data points are sampled from the replay pool, and the value network and policy network are updated using gradient descent. Every 1000 steps, the target network and the main network are synchronized via a soft update.

[0103] In some embodiments of this application, in the above-described apparatus, the DDPG-based reinforcement learning agent is trained online using the following process: Every preset control cycle, new measured data is sent back to the playback pool for incremental training, enabling the agent to adapt to the drift of actual power grid characteristics.

[0104] It should be noted that any of the above-mentioned energy storage frequency regulation control devices can implement the aforementioned energy storage frequency regulation control method one by one, which will not be elaborated here.

[0105] Figure 11 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Figure 11 As shown, at the hardware level, this electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may include non-volatile memory, such as at least one disk drive. Of course, this electronic device may also include other hardware required for other business operations.

[0106] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 11 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0107] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0108] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming an energy storage frequency modulation control device at the logical level. The processor executes the program stored in memory and specifically performs the aforementioned method.

[0109] The processor may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0110] This electronic device can execute the energy storage frequency modulation control method provided in several embodiments of this application, and is implemented as an energy storage frequency modulation control device. Figure 10 The functions of the embodiments shown are not described in detail here.

[0111] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform the energy storage frequency modulation control method provided in several embodiments of this application.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0117] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0118] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0119] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for frequency regulation control of energy storage, characterized in that, The method includes: Within each control cycle, the MPC-based mechanism model and the LSTM-based load forecasting model predict the system state variables. Based on system state variables and load disturbance state variables, the model predictive controller based on MPC predicts the optimal power allocation command in the future control time domain and outputs the basic power command at the current moment. Based on system state variables and load disturbance state variables, a power compensation command is output through a reinforcement learning agent based on DDPG. The basic power command output by the model prediction controller is superimposed with the power compensation command output by the reinforcement learning agent module to generate a target control command and apply it to the controlled object. The output state of the controlled object based on the target control command is obtained, and the output state is used as a feedback signal to be sent to the MPC-based mechanism model and the LSTM-based load prediction model for feedback correction, and to the reinforcement learning agent module for policy update.

2. The method according to claim 1, characterized in that, The output vector of the MPC-based mechanism model includes the grid frequency deviation and the combined state of charge of the hybrid energy storage; wherein, the grid frequency deviation is determined based on the power output of thermal power units, the power output of battery energy storage, the power output of supercapacitors, and system load disturbances; the power output of the hybrid energy storage is determined based on the state of charge of battery energy storage and the state of charge of the super battery energy storage system.

3. The method according to claim 1 or 2, characterized in that, The MPC-based model predictive controller is optimized based on a multi-objective function, which includes: a frequency deviation minimization term; a state-of-charge deviation minimization term; and an economic cost minimization term, whereby the economic cost includes coal consumption cost of thermal power units, environmental governance cost, energy storage loss cost, and operation and maintenance cost. The constraints of the MPC-based model predictive controller include thermal power unit ramping constraints, energy storage charging and discharging power constraints, and energy storage state of charge constraints.

4. The method according to claim 1, characterized in that, The DDPG-based reinforcement learning agent includes a policy network and a value network; the policy network, as a policy function, is used to generate continuous power compensation instructions based on the system state mapping; the value network, as an evaluator of the reward function, is used to evaluate the long-term value of actions and guide the policy optimization of the policy network.

5. The method according to claim 4, characterized in that, The DDPG-based reinforcement learning agent includes a hierarchical state encoder, which employs a CNN-LSTM hybrid network. The input vector of the hierarchical state encoder includes the current frequency deviation, frequency change rate, energy storage state of charge, state of charge deviation, and load allocation command prediction value. After being encoded by the hierarchical state encoder, the vector is input to the policy network and value network of the DDPG-based reinforcement learning agent.

6. The method according to claim 1, characterized in that, The reinforcement learning agent based on DDPG employs a multi-objective reward function, which includes a frequency stability reward function and a state-of-charge maintenance reward function.

7. The method according to claim 1, characterized in that, The DDPG-based reinforcement learning agent inputs the frequency deviation and state of charge of the next preset number of control cycles predicted by the MPC-based mechanism model and the LSTM-based load prediction model into the value network, and optimizes its network parameters by minimizing the prediction error.

8. The method according to claim 1, characterized in that, The reinforcement learning agent based on DDPG adopts a priority experience replay mechanism, which assigns high sampling probability to high reward or large error samples with sudden increases in frequency deviation or overload conditions, and uses them for DDPG network updates.

9. The method according to claim 1, characterized in that, The DDPG-based reinforcement learning agent is trained offline using the following process: Initialize the parameters of the policy network, value network, and target network; Generate simulation conditions to simulate the dynamic fluctuations of the primary frequency regulation of the power grid; Control the intelligent agent to perform actions in the simulation environment, collect experience data and store it in the playback pool; Every 100 steps, 128 data points are sampled from the replay pool, and the value network and policy network are updated using gradient descent. Every 1000 steps, the target network and the main network are synchronized via a soft update.

10. The method according to claim 1, characterized in that, The DDPG-based reinforcement learning agent is trained online using the following process: Every preset control cycle, new measured data is sent back to the playback pool for incremental training, enabling the agent to adapt to the drift of actual power grid characteristics.

11. An energy storage frequency regulation control device, characterized in that, The device includes: The prediction module is used to predict system state variables in each control cycle using an MPC-based mechanistic model and an LSTM-based load prediction model. The first output module is used to predict the optimal power allocation command in the future control time domain based on system state variables and load disturbance state variables, and output the basic power command at the current moment. The second output module is used to output power compensation commands based on system state variables and load disturbance state variables through a reinforcement learning agent based on DDPG. The control module is used to superimpose the basic power command output by the model prediction controller with the power compensation command output by the reinforcement learning agent module to generate a target control command and apply it to the controlled object. The feedback module is used to obtain the output state of the controlled object based on the target control command, and send the output state as a feedback signal to the MPC-based mechanism model and the LSTM-based load prediction model for feedback correction, and to the reinforcement learning agent module for policy update.

12. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the energy storage frequency modulation control method as described in any one of claims 1-10.

13. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the steps of the energy storage frequency modulation control method as described in any one of claims 1-10.

Images