Energy storage system deep reinforcement learning frequency modulation control method based on health state perception

Abstract

The invention discloses an energy storage system deep reinforcement learning frequency modulation control method based on health state perception, and relates to the field of power system frequency modulation. The method comprises the steps of S1, collecting and preprocessing power grid frequency deviation, load disturbance and energy storage SOC, SOH and power data; s2, constructing a DRL control framework of MDP modeling, and defining a state space containing SOH, a continuous action space and a multi-target composite reward function; s3, adopting a DDPG algorithm to train the model, and optimizing an energy storage power instruction through an Actor-Critic network to meet frequency and energy storage security constraints; and S4, evaluating a frequency index and an energy storage SOH attenuation rate through step and fluctuation disturbance verification. Through SOH perception and DRL optimization, the problems that traditional control response is slow and energy storage life is ignored are solved, balance between frequency stability and energy storage life is achieved, and the method is suitable for primary frequency modulation of a high-proportion new energy power grid and has remarkable engineering application value.

Description

Technical Field

[0001] This invention relates to the field of frequency regulation control in power systems, and in particular to a deep reinforcement learning-based frequency regulation control method for energy storage systems based on health status awareness. Background Technology

[0002] As the global energy structure shifts towards cleaner and lower-carbon energy, the installed capacity of renewable energy sources such as wind power and photovoltaics is growing rapidly. In 2023, the proportion of new energy in my country's total installed capacity exceeded 40%. However, the intermittency and volatility of new energy power generation lead to a significant decrease in the equivalent inertia of the power grid and increasingly complex frequency fluctuation characteristics, which places higher demands on the primary frequency regulation performance of the system.

[0003] Energy storage systems, with their advantages of fast response, flexible power regulation, and bidirectional operation, have become key equipment for assisting thermal power units in primary frequency regulation and improving the dynamic characteristics of system frequency. Traditional energy storage frequency regulation strategies mainly rely on vertical control and PID control. While these methods are simple in structure, they have inherent drawbacks: fixed control parameters, poor robustness, and difficulty in adapting to complex operating conditions such as load disturbances and fluctuations in renewable energy output; moreover, they do not consider the degradation of the energy storage's state of health (SOH), and frequent high-power charging and discharging can easily lead to a shortened energy storage life and increased operating costs.

[0004] In recent years, deep reinforcement learning (DRL) technology has been widely used in the control of complex dynamic systems due to its integration of the feature extraction capabilities of deep neural networks and the policy optimization mechanism of reinforcement learning. However, existing research on DRL-based energy storage frequency regulation still has shortcomings: most studies only use grid frequency deviation and energy storage state of charge (SOC) as input features, neglecting the impact of SOC on the long-term economic efficiency of frequency regulation; the reward function design is simplistic and cannot balance frequency stability, energy storage degradation, and operating costs; some algorithms exhibit slow convergence and are prone to oscillations during training, making it difficult to meet the real-time requirements of actual power grids.

[0005] Therefore, there is an urgent need for a DRL frequency regulation control method that integrates energy storage health status perception and multi-objective optimization reward function, which can improve frequency regulation performance while delaying the decay of energy storage SOH, and provide technical support for the safe and stable operation of a high proportion of new energy power grids. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of existing energy storage frequency regulation control methods, such as slow response speed, neglect of energy storage health status, and insufficient multi-objective balance. It provides a deep reinforcement learning frequency regulation control method for energy storage systems based on health status perception, and realizes adaptive adjustment of energy storage power through the DRL algorithm, taking into account both grid frequency stability and energy storage life maintenance.

[0007] The technical solution of the present invention is as follows:

[0008] A deep reinforcement learning-based frequency regulation control method for energy storage systems based on health status awareness includes the following steps:

[0009] Step S1, Data Collection and Preprocessing: Collect grid frequency deviation, load disturbance data, and energy storage system state of charge (SOC), state of health (SOH), and charge / discharge power data; perform Butterworth low-pass filtering for noise reduction, detrending processing, and normalization to the [0,1] interval to form a standardized time-series input;

[0010] Step S2: Construct the DRL frequency modulation control framework: modeled using a Markov decision process (MDP), including:

[0011] State space: includes grid frequency deviation, SOC deviation, SOH deviation (i.e., the difference between the initial SOH and the current SOH) and real-time charging and discharging power;

[0012] Action space: Continuous charging and discharging power commands for the energy storage system;

[0013] Composite reward function: integrates frequency stability reward term, SOH decay penalty term, operating cost term and out-of-bounds penalty term, and dynamically balances multi-objective optimization through weight coefficients;

[0014] Step S3, DRL Model Training and Deployment: The Deep Deterministic Policy Gradient (DDPG) algorithm is adopted. Power commands are generated through the Actor network and the Critic network evaluates the value of actions to minimize the loss function of the composite reward. During training, frequency security constraints and energy storage SOH decay constraints are forcibly satisfied. After training, the model is deployed to the actual power grid and outputs adaptive power commands.

[0015] Step S4, Performance Verification: Under load step disturbance and continuous fluctuation scenarios, verify the frequency regulation performance (maximum frequency deviation, ITAE) and energy storage lifetime index (SOH decay rate) of the DRL model. The requirements are: compared with traditional droop control, the maximum frequency deviation is reduced by ≥8%, ITAE is reduced by ≥25%, and SOH monthly decay rate is reduced by ≥10%.

[0016] Furthermore, the specific calculation model for the energy storage SOH in step S1 is as follows:

[0017] (1)

[0018] in, For the current available energy storage capacity, This represents the initial rated capacity of the energy storage.

[0019] Battery degradation model:

[0020] (2)

[0021] in, This is the capacity decay coefficient. Energy storage system charging and discharging power, This refers to the depth of charge / discharge.

[0022] Furthermore, the specific definitions of the state space and action space in step S2 are as follows:

[0023] state space vector ,in:

[0024] This is the power grid frequency deviation, which is the difference between the current frequency and the nominal frequency, in Hz.

[0025] This refers to the state of charge deviation of energy storage, which is the difference between the current SOC and the target SOC, and has no unit.

[0026] This represents the energy storage health status deviation, which is the difference between the initial SOH (value 1.0) and the current SOH, without units.

[0027] This refers to the real-time charging and discharging power of energy storage, with discharge being positive and charging being negative. The unit is pu, and the rated power of energy storage is used as the benchmark.

[0028] All state variables are input into the DRL model after being normalized in step S1.

[0029] The action space is the continuous charge and discharge power command of the energy storage system. ,in The target charging and discharging power of the energy storage is set; the action command is issued through the communication interface between the power grid and the energy storage to control the energy storage converter to perform power regulation.

[0030] Furthermore, the specific forms of the composite reward function and the cumulative discount reward in step S2 are as follows:

[0031] Composite reward function:

[0032] (3)

[0033] in, These are weighting coefficients used to balance the priorities of various objectives;

[0034] Frequency performance items:

[0035] (4)

[0036] in, For system frequency deviation, To maximize the allowable deviation, the reward tends to be maximized when the system frequency is close to the rated value, thereby guiding the agent to prioritize frequency stability as the primary optimization objective.

[0037] Health penalty items:

[0038] (5)

[0039] in, Initial healthy state, The current health status is directly related to the SOH deviation, and the penalty increases non-linearly as the SOH deviation increases.

[0040] Operating cost item:

[0041] (6)

[0042] in, For energy storage charging and discharging power, Cost per unit of energy consumption This refers to the rated capacity of the energy storage.

[0043] Penalties for crossing boundaries:

[0044] (7)

[0045] This item is triggered when the SOC or power exceeds the limit, and outputs a fixed negative reward to enforce safety boundaries.

[0046] Cumulative Discount Rewards:

[0047] (8)

[0048] in, This is a discount factor used to balance short-term and long-term returns. To return the reward;

[0049] Furthermore, the specific training process of the DRL model, i.e., DDPG, in step S3 is as follows:

[0050] Network architecture design:

[0051] Actor Policy Network: The input layer has 4 neurons, which is consistent with the dimension of the state space. There are 3 hidden layers, with 128 neurons in the first 2 layers and 64 neurons in the 3rd layer. The output layer has 1 neuron corresponding to the energy storage power instruction. The activation function is ReLU.

[0052] Critic Value Network: The input layer consists of a concatenation of the state vector and the action vector, with a total of 5 neurons. There are 3 hidden layers, with 256 neurons in the first 2 layers and 128 neurons in the 3rd layer. The output layer has 1 neuron, which is the action value. The activation function used is ReLU.

[0053] Training parameter settings: The Adam optimization algorithm is used to update network parameters. The learning rate of the Actor network is set to 0.001, and the learning rate of the Critic network is set to 0.002; the capacity of the experience replay pool is set to... The batch size for each sampling is 64.

[0054] Training Iteration Process: The agent interacts with a power grid simulation environment containing a thermal power unit-energy storage joint frequency regulation model, storing state S, action a, reward R, and next state S' in an experience replay pool. In each iteration, batch data is sampled from the replay pool, and the loss between the target value and the current value is calculated through the Critic network. The Critic parameters are then updated via backpropagation. The Actor parameters are updated using the policy gradient method to maximize the expected reward. Iterative training continues until the loss function converges, i.e., the MSE loss is less than [a certain value]. Or the number of training steps reaches step.

[0055] Furthermore, the specific operational constraints of the frequency modulation control in step S3 are as follows:

[0056] (9)

[0057] (10)

[0058] (11)

[0059] (12)

[0060] In the formula, These are the minimum and maximum frequency deviations allowed by the system, respectively; These are the lower and upper limits of the energy storage state of charge, respectively, to prevent overcharging and over-discharging; These represent the lower and upper limits of energy storage health status, respectively. When the health level is below 0.8, energy storage enters the decommissioning assessment stage. This sets the upper limit of the rated charging and discharging power of the energy storage system to ensure the safe operation of the equipment within the rated range.

[0061] Furthermore, in step S4, the performance verification uses an independent test dataset (including load step disturbance and continuous fluctuation disturbance scenarios) to verify the model, and the evaluation metrics are as follows:

[0062] Frequency regulation performance indicators:

[0063] Maximum frequency deviation: The maximum frequency deviation under DRL control is reduced by ≥8% compared to traditional droop control;

[0064] The ITAE metric, or Time-based integral of absolute error, is as follows:

[0065] (13)

[0066] in, The duration of the disturbance. for Frequency deviation at specific times; The ITAE value under DRL control is reduced by ≥25% compared to traditional droop control;

[0067] The frequency recovery time under DRL control is ≥4% shorter than that under traditional droop control;

[0068] Energy storage performance indicators:

[0069] SOH decay rate: The monthly SOH decay rate under DRL control is ≥10% lower than that under traditional droop control;

[0070] Average charge / discharge power: The average output power of energy storage under DRL control is reduced by ≥5% compared with traditional droop control;

[0071] If the above indicators are met simultaneously, the model is deemed to have qualified performance and can be put into practical application.

[0072] Compared with existing solutions, the present invention, employing the above-described technical solution, has the following beneficial effects:

[0073] 1. Balancing frequency stability and energy storage lifespan, delaying the decline in health status.

[0074] By incorporating the State of Health (SOH) of energy storage into the state space and designing a SOH decay penalty term, this patent effectively constrains the overcharging and discharging behavior of energy storage while optimizing grid frequency regulation. Compared to traditional solutions that only focus on State of Charge (SOC), this patent can reduce the monthly SOH decay rate by ≥10%, significantly extending the service life of energy storage equipment and reducing operation and maintenance costs.

[0075] 2. Multi-objective collaborative optimization enhances control flexibility and economy.

[0076] A composite reward function integrating frequency stability, state of equilibrium (SOH) attenuation, operating cost, and out-of-bounds constraints is constructed, and multiple objectives are prioritized through dynamic weighting. This ensures frequency regulation performance with a maximum frequency deviation reduction of ≥8% and an ITAE reduction of ≥25%, while optimizing energy storage operating costs, avoiding SOC or power outage risks, and adapting to the complex operating conditions of high-proportion renewable energy grids.

[0077] 3. High control precision, rapid response, and strong engineering applicability.

[0078] The Actor-Critic network architecture based on the DDPG algorithm, combined with data preprocessing procedures such as Butterworth filtering and normalization, enables adaptive and precise control of energy storage charging and discharging power. Compared with traditional droop control, the frequency recovery time is reduced by ≥4%, and it can quickly suppress frequency oscillations in both load step and continuous fluctuation scenarios. Moreover, the network training parameters are clear and the deployment process is straightforward, making it directly valuable for engineering applications. Attached Figure Description

[0079] Figure 1 A flowchart of a deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness;

[0080] Figure 2 This is a primary frequency regulation model for a regional power grid containing a combined thermal power and energy storage frequency regulation system.

[0081] Figure 3 For energy storage frequency regulation control framework;

[0082] Figure 4 Simulations were performed to compare the effects of frequency control.

[0083] Figure 5 For comparison of energy storage output;

[0084] Figure 6 For energy storage frequency regulation control framework;

[0085] Figure 7 For comparison of energy storage output; Detailed Implementation

[0086] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and simulation experiments. The technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0087] A deep reinforcement learning-based frequency regulation control method for energy storage systems based on health status awareness, such as Figure 1 The steps shown are as follows:

[0088] Step S1, Data Collection and Preprocessing: Collect grid frequency deviation, load disturbance data, and energy storage system state of charge (SOC), state of health (SOH), and charge / discharge power data; perform Butterworth low-pass filtering for noise reduction, detrending processing, and normalization to the [0,1] interval to form a standardized time-series input;

[0089] Step S2: Construct the DRL frequency modulation control framework: modeled using a Markov decision process (MDP), including:

[0090] State space: includes grid frequency deviation, SOC deviation, SOH deviation (i.e., the difference between the initial SOH and the current SOH) and real-time charging and discharging power;

[0091] Action space: Continuous charging and discharging power commands for the energy storage system;

[0092] Composite reward function: integrates frequency stability reward term, SOH decay penalty term, operating cost term and out-of-bounds penalty term, and dynamically balances multi-objective optimization through weight coefficients;

[0093] Step S3, DRL Model Training and Deployment: The Deep Deterministic Policy Gradient (DDPG) algorithm is adopted. Power commands are generated through the Actor network and the Critic network evaluates the value of actions to minimize the loss function of the composite reward. During training, frequency security constraints and energy storage SOH decay constraints are forcibly satisfied. After training, the model is deployed to the actual power grid and outputs adaptive power commands.

[0094] Step S4, Performance Verification: Under load step disturbance and continuous fluctuation scenarios, verify the frequency regulation performance (maximum frequency deviation, ITAE) and energy storage lifetime index (SOH decay rate) of the DRL model. The requirements compared to traditional droop control are: maximum frequency deviation reduced by ≥8%, ITAE reduced by ≥25%, and SOH monthly decay rate reduced by ≥10%.

[0095] Furthermore, the specific calculation model for the energy storage SOH in step S1 is as follows:

[0096] (1)

[0097] in, For the current available energy storage capacity, This represents the initial rated capacity of the energy storage.

[0098] Battery degradation model:

[0099] (2)

[0100] in, This is the capacity decay coefficient. Energy storage system charging and discharging power, This refers to the depth of charge / discharge.

[0101] Furthermore, the specific definitions of the state space and action space in step S2 are as follows:

[0102] state space vector ,in:

[0103] This is the power grid frequency deviation, which is the difference between the current frequency and the nominal frequency, in Hz.

[0104] This refers to the state of charge deviation of energy storage, which is the difference between the current SOC and the target SOC, and has no unit.

[0105] This represents the energy storage health status deviation, which is the difference between the initial SOH (value 1.0) and the current SOH, without units.

[0106] This refers to the real-time charging and discharging power of energy storage, with discharge being positive and charging being negative. The unit is pu, and the rated power of energy storage is used as the benchmark.

[0107] All state variables are input into the DRL model after being normalized in step S1.

[0108] The action space is the continuous charge and discharge power command of the energy storage system. ,in The target charging and discharging power of the energy storage is set; the action command is issued through the communication interface between the power grid and the energy storage to control the energy storage converter to perform power regulation.

[0109] Furthermore, the specific forms of the composite reward function and the cumulative discount reward in step S2 are as follows:

[0110] Composite reward function:

[0111] (3)

[0112] in, These are weighting coefficients used to balance the priorities of various objectives;

[0113] Frequency performance items:

[0114] (4)

[0115] in, For system frequency deviation, To maximize the allowable deviation, the reward tends to be maximized when the system frequency is close to the rated value, thereby guiding the agent to prioritize frequency stability as the primary optimization objective.

[0116] Health penalty items:

[0117] (5)

[0118] in, Initial healthy state, The current health status is directly related to the SOH deviation, and the penalty increases non-linearly as the SOH deviation increases.

[0119] Operating cost item:

[0120] (6)

[0121] in, For energy storage charging and discharging power, Cost per unit of energy consumption This refers to the rated capacity of the energy storage.

[0122] Penalties for crossing boundaries:

[0123] (7)

[0124] This item is triggered when the SOC or power exceeds the limit, and outputs a fixed negative reward to enforce safety boundaries.

[0125] Cumulative Discount Rewards:

[0126] (8)

[0127] in, This is a discount factor used to balance short-term and long-term returns. To return the reward;

[0128] Furthermore, the specific training process of the DRL model, i.e., DDPG, in step S3 is as follows:

[0129] Network architecture design:

[0130] Actor Policy Network: The input layer has 4 neurons, which is consistent with the dimension of the state space. There are 3 hidden layers, with 128 neurons in the first 2 layers and 64 neurons in the 3rd layer. The output layer has 1 neuron corresponding to the energy storage power instruction. The activation function is ReLU.

[0131] Critic Value Network: The input layer consists of a concatenation of the state vector and the action vector, with a total of 5 neurons. There are 3 hidden layers, with 256 neurons in the first 2 layers and 128 neurons in the 3rd layer. The output layer has 1 neuron, which is the action value. The activation function used is ReLU.

[0132] Training parameter settings: The Adam optimization algorithm is used to update network parameters. The learning rate of the Actor network is set to 0.001, and the learning rate of the Critic network is set to 0.002; the capacity of the experience replay pool is set to... The batch size for each sampling is 64.

[0133] Training Iteration Process: The agent interacts with a power grid simulation environment containing a thermal power unit-energy storage joint frequency regulation model, storing state S, action a, reward R, and next state S' in an experience replay pool. In each iteration, batch data is sampled from the replay pool, and the loss between the target value and the current value is calculated through the Critic network. The Critic parameters are then updated via backpropagation. The Actor parameters are updated using the policy gradient method to maximize the expected reward. Iterative training continues until the loss function converges, i.e., the MSE loss is less than [a certain value]. Or the number of training steps reaches step.

[0134] Furthermore, the specific operational constraints of the frequency modulation control in step S3 are as follows:

[0135] (9)

[0136] (10)

[0137] (11)

[0138] (12)

[0139] In the formula, These are the minimum and maximum frequency deviations allowed by the system, respectively; These are the lower and upper limits of the energy storage state of charge, respectively, to prevent overcharging and over-discharging; These represent the lower and upper limits of energy storage health status, respectively. When the health level is below 0.8, energy storage enters the decommissioning assessment stage. This sets the upper limit of the rated charging and discharging power of the energy storage system to ensure the safe operation of the equipment within the rated range.

[0140] Furthermore, in step S4, the performance verification uses an independent test dataset (including load step disturbance and continuous fluctuation disturbance scenarios) to verify the model, and the evaluation metrics are as follows:

[0141] Frequency regulation performance indicators:

[0142] Maximum frequency deviation: The maximum frequency deviation under DRL control is reduced by ≥8% compared to traditional droop control;

[0143] The ITAE metric, or Time-based integral of absolute error, is as follows:

[0144] (13)

[0145] in, The duration of the disturbance. for Frequency deviation at specific times; The ITAE value under DRL control is reduced by ≥25% compared to traditional droop control;

[0146] The frequency recovery time under DRL control is ≥4% shorter than that under traditional droop control;

[0147] Energy storage performance indicators:

[0148] SOH decay rate: The monthly SOH decay rate under DRL control is ≥10% lower than that under traditional droop control;

[0149] Average charge / discharge power: The average output power of energy storage under DRL control is reduced by ≥5% compared with traditional droop control;

[0150] If the above indicators are met simultaneously, the model is deemed to have qualified performance and can be put into practical application.

[0151] Specific embodiments of the present invention are as follows:

[0152] 1. Experimental setup

[0153] A joint frequency regulation model of thermal power unit and energy storage was built based on MATLAB R2024b / Simulink. The hardware platform is an Intel(R) Core(TM) Ultra5125H 1.20GHz processor and 32GB of memory.

[0154] Model parameters:

[0155] Thermal power units: governor time constant High-pressure cylinder front steam chamber volume time constant Reheater volumetric time constant Reheat steam gain Primary frequency modulation valve position factor ;

[0156] Energy storage system: rated power 10MW, capacity Response time constant SOC range 20%~80%, power change rate ≤15% rated power / s;

[0157] Power grid parameters: inertial time constant Equivalent damping constant

[0158] Disturbance scenario:

[0159] Scene 1: Add a 0.1 pu load step disturbance at the time;

[0160] Scenario 2: Load fluctuation disturbance lasting 200 seconds (simulating random load changes in the actual power grid).

[0161] 2. Experimental Results and Analysis

[0162] Load step disturbance results

[0163] The performance metrics of the two control strategies are compared in Table 1:

[0164]

[0165] As shown in Table 1, under step disturbance, the method of the present invention reduces the maximum frequency difference by 8.3%, effectively suppressing transient frequency shift; reduces ITAE by 26.6%, significantly reducing the overall dynamic error; and reduces energy storage output by 50%, making energy utilization more efficient.

[0166] Load fluctuation disturbance results

[0167] The performance metrics of the two control strategies are compared in Table 2:

[0168]

[0169] As shown in Table 2, under fluctuation disturbances, the method of the present invention reduces the peak-to-peak frequency by 74.1%, significantly narrowing the fluctuation range; reduces ITAE by 78.7%, greatly improving dynamic stability; and reduces the monthly SOH decay rate by 12.5%, effectively delaying the decay of energy storage life.

[0170] In summary, the method of this invention, through health status perception and deep reinforcement learning, can achieve synergistic optimization of frequency regulation performance and energy storage life under different disturbance scenarios, which is superior to traditional droop control and is suitable for the primary frequency regulation needs of high-proportion renewable energy power grids.

[0171] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0172] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning frequency modulation control method for energy storage systems based on health status perception, characterized in that, Includes the following steps: Step S1, Data Collection and Preprocessing: Collect grid frequency deviation, load disturbance data, and energy storage system state of charge (SOC), state of health (SOH), and charge / discharge power data; perform Butterworth low-pass filtering for noise reduction, detrending processing, and normalization to the [0,1] interval to form a standardized time-series input; Step S2: Construct the DRL frequency modulation control framework: modeled using a Markov decision process (MDP), including: State space: includes grid frequency deviation, SOC deviation, SOH deviation (i.e., the difference between the initial SOH and the current SOH) and real-time charging and discharging power; Action space: Continuous charging and discharging power commands for the energy storage system; Composite reward function: integrates frequency stability reward term, SOH decay penalty term, operating cost term and out-of-bounds penalty term, and dynamically balances multi-objective optimization through weight coefficients; Cumulative Discount Rewards: These are weighted sums of composite rewards at each time step, with the weights determined by the discount factor. They balance short-term and long-term returns and aim to maximize total returns. Step S3, DRL Model Training and Deployment: The Deep Deterministic Policy Gradient (DDPG) algorithm is adopted. Power commands are generated through the Actor network and the Critic network evaluates the value of actions to minimize the loss function of the composite reward. During training, frequency security constraints and energy storage SOH decay constraints are forcibly satisfied. After training, the model is deployed to the actual power grid and outputs adaptive power commands. Step S4, Performance Verification: Under load step disturbance and continuous fluctuation scenarios, verify the frequency regulation performance of the DRL model, namely the maximum frequency deviation, ITAE and energy storage lifetime index SOH decay rate. The requirements are compared with traditional droop control: maximum frequency deviation reduced by ≥8%, ITAE reduced by ≥25%, and SOH monthly decay rate reduced by ≥10%.

2. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness as described in claim 1, characterized in that... The specific calculation model for the energy storage SOH in step S1 is as follows: (1) in, For the current available energy storage capacity, This represents the initial rated capacity of the energy storage. Battery degradation model: (2) in, This is the capacity decay coefficient. Energy storage system charging and discharging power, This refers to the depth of charge / discharge.

3. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific definition of the state space in step S2 is as follows: state space vector ,in: This is the power grid frequency deviation, which is the difference between the current frequency and the nominal frequency, in Hz. This refers to the state of charge deviation of energy storage, which is the difference between the current SOC and the target SOC, and has no unit. This represents the energy storage health status deviation, which is the difference between the initial SOH (value 1.0) and the current SOH, without units. This refers to the real-time charging and discharging power of energy storage, with discharge being positive and charging being negative. The unit is pu, and the rated power of energy storage is used as the benchmark. All state variables are input into the DRL model after being normalized in step S1.

4. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific definition of the action space in step S2 is as follows: The action space is the continuous charge and discharge power command of the energy storage system. ,in The target charging and discharging power of the energy storage is set; the action command is issued through the communication interface between the power grid and the energy storage to control the energy storage converter to perform power regulation.

5. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific form of the composite reward function in step S2 is as follows: Composite reward function: (3) in, These are weighting coefficients used to balance the priorities of various objectives; For frequency performance, As a health penalty item, For operating costs, Penalties for crossing boundaries; Frequency performance items: (4) in, For system frequency deviation, To maximize the allowable deviation, the reward tends to be maximized when the system frequency is close to the rated value, thereby guiding the agent to prioritize frequency stability as the primary optimization objective. Health penalty items: (5) in, Initial healthy state, The current health status is directly related to the SOH deviation, and the penalty increases non-linearly as the SOH deviation increases. Operating cost item: (6) in, For energy storage charging and discharging power, Cost per unit of energy consumption This refers to the rated capacity of the energy storage. Penalties for crossing boundaries: (7) This item is triggered when the SOC or power exceeds the limit, and outputs a fixed negative reward to enforce safety boundaries.

6. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific form of the cumulative discount reward in step S2 is as follows: Cumulative Discount Rewards: (8) in, This is a discount factor used to balance short-term and long-term returns. To return the reward.

7. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific training process of the DRL model, i.e., DDPG, in step S3 is as follows: Network architecture design: Actor Policy Network: The input layer has 4 neurons, which is consistent with the dimension of the state space. There are 3 hidden layers, with 128 neurons in the first 2 layers and 64 neurons in the 3rd layer. The output layer has 1 neuron corresponding to the energy storage power instruction. The activation function is ReLU. Critic Value Network: The input layer consists of a concatenation of the state vector and the action vector, with a total of 5 neurons. There are 3 hidden layers, with 256 neurons in the first 2 layers and 128 neurons in the 3rd layer. The output layer has 1 neuron, which is the action value. The activation function used is ReLU. Training parameter settings: The Adam optimization algorithm is used to update network parameters. The learning rate of the Actor network is set to 0.001, and the learning rate of the Critic network is set to 0.002; the capacity of the experience replay pool is set to... The batch size for each sampling is 64. Training iteration process: The agent interacts with the power grid simulation environment containing the thermal power unit-energy storage joint frequency regulation model, and stores the state S, action a, reward R, and next state S' into the experience playback pool; Each iteration samples a batch of data from the replay pool, calculates the loss between the target value and the current value using the Critic network, and updates the Critic parameters via backpropagation. The Actor parameters are updated using the policy gradient method to maximize the expected reward. Training continues iteratively until the loss function converges, i.e., the MSE loss is less than [a certain value]. Or the number of training steps reaches step.

8. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific operational constraints for frequency modulation control in step S3 are as follows: (9) (10) (11) (12) In the formula, These are the minimum and maximum frequency deviations allowed by the system, respectively; These are the lower and upper limits of the energy storage state of charge, respectively, to prevent overcharging and over-discharging; These represent the lower and upper limits of energy storage health status, respectively. When the health level is below 0.8, energy storage enters the decommissioning assessment stage. This sets the upper limit of the rated charging and discharging power of the energy storage system to ensure the safe operation of the equipment within the rated range.

9. The deep reinforcement learning frequency modulation control method for energy storage systems based on health status awareness according to claim 1, characterized in that, The specific metrics and criteria for performance verification in step S4 are as follows: Frequency regulation performance indicators: Maximum frequency deviation: The maximum frequency deviation under DRL control is reduced by ≥8% compared to traditional droop control; The ITAE metric, or Time-based integral of absolute error, is as follows: (13) in, The duration of the disturbance. for Frequency deviation at specific times; The ITAE value under DRL control is reduced by ≥25% compared to traditional droop control; The frequency recovery time under DRL control is ≥4% shorter than that under traditional droop control; Energy storage performance indicators: SOH decay rate: The monthly SOH decay rate under DRL control is ≥10% lower than that under traditional droop control; Average charge / discharge power: The average output power of energy storage under DRL control is reduced by ≥5% compared with traditional droop control; If the above indicators are met simultaneously, the model is deemed to have qualified performance and can be put into practical application.

Images