A wind storage unified frequency modulation method based on deep reinforcement learning

Abstract

The application discloses a kind of wind storage unified frequency modulation methods based on deep reinforcement learning, it is related to new energy grid-connected primary frequency modulation, wind storage joint control and power system intelligent control technical field.First, build comprehensive state vector;Second, the additional active reference value of wind turbine side and the active reference value of energy storage side are generated using unified deep reinforcement learning agent;Then, a composite reward function considering the characteristics of frequency drop period and recovery period is constructed;Finally, the two-dimensional continuous joint action is sequentially subjected to dynamic saturation limiting, and the processed active instructions are respectively issued to wind turbine and energy storage system.The application can reduce the control mutation caused by fixed stage switching, realize the adaptive call of wind turbine rotor kinetic energy and energy storage available energy, suppress the deterioration of frequency change rate and frequency minimum, reduce the frequency secondary drop caused by wind turbine active withdrawal, and improve the smoothness, boundary safety and engineering implementability of reinforcement learning frequency modulation instructions.

Description

Technical Field

[0001] This invention relates to the fields of primary frequency regulation for new energy grid connection, joint control of wind and energy storage, and intelligent control technology of power systems, specifically to a unified and coordinated frequency regulation method for wind and energy storage based on deep reinforcement learning. Background Technology

[0002] With the large-scale integration of renewable energy sources such as wind power into the power system, the proportion of traditional synchronous generator units is decreasing, leading to a weakening of the system's equivalent inertia and primary frequency regulation capability. When load disturbances, renewable energy output fluctuations, or changes in operating modes occur, the system frequency change rate increases, the frequency minimum point decreases, and secondary drops or continuous oscillations may occur during the recovery phase, posing a risk to the frequency stability of high-proportion renewable energy power systems. Existing renewable energy power plants typically employ virtual inertia control, droop control, or energy storage for primary frequency regulation, but fixed control parameters are difficult to adapt to continuous changes in wind speed, disturbance intensity, and equipment status. While existing deep reinforcement learning methods can improve the adaptive allocation capability of frequency regulation commands, they still have shortcomings in terms of controlled objects, state space, and action constraints.

[0003] Wind turbines can rapidly increase active power output by releasing rotor kinetic energy, providing short-term inertia and primary frequency regulation support to the power grid. However, their regulation capability is constrained by the current speed, speed safety boundary, rated power, and speed recovery process. When the speed approaches the safety lower limit, the ability to continue releasing kinetic energy decreases significantly. If additional active power is withdrawn or speed recovery is not properly controlled, a new active power deficit will be created, inducing a secondary frequency drop. Existing wind-storage coordination methods mostly determine the input relationship between wind turbines and energy storage based on frequency dead zones, dispatch instructions, or preset rules, still relying on phase switching and manually set logic, making it difficult to achieve continuous coordination under complex disturbances.

[0004] Energy storage systems are characterized by fast response, flexible bidirectional adjustment, and high power control accuracy. They can compensate for active power deficits when wind turbine support is insufficient or power is withdrawn. However, their frequency regulation capability is constrained by rated power, state of charge, safe range, capacity, and continuously available energy. If the call intensity is not dynamically adjusted based on the current state of charge and remaining energy, overcharging, over-discharging, insufficient frequency regulation power, or increased lifespan loss can easily occur. Existing wind-storage frequency regulation methods based on reinforcement learning mostly focus on optimizing virtual inertia coefficients, droop coefficients, or frequency regulation power, and have not yet fully addressed the coordination issues between wind turbine speed recovery, energy storage maintenance, and continuous execution of joint actions.

[0005] Furthermore, existing wind-storage joint frequency regulation typically employs virtual inertia, droop, fuzzy, or phased control, assigning different tasks to wind turbines and energy storage during frequency drop and recovery periods. This can easily lead to problems such as discontinuous power handover, abrupt control changes near thresholds, and insufficient adaptability to complex operating conditions. Some reinforcement learning methods fail to simultaneously incorporate grid frequency dynamics, wind turbine speed margin, available energy storage capacity, capacity configuration, and external disturbances into the state space. The action space also struggles to synchronously represent active power commands from both the wind turbine and energy storage sides. Reward functions often focus on frequency deviation or regulation costs, lacking a unified evaluation of frequency change rate, wind turbine speed exceeding limits, initial energy storage capacity, steady-state reset, state of charge maintenance, and smoothness during recovery periods. Additionally, the actions directly output by the policy network may abruptly change; without filtering and dynamic limiting, this can easily lead to power exceeding limits, state of charge exceeding limits, and command inoperability.

[0006] Therefore, there is an urgent need to construct a unified deep reinforcement learning method for wind and energy storage frequency regulation. Under the same decision-making framework, it can comprehensively perceive the status of the power grid, wind turbines, energy storage and external disturbances, continuously and synchronously generate active power commands on the wind turbine side and energy storage side, and combine phased composite rewards, action smoothing and real-time physical boundary limiting to achieve coordinated, safe and stable control of the entire process of wind and energy storage joint primary frequency regulation. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a unified frequency regulation method for wind and energy storage based on deep reinforcement learning. This method addresses issues such as reduced inertia in high-proportion renewable energy power systems, uncoordinated power distribution between wind turbines and energy storage, frequency dips during wind turbine speed recovery, and limited frequency regulation capabilities of energy storage under low charge conditions. A complete technical solution is constructed, encompassing comprehensive state perception, physical operation boundary calculation, two-dimensional continuous joint action generation, composite reward design in the response phase, action smoothing and dynamic limiting, and closed-loop state feedback. This achieves unified decision-making and continuous coordination between the release of rotor kinetic energy from wind turbines and the mobilization of available energy from energy storage. It overcomes the problems of existing technologies, such as reliance on fixed thresholds or phased manual switching, independent wind and energy storage control commands, insufficient state regression constraints during the recovery phase, and the tendency for reinforcement learning outputs to abruptly change, exceed limits, and lack of engineering feasibility.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: A wind-storage unified frequency regulation method based on deep reinforcement learning includes the following steps: Step S1: Construct a comprehensive state vector: Construct a comprehensive state vector by collecting information to characterize the grid frequency support demand, the wind turbine rotor kinetic energy release capacity, and the energy storage system's continuous regulation capacity. Step S2: Calculate the physically feasible output range of wind turbine and energy storage system under the current operating conditions. Step S3: Generate two-dimensional continuous joint action: Based on the comprehensive state vector described in step S1, a unified deep reinforcement learning agent is used to generate a two-dimensional continuous joint action consisting of the active power reference value added on the wind turbine side and the active power reference value on the energy storage side at the same decision time. Step S4: Construct a phased composite reward function: Combine the drop period reward sub-function and the recovery period reward sub-function according to the frequency response phase characteristics to construct a composite reward function that takes into account frequency deviation, frequency change rate, wind turbine speed, initial available energy of the energy storage system, steady-state reset of the wind turbine and maintenance of the state of charge of the energy storage system, and optimize the joint frequency regulation strategy of the unified deep reinforcement learning agent. Step S5, Action Smoothing and Dynamic Limiting: The two-dimensional continuous joint action is sequentially subjected to first-order low-pass filtering and dynamic saturation limiting based on the physical feasible output range to obtain wind turbine-side additional active power command and energy storage-side active power command that satisfy the action continuity and the operating boundaries of wind turbine and energy storage system. Step S6: Jointly regulate and update the operating status: Send the additional active power command on the wind turbine side and the active power command on the energy storage side to the wind turbine converter control loop and the energy storage grid-connected converter respectively, and perform joint wind and energy storage active power regulation and update the grid frequency status, wind turbine operating status and energy storage system operating status. Step S7, State Feedback and Loop Execution: Feedback the updated grid frequency status, wind turbine operating status and energy storage system operating status to the unified deep reinforcement learning agent, and loop through steps S1 to S6.

[0009] Furthermore, the comprehensive state vector constructed in step S1 is a fourteen-dimensional continuous variable, divided into five layers of state variables: Dynamic layer of power grid: system frequency deviation, frequency change rate; Fan state layer: current fan speed, current fan active power, steady-state speed before frequency regulation, and steady-state power ratio before disturbance; Energy storage state layer: current SOC of energy storage, initial SOC of energy storage, real-time output power of energy storage, rated proportion of energy storage, and energy storage capacity parameters; Environmental parameter layer: steady-state wind speed, equivalent load disturbance; Phase Identification Layer: Auxiliary indicators for frequency drop and recovery period; The frequency response stage identifier is only used for agent observation input and does not set frequency threshold control law hard switching logic.

[0010] Furthermore, in step S2, calculating the physically feasible output force range includes: Energy storage side constraint calculation: Based on the rated power to lock the bidirectional power range of charging and discharging, and combined with the upper and lower limits of SOC, capacity, and continuous charging and discharging capability, calculate the real-time available charging and discharging power; Wind turbine side constraint calculation: Combine the upper and lower safe speeds of the wind turbine and the steady-state speed of frequency regulation to calculate the speed margin. Based on the speed margin, rated power, and active power ramp constraints, determine the upper and lower limits of the wind turbine's additional active power and the allowable range of single-step power change. The real-time output constraints of both energy storage and wind turbines will be used as the boundary basis for subsequent dynamic limiting.

[0011] Furthermore, in step S3, the specific process of generating the two-dimensional continuous joint action is as follows: the comprehensive state vector obtained in step S1 is imported into the policy network of the unified deep reinforcement learning agent to extract the joint state features between the grid frequency support demand, the wind turbine speed margin, and the available energy of the energy storage system; and based on the joint state features, the additional active power reference value on the wind turbine side and the active power reference value on the energy storage side are calculated simultaneously to obtain the two-dimensional continuous joint action corresponding to the execution interface of the wind turbine and the energy storage system; when the system frequency decreases and the wind turbine speed margin meets the rotor kinetic energy release condition, the additional active power reference value on the wind turbine side is increased; when the wind turbine approaches the lower limit of the safe speed and the additional active power reference value on the wind turbine side enters the withdrawal process, the active power reference value on the energy storage side is increased; when the current state of charge of the energy storage system approaches the boundary of the safe state of charge range, the energy storage system call intensity is reduced and the additional active power reference value on the wind turbine side is adjusted.

[0012] Furthermore, in step S4, the specific process of constructing the drop-period reward sub-function is as follows: A frequency deviation penalty term is set based on the system frequency deviation to guide the unified deep reinforcement learning agent to reduce the frequency drop amplitude; a frequency change rate penalty term is set based on the frequency change rate to guide the unified deep reinforcement learning agent to reduce the frequency change rate in the initial stage of the disturbance; a wind turbine speed exceeding limit penalty term is set based on the relationship between the current wind turbine speed and the speed safety boundary to limit the speed change range of the wind turbine after releasing rotor kinetic energy; an initial energy perception penalty term is set based on the initial state of charge, capacity parameters, and initial available energy of the energy storage system to dynamically adjust the penalty intensity corresponding to the discharge power of the energy storage system; finally, a weighted combination is performed to obtain the drop-period reward sub-function.

[0013] Furthermore, in step S4, the specific process of constructing the recovery period reward sub-function and the composite reward function is as follows: A frequency smoothing recovery penalty term is set based on the system frequency deviation during the recovery period and the frequency change at adjacent control times to suppress continuous oscillations during the frequency recovery process; a wind turbine speed regression penalty term is set based on the deviation between the current wind turbine speed and the steady-state speed before frequency regulation begins to guide the wind turbine to recover to the steady-state operating point before the disturbance; a wind turbine active power reverse mutation penalty term is set based on the change in active power at adjacent control times to suppress the secondary power deficit formed by the additional active power withdrawal on the wind turbine side; a state of charge maintenance penalty term is set based on the deviation between the current state of charge of the energy storage system and the median and initial state of charge to guide the energy storage system back to a continuous operating level; an ineffective deep discharge penalty term is set based on the discharge power of the energy storage system during the recovery period and the current state of charge of the energy storage system to limit continuous discharge during the recovery phase; the above penalty terms are weighted and combined to obtain the recovery period reward sub-function, and then combined with the drop period reward sub-function to obtain the composite reward function.

[0014] Furthermore, the training process of the unified deep reinforcement learning agent is as follows: the comprehensive state vector is input into the Actor network and the two-dimensional continuous joint action is output to realize the synchronous decision-making of the active power reference value on the wind turbine side and the active power reference value on the energy storage side; the comprehensive state vector, the two-dimensional continuous joint action, and the reward value corresponding to the composite reward function are input into the Critic network to calculate the long-term benefit evaluation value of the two-dimensional continuous joint action; the current state, current action, current reward, and the state at the next moment are combined into interaction samples and stored in the experience replay pool to form historical interaction data for random sampling training; interaction samples are extracted from the experience replay pool to update the network parameters of the Actor network and the Critic network; a target network corresponding to the Actor network and the Critic network is set, and the network parameters of the target network are updated using a soft update mechanism to improve the convergence stability of the training process; the unified deep reinforcement learning agent is first trained to learn the frequency support and wind-storage power relay law using step load disturbance conditions, and then trained to adapt to time-varying frequency support requirements using continuous random disturbance conditions.

[0015] Furthermore, in step S5, the specific process of the first-order low-pass filtering is as follows: Read the wind turbine-side additional active power reference value at the current control moment and the wind turbine-side smoothed active power command at the previous control moment; perform weighted calculation according to the preset wind turbine-side filtering coefficient to obtain the wind turbine-side smoothed active power command at the current control moment; read the energy storage-side active power reference value at the current control moment and the energy storage-side smoothed active power command at the previous control moment; perform weighted calculation according to the preset energy storage-side filtering coefficient to obtain the energy storage-side smoothed active power command at the current control moment; determine the wind turbine-side filtering coefficient and the energy storage-side filtering coefficient based on the control cycle, the response speed of the wind turbine converter control loop, and the response speed of the energy storage grid-connected converter, limiting the change in the joint active power command at adjacent control moments; combine the wind turbine-side smoothed active power command and the energy storage-side smoothed active power command to obtain the smoothed two-dimensional continuous joint action.

[0016] Furthermore, in step S5, the specific process of dynamically saturating and limiting the smoothed two-dimensional continuous joint action is as follows: the wind turbine side updates the dynamic active power upper and lower limits in real time based on the current speed, rated power, and ramping capability, and limits the smoothed wind turbine command; the energy storage side updates the bidirectional output boundary based on the real-time SOC and rated charging and discharging power, and limits the smoothed energy storage command; after limiting, the command completely falls within the physically feasible output range calculated in step S2.

[0017] Furthermore, in steps S6 and S7, the specific process of performing wind-storage joint active power regulation and state feedback is as follows: The additional active power command from the wind turbine side is superimposed onto the steady-state active power reference of the wind turbine and input into the wind turbine converter control loop to adjust the output active power of the wind turbine; the active power command from the energy storage side is input into the energy storage grid-connected converter to adjust the charging active power or discharging active power of the energy storage system; the grid frequency deviation and frequency change rate are updated based on the joint active power output after execution by the wind turbine and energy storage system to obtain the grid frequency state at the next control moment; the wind turbine speed and active power are updated based on the wind turbine execution result to obtain the wind turbine operating state at the next control moment; the energy storage system state of charge and actual output power are updated based on the energy storage system execution result to obtain the energy storage system operating state at the next control moment; the states of the next control are reconstructed into a comprehensive state vector and fed back to the unified deep reinforcement learning agent to realize continuous joint frequency regulation decision-making between the wind turbine and the energy storage system.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) The present invention provides a wind-storage unified frequency regulation method based on deep reinforcement learning. It adopts a unified reinforcement learning agent to synchronously generate additional active power commands on the wind turbine side and the energy storage side, so that the two types of frequency regulation resources can complete dynamic power output allocation under the same state feedback, avoiding the power output mismatch and stage switching sudden changes caused by traditional independent control or fixed stage control.

[0019] (2) The present invention provides a wind-storage unified frequency regulation method based on deep reinforcement learning, which incorporates wind turbine speed, wind turbine active power, energy storage SOC, energy storage power, capacity configuration, wind speed, disturbance amount and frequency dynamics into the state space, so that the controller can simultaneously sense the frequency support demand, the wind turbine release kinetic energy and the energy storage available energy, thereby improving the global coordination capability under complex working conditions.

[0020] (3) The present invention provides a wind-storage unified frequency regulation method based on deep reinforcement learning, which takes into account different control objectives during the frequency drop period and the recovery period through a staged composite reward function. During the drop period, the focus is on suppressing the maximum frequency deviation and the rate of frequency change, while during the recovery period, the focus is on constraining the wind turbine speed return, SOC maintenance and frequency smooth recovery, so that the system maintains good frequency dynamic quality throughout the process.

[0021] (4) The wind and energy storage unified frequency regulation method provided by the present invention internalizes the lower limit of wind turbine speed, the safe range of energy storage SOC, the upper limit of energy storage power and the active power boundary of wind turbine into the action execution link through first-order low-pass filtering and dynamic saturation limiting. This can reduce the risk of sudden changes and over-limits in the original output of reinforcement learning and improve the feasibility of the project. Attached Figure Description

[0022] Figure 1 The overall flowchart of a wind-storage unified frequency regulation method based on deep reinforcement learning provided by the present invention; Figure 2 The structural diagram for constructing a five-layer integrated state vector provided by this invention; Figure 3 The structural diagram provided by this invention illustrates the computational physical feasible output force range. Figure 4 This is a schematic diagram of the internal architecture of the unified deep reinforcement learning agent of the present invention; Figure 5 This is a schematic diagram of the decomposition of the phased composite reward function of the present invention; Figure 6 This is a block diagram of the hardware functional modules of the wind-storage unified frequency regulation system of the present invention; Figure 7 This is a comparison curve of system frequency under a step load disturbance with a wind power penetration rate of 45% according to an embodiment of the present invention; Figure 8This is a comparison curve of system frequency under different control strategies under continuous random load disturbances according to an embodiment of the present invention; Figure 9 This is a comparison curve of wind turbine speeds under different control strategies under continuous random load disturbances according to an embodiment of the present invention; Figure 10 The above are comparison curves of the energy storage SOC changes under different control strategies under continuous random load disturbances according to embodiments of the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings.

[0024] This invention proposes a unified collaborative frequency regulation method for wind and energy storage based on deep reinforcement learning, such as... Figure 1 As shown, the overall approach is as follows: I. Overall Control Architecture This invention provides a unified wind and energy storage frequency regulation control method based on deep reinforcement learning. This method establishes a closed-loop interactive environment comprising wind turbines, energy storage systems, and grid frequency response models, and sets up a unified intelligent agent as the joint wind and energy storage frequency regulation controller. The environment provides real-time feedback to the intelligent agent on grid frequency deviation, frequency change rate, wind turbine speed, wind turbine active power, energy storage SOC, actual energy storage output power, wind speed, load disturbance, and capacity configuration, among other status information. Based on the above global status, the intelligent agent synchronously generates additional active power reference values ​​for the wind turbine side and active power reference values ​​for the energy storage side, and after smoothing and constraint processing, sends them to the wind turbine and energy storage converter, respectively.

[0025] This unified architecture no longer simply divides wind turbines and energy storage into master-slave resources in fixed phases, nor does it rely on a single frequency threshold-based forced switching control law. Instead, it dynamically adjusts the output ratio of the two types of frequency regulation resources based on the current system state during the same decision-making process. When the frequency drops rapidly and the wind turbine speed margin is sufficient, the agent can increase the additional support strength of the wind turbine; when the wind turbine approaches the speed safety boundary or the support power needs to be withdrawn, the agent can increase energy storage compensation to maintain continuous joint active power output; when the energy storage SOC approaches the safety boundary, the agent can reduce the energy storage call intensity and increase the available support ratio on the wind turbine side.

[0026] II. Operational Boundary Modeling of Wind-Storage Integrated System S1. Establish the physical operating boundary of the wind-storage integrated system. Set a bidirectional active power boundary on the energy storage side to ensure that the output power of the energy storage grid connection interface is always within the allowable charging and discharging range; set a SOC safety range to ensure that the energy storage state of charge is not lower than the minimum allowable value and not higher than the maximum allowable value throughout the frequency regulation process, and gradually returns to a sustainable operating level after the disturbance is relieved.

[0027] S2, establish the operating boundaries of the wind turbine. Set the lower and upper limits of the safe speed, the upper limit of the rated active power, the steady-state power benchmark, and the active power ramp constraint on the wind turbine side, so that the wind turbine can smoothly return to the steady-state operating point before the disturbance after releasing the rotor kinetic energy to participate in frequency support, and avoid secondary power deficit caused by excessive active power withdrawal.

[0028] S3 uses the aforementioned energy storage power boundary, SOC boundary, wind turbine speed boundary, wind turbine active power boundary, and steady-state reset requirement as the state perception information, reward evaluation basis, and action execution constraints of the unified reinforcement learning controller, so that the joint control commands output by the intelligent agent are located within the physically feasible domain that the wind and energy storage system can actually implement.

[0029] III. Integrated State Space and Two-Dimensional Continuous Action Space S4, construct the comprehensive state space, such as Figure 2 As shown. Preferably, the state vector includes 14 continuous variables: system frequency deviation, frequency change rate, current turbine speed, current active power of the turbine, steady-state turbine speed before frequency regulation begins, steady-state power ratio of the turbine before the disturbance, current SOC of energy storage, initial SOC of energy storage, current actual output power of energy storage, rated power ratio of energy storage, energy storage capacity parameters, steady-state wind speed, equivalent load disturbance, and frequency response stage identifier. These variables collectively characterize the grid-side frequency support demand, the turbine's releaseable rotor kinetic energy, the energy storage's sustainable regulation capability, and the intensity of external disturbances.

[0030] The frequency response phase identifier is used to provide the agent with auxiliary information about whether it is currently in a frequency drop or recovery phase, but it is not used as a hard switch. For example... Figure 4 As shown, the agent continues to calculate the power output allocation between the wind turbine and energy storage within the unified policy network, thereby avoiding the sudden changes in control logic near the threshold that occur in traditional staged control.

[0031] S5 constructs a two-dimensional continuous action space. The actions output by the unified intelligent agent include the additional active power reference value on the wind turbine side and the active power reference value on the energy storage side. This two-dimensional action directly corresponds to the executable active power regulation command of the wind-storage integrated system, enabling the wind turbine and energy storage to complete coordinated allocation at the same decision moment, rather than having two independent controllers issue uncoordinated control quantities.

[0032] IV. Composite reward function taking into account the characteristics of the response phase ( Figure 5 ) S6, Set up a composite reward function. The total reward is composed of a drop-period reward sub-function and a recovery-period reward sub-function, combined according to the characteristics of each stage. The drop-period reward focuses on constraining frequency deviation, frequency change rate, wind turbine speed exceeding limits, and the intensity of initial available energy call for energy storage, in order to guide the wind-storage system to quickly provide active power support, reduce RoCoF, and raise the minimum frequency point in the early stages of disturbance.

[0033] An initial energy perception penalty is introduced into the drop-period reward. This penalty dynamically adjusts the energy storage output penalty intensity based on the initial SOC of the energy storage, the energy storage capacity, and the available energy status: when the energy storage is at a low initial SOC or the available energy is insufficient, the energy storage discharge penalty is increased, guiding the strategy to prioritize the release of wind turbine rotor kinetic energy and cautiously utilize the energy storage; when the energy storage margin is sufficient, the energy storage is allowed to undertake more rapid compensation tasks.

[0034] The recovery period incentive focuses on ensuring smooth frequency recovery, wind turbine speed return, suppression of reverse abrupt changes in wind turbine active power, maintenance of median or initial SOC levels in energy storage, and suppression of ineffective deep discharge in energy storage. Through this incentive mechanism, the control objective can naturally transition from suppressing extreme frequency values ​​in the initial stage of disturbance to the return of wind and energy storage operating status and energy maintenance in the recovery phase, without requiring external fixed switching logic.

[0035] V. Combined motion generation, smoothing, and dynamic limiting S7 employs a deep deterministic policy gradient algorithm to train the unified agent. The Actor network receives the integrated state vector and outputs two-dimensional continuous actions; the Critic network evaluates the long-term benefits of the current joint control behavior based on state, action, and reward feedback; during training, experience replay, a target network, and a soft update mechanism are used to improve the policy convergence stability. The training conditions can be gradually expanded from step perturbations to continuous random perturbations, allowing the agent to first learn the basic frequency support and power relay laws, and then adapt to complex time-varying scenarios.

[0036] S8 performs a first-order low-pass filter on the raw wind turbine and energy storage actions output from the Actor network to obtain a smoothed combined active power command. The filter coefficient can be set according to the control cycle and the response speed of the actuator to suppress rapid changes in actions between adjacent control moments and reduce the adverse effects of sudden changes in the combined wind and energy storage output on the system frequency recovery process.

[0037] S9 applies dynamic saturation limiting to the smoothed combined commands. The wind turbine-side limiting is determined in real-time based on the current speed margin, steady-state operating point, rated power, and ramp capability; the energy storage-side limiting is determined in real-time based on rated power, SOC safety range, current SOC, capacity configuration, and continuous charging / discharging capability. The limited wind turbine active power commands and energy storage active power commands are then applied to the wind turbine converter control circuit and the energy storage grid-connected converter, respectively.

[0038] In S10, the power grid environment updates state variables such as frequency deviation, frequency change rate, wind turbine speed, wind turbine active power, energy storage SOC, and energy storage power based on the execution results of the wind turbine and energy storage, and feeds back the state at the next moment to the intelligent agent. The above S4 to S10 are executed cyclically to form a unified collaborative closed-loop control of wind and energy storage for primary frequency regulation throughout the entire process.

[0039] This invention is not limited to the DDPG algorithm. While maintaining the integrated state space, two-dimensional continuous joint actions, staged composite rewards, action smoothing, and dynamic limiting mechanism, TD3, SAC, multi-agent reinforcement learning, or other reinforcement learning algorithms suitable for continuous action spaces can also be used. The state variables can be added or removed according to the engineering object, such as adding wind turbine pitch angle, energy storage temperature, communication delay, or measurement noise status. The reward weight, filtering coefficient, and limiting threshold can also be tuned according to different wind farms and energy storage configurations.

[0040] To verify the frequency regulation advantages of the wind-storage unified frequency regulation method based on deep reinforcement learning in this invention, a power grid simulation platform with a high wind power penetration rate of 45% and low inertia was built. Two typical power grid operating conditions were set up: step load disturbance and random continuous load disturbance. Four existing strategies were selected as controls: traditional wind turbine inertial control, conventional wind-storage inertial coordination, wind turbine fuzzy frequency regulation, and single energy storage reinforcement learning. The performance of this scheme was verified by comparing and verifying it from multiple dimensions, including system frequency dynamics, wind turbine and energy storage operating status, and quantitative frequency regulation indicators. Representative curves were selected and attached to illustrate the core effects.

[0041] 1. Simulation of step load disturbance (corresponding to) Figure 7 ) A step load drop simulates a high-power deficit fault in the power grid. The system frequency variation curves for each scheme are shown in the appendix. Figure 7 The quantitative indicators are shown in Table 1. Traditional control schemes exhibit significant frequency drops at the lowest point and obvious secondary frequency drops during the disturbance recovery phase. The unified wind-storage reinforcement learning scheme of this invention relies on hierarchical global state perception, phased composite rewards, and action smoothing and limiting mechanisms. The maximum frequency deviation is only 0.0811Hz, and the ITAE integral error is reduced to 260.52, which is far lower than other comparative strategies. The wind turbine speed drop is gradual, and the energy storage is charged and discharged on demand, effectively eliminating secondary frequency oscillations and significantly reducing the frequency regulation burden of synchronous units.

[0042] 2. Simulation of random continuous load disturbance (corresponding to) Figure 8 , Figure 9 , Figure 10 ) A continuous time-varying stochastic load simulation was used to model the daily fluctuations of the power grid. The system frequency curve, wind turbine speed curve, and energy storage SOC curve were selected as representative figures (see attached figures). Figure 8 , 9 (10), quantitative indicators are shown in Table 2. Under long-term random disturbances, the traditional control strategy experiences continuous and significant frequency deviations, resulting in drastic fluctuations in wind turbine speed and frequent deep charging and discharging of energy storage. The method of this invention can adaptively allocate the kinetic energy of the wind turbine rotor and the active energy of the energy storage in real time. The maximum frequency deviation throughout the entire period is only 0.1500Hz. The peak-to-peak frequency and integral error are significantly better than the comparative schemes. The fluctuation of wind turbine speed is significantly narrowed, and the energy storage SOC is maintained in a healthy range, taking into account both frequency regulation stability and energy storage lifespan.

[0043] Based on the simulation data of the two types of working conditions, it can be seen that compared with the existing discrete and single-objective reinforcement learning frequency modulation schemes, the whole cooperative control architecture of the present invention has better frequency suppression capability and equipment friendliness under fault disturbance and long-term random fluctuation scenarios. The frequency modulation gain brought about by the technical improvement is quantifiable and reproducible, and has practical engineering value.

[0044] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0045] Table 1. Performance comparison of different control strategies under step disturbance at a wind power penetration rate of 45%.

[0046] Table 2 Performance Comparison of Different Control Strategies under Random Disturbances

Claims

1. A wind-storage unified frequency regulation method based on deep reinforcement learning, characterized in that, Includes the following steps: Step S1: Construct a comprehensive state vector: Construct a comprehensive state vector by collecting information to characterize the grid frequency support demand, the wind turbine rotor kinetic energy release capacity, and the energy storage system's continuous regulation capacity. Step S2: Calculate the physically feasible output range of wind turbine and energy storage system under the current operating conditions. Step S3: Generate two-dimensional continuous joint action: Based on the comprehensive state vector described in step S1, a unified deep reinforcement learning agent is used to generate a two-dimensional continuous joint action consisting of the active power reference value added on the wind turbine side and the active power reference value on the energy storage side at the same decision time. Step S4: Construct a phased composite reward function: Combine the drop period reward sub-function and the recovery period reward sub-function according to the frequency response phase characteristics to construct a composite reward function that takes into account frequency deviation, frequency change rate, wind turbine speed, initial available energy of the energy storage system, steady-state reset of the wind turbine and maintenance of the state of charge of the energy storage system, and optimize the joint frequency regulation strategy of the unified deep reinforcement learning agent. Step S5, Action Smoothing and Dynamic Limiting: The two-dimensional continuous joint action is sequentially subjected to first-order low-pass filtering and dynamic saturation limiting based on the physical feasible output range to obtain wind turbine-side additional active power command and energy storage-side active power command that satisfy the action continuity and the operating boundaries of wind turbine and energy storage system. Step S6: Jointly regulate and update the operating status: Send the additional active power command on the wind turbine side and the active power command on the energy storage side to the wind turbine converter control loop and the energy storage grid-connected converter respectively, and perform joint wind and energy storage active power regulation and update the grid frequency status, wind turbine operating status and energy storage system operating status. Step S7, State Feedback and Loop Execution: Feedback the updated grid frequency status, wind turbine operating status and energy storage system operating status to the unified deep reinforcement learning agent, and loop through steps S1 to S6.

2. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, The comprehensive state vector constructed in step S1 is a fourteen-dimensional continuous variable, divided into five layers of state variables: Dynamic layer of power grid: system frequency deviation, frequency change rate; Fan state layer: current fan speed, current fan active power, steady-state speed before frequency regulation, and steady-state power ratio before disturbance; Energy storage state layer: current SOC of energy storage, initial SOC of energy storage, real-time output power of energy storage, rated proportion of energy storage, and energy storage capacity parameters; Environmental parameter layer: steady-state wind speed, equivalent load disturbance; Phase Identification Layer: Auxiliary indicators for frequency drop and recovery period; The frequency response stage identifier is only used for agent observation input and does not set frequency threshold control law hard switching logic.

3. The wind-storage unified frequency regulation method based on deep reinforcement learning as described in claim 1, characterized in that, In step S2, calculating the physically feasible output range includes: Energy storage side constraint calculation: Based on the rated power to lock the bidirectional power range of charging and discharging, and combined with the upper and lower limits of SOC, capacity, and continuous charging and discharging capability, calculate the real-time available charging and discharging power; Wind turbine side constraint calculation: Combine the upper and lower safe speeds of the wind turbine and the steady-state speed of frequency regulation to calculate the speed margin. Based on the speed margin, rated power, and active power ramp constraints, determine the upper and lower limits of the wind turbine's additional active power and the allowable range of single-step power change. The real-time output constraints of both energy storage and wind turbines will be used as the boundary basis for subsequent dynamic limiting.

4. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, In step S3, the specific process of generating the two-dimensional continuous joint action is as follows: the comprehensive state vector obtained in step S1 is imported into the policy network of the unified deep reinforcement learning agent to extract the joint state features between the grid frequency support demand, the wind turbine speed margin, and the available energy of the energy storage system; and based on the joint state features, the additional active power reference value on the wind turbine side and the active power reference value on the energy storage side are calculated simultaneously to obtain the two-dimensional continuous joint action corresponding to the execution interface of the wind turbine and the energy storage system; when the system frequency decreases and the wind turbine speed margin meets the rotor kinetic energy release condition, the additional active power reference value on the wind turbine side is increased; when the wind turbine approaches the lower limit of the safe speed and the additional active power reference value on the wind turbine side enters the withdrawal process, the active power reference value on the energy storage side is increased; when the current state of charge of the energy storage system approaches the boundary of the safe state of charge range, the energy storage system call intensity is reduced and the additional active power reference value on the wind turbine side is adjusted.

5. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, In step S4, the specific process of constructing the drop-period reward sub-function is as follows: A frequency deviation penalty term is set based on the system frequency deviation to guide the unified deep reinforcement learning agent to reduce the frequency drop amplitude; a frequency change rate penalty term is set based on the frequency change rate to guide the unified deep reinforcement learning agent to reduce the frequency change rate in the initial stage of the disturbance; a wind turbine speed exceeding limit penalty term is set based on the relationship between the current wind turbine speed and the speed safety boundary to limit the speed change range of the wind turbine after releasing rotor kinetic energy; an initial energy perception penalty term is set based on the initial state of charge, capacity parameters, and initial available energy of the energy storage system to dynamically adjust the penalty intensity corresponding to the discharge power of the energy storage system; finally, a weighted combination is performed to obtain the drop-period reward sub-function.

6. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, In step S4, the specific process of constructing the recovery period reward sub-function and the composite reward function is as follows: A frequency smoothing recovery penalty term is set based on the system frequency deviation during the recovery period and the frequency change at adjacent control times to suppress continuous oscillations during the frequency recovery process; a wind turbine speed regression penalty term is set based on the deviation between the current wind turbine speed and the steady-state speed before frequency regulation begins to guide the wind turbine to recover to the steady-state operating point before the disturbance; a wind turbine active power reverse mutation penalty term is set based on the change in active power at adjacent control times to suppress the secondary power deficit caused by the additional active power withdrawal on the wind turbine side; a state of charge maintenance penalty term is set based on the deviation between the current state of charge of the energy storage system and the median and initial state of charge to guide the energy storage system back to a continuous operating level; an ineffective deep discharge penalty term is set based on the energy storage system discharge power and the current state of charge during the recovery period to limit continuous discharge during the recovery phase; the above penalty terms are weighted and combined to obtain the recovery period reward sub-function, and then combined with the drop period reward sub-function to obtain the composite reward function.

7. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, The training process of the unified deep reinforcement learning agent is as follows: The comprehensive state vector is input into the Actor network, which outputs the two-dimensional continuous joint action to achieve synchronous decision-making on the wind turbine-side active power reference value and the energy storage-side active power reference value; the comprehensive state vector, the two-dimensional continuous joint action, and the reward value corresponding to the composite reward function are input into the Critic network to calculate the long-term benefit evaluation value of the two-dimensional continuous joint action; the current state, current action, current reward, and next-time state are combined into interaction samples and stored in the experience replay pool to form historical interaction data for random sampling training; interaction samples are extracted from the experience replay pool to update the network parameters of the Actor network and the Critic network; a target network corresponding to the Actor network and the Critic network is set, and the network parameters of the target network are updated using a soft update mechanism to improve the convergence stability of the training process; the unified deep reinforcement learning agent is first trained to learn frequency support and wind-storage power relay laws using step load disturbance conditions, and then trained to adapt to time-varying frequency support requirements using continuous random disturbance conditions.

8. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, In step S5, the specific process of the first-order low-pass filtering is as follows: Read the wind turbine-side additional active power reference value at the current control moment and the wind turbine-side smoothed active power command at the previous control moment; perform weighted calculation according to the preset wind turbine-side filtering coefficient to obtain the wind turbine-side smoothed active power command at the current control moment; read the energy storage-side active power reference value at the current control moment and the energy storage-side smoothed active power command at the previous control moment; perform weighted calculation according to the preset energy storage-side filtering coefficient to obtain the energy storage-side smoothed active power command at the current control moment; determine the wind turbine-side filtering coefficient and the energy storage-side filtering coefficient based on the control cycle, the response speed of the wind turbine converter control loop, and the response speed of the energy storage grid-connected converter to limit the change in the joint active power command at adjacent control moments; combine the wind turbine-side smoothed active power command and the energy storage-side smoothed active power command to obtain the smoothed two-dimensional continuous joint action.

9. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, In step S5, the specific process of dynamically saturating and limiting the smoothed two-dimensional continuous joint action is as follows: the wind turbine side updates the dynamic active power upper and lower limits in real time based on the current speed, rated power, and climbing ability, and limits the smoothed wind turbine command; the energy storage side updates the bidirectional output boundary based on the real-time SOC and rated charging and discharging power, and limits the smoothed energy storage command; after limiting, the command completely falls within the physically feasible output range calculated in step S2.

10. The wind-storage unified frequency regulation method based on deep reinforcement learning according to claim 1, characterized in that, In steps S6 and S7, the specific process of performing wind-storage joint active power regulation and state feedback is as follows: The additional active power command from the wind turbine side is superimposed onto the steady-state active power reference of the wind turbine and input into the wind turbine converter control loop to adjust the output active power of the wind turbine; the active power command from the energy storage side is input into the energy storage grid-connected converter to adjust the charging or discharging active power of the energy storage system; the grid frequency deviation and frequency change rate are updated based on the joint active power output after execution by the wind turbine and energy storage system to obtain the grid frequency state at the next control moment; the wind turbine speed and active power are updated based on the wind turbine execution result to obtain the wind turbine operating state at the next control moment; the energy storage system state of charge and actual output power are updated based on the energy storage system execution result to obtain the energy storage system operating state at the next control moment; the states of the next control are reconstructed into a comprehensive state vector and fed back to the unified deep reinforcement learning agent to achieve continuous joint frequency regulation decision-making between the wind turbine and the energy storage system.

Images