Wind turbine grid inertia support method based on deep reinforcement learning

Abstract

The application discloses a wind turbine grid inertia support method based on deep reinforcement learning, comprising the following steps: (1) obtaining wind turbine speed and grid frequency data; (2) designing an improved wind turbine primary frequency modulation control strategy based on a DDPG algorithm; (3) training a DDPG intelligent agent under training wind speed; (4) using the intelligent agent trained in step (3), outputting inertia support power according to wind turbine speed and grid frequency under test wind speed, and realizing inertia support of the wind turbine to the grid. The technical scheme of the application obtains wind turbine inertia support power through a deep deterministic policy gradient reinforcement learning algorithm on the basis of RSC control of the wind turbine. Compared with the traditional wind turbine fixed coefficient primary frequency modulation control, the method proposed by the application comprehensively considers wind turbine stability and inertia support effect, so that the wind turbine fully utilizes rotor kinetic energy to provide inertia support for the grid under the premise of maintaining stable operation.

Description

Technical Field

[0001] This invention pertains to wind turbine primary frequency regulation control strategies, specifically involving a method for supporting the grid inertia of wind turbine units based on deep reinforcement learning. Background Technology

[0002] Large-scale wind power grid connection poses challenges to grid frequency stability. On the one hand, wind turbines are connected to the grid via converters and do not have inertial response characteristics, causing the grid's equivalent inertia to decrease as the proportion of wind power increases. On the other hand, as synchronous turbines are continuously replaced by wind power, continuing to rely on synchronous turbines for frequency regulation will inevitably lead to insufficient grid frequency regulation capacity.

[0003] Considering the large moment of inertia and speed variation range of wind turbines, they can rely on the storage and release of rotor kinetic energy to provide active power to support the grid frequency. Based on this characteristic, wind turbine virtual inertia control and droop control simulate the inertia response characteristics and governor response characteristics of synchronous units by adding frequency derivative and proportional components to the original controller, thereby responding to system frequency changes and providing inertia support for the grid.

[0004] In traditional wind turbine primary frequency control, the droop coefficient and virtual inertia coefficient are often fixed values, typically set empirically. However, facing complex and variable operating conditions, a fixed frequency control coefficient cannot guarantee consistently good frequency regulation performance. On one hand, an excessively small frequency control coefficient makes it difficult for the wind turbine to provide effective inertia support to the grid; on the other hand, an excessively large frequency control coefficient can easily cause the wind turbine to excessively release rotor kinetic energy, increasing the risk of instability and shutdown. Compared to a fixed frequency control coefficient, a variable coefficient approach allows the wind turbine to better balance inertia support effectiveness and its own stability. Therefore, the frequency change rate, frequency deviation, and turbine speed are all used to tune the frequency control coefficient, enabling the wind turbine to adjust the inertia support strength in real time according to the grid frequency, avoiding excessive support that could affect the stable operation of the wind turbine.

[0005] However, the parameter tuning of current variable coefficient primary frequency control methods is often determined empirically. Faced with complex scenarios involving turbulent wind speeds and fluctuating loads, controller parameters tuned empirically are insufficient to ensure both stability and frequency regulation performance of the wind turbine under such conditions. The reason for this is that the aerodynamic power and rotor kinetic energy of the wind turbine vary at different wind speeds, affecting its inertia support capacity. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a method for grid inertia support of wind turbines based on deep reinforcement learning, which can solve the problem of difficulty in balancing wind turbine stability and inertia support effect under complex scenarios of turbulent wind speed and fluctuating load.

[0007] Technical solution: The present invention provides a method for supporting the grid inertia of wind turbine generators based on deep reinforcement learning, comprising the following steps:

[0008] (1) Obtain wind turbine speed and power grid frequency data;

[0009] (2) An improved primary frequency regulation control strategy for wind turbines is designed based on the DDPG algorithm;

[0010] (3) Train the DDPG agent under training wind speed;

[0011] (4) Using the intelligent agent trained in step (3), the inertial support power is output online according to the wind turbine speed and the grid frequency under the test wind speed, so as to realize the inertial support of the wind turbine to the grid.

[0012] In step (2), the improved primary frequency control strategy for wind turbines based on the DDPG algorithm specifically includes the following steps:

[0013] (2.1) Select the system state variables as wind turbine speed and grid frequency deviation, i.e., S=[ω,Δf];

[0014] (2.2) Take the action output of the intelligent agent as the power supported by the inertia of the wind turbine, that is, A=[ΔP vic ];

[0015] (2.3) Design the agent reward function so that the power output of the wind turbine inertia support takes into account both the wind turbine speed and the inertia support effect.

[0016] (2.4) The sum of the active power and the inertial support power under the RSC control of the wind turbine is the electromagnetic power of the wind turbine.

[0017] In step (2.3), the reward function for the intelligent agent is designed, and the expression of the reward function is as follows:

[0018] R=-k1|Δf|-k2|h(ω)| (1)

[0019] In the formula, h(ω) is the reward function for the wind turbine speed term. The agent reward function is divided into two parts, k1 and k2, which are the reward weight factors for the two parts, respectively. The first part of the reward is related to the grid frequency, and the agent's behavior tends to be related to the wind turbine inertia support effect. The second part of the reward is related to the wind turbine speed, and the agent's behavior tends to be related to the unit stability.

[0020] The expression for h(ω) is as follows:

[0021]

[0022]

[0023] In the formula, ω minThis indicates the fan tripping speed; when the fan speed deviates from the set upper and lower penalty boundaries ω up ω down The agent's action output is penalized, and the greater the deviation, the greater the penalty; when the fan speed drops to the cut-off speed, the penalty value becomes infinite, and the training for the current round ends.

[0024] In step (3), training the DDPG agent under the training wind speed specifically includes the following steps:

[0025] (3.1) Initialize the neural network parameters. The neural network consists of two parts: a policy network (Actor) and a value network (Critic). The Actor part is responsible for outputting actions based on the state, while the Critic part outputs Q(S). t A t Each part is further divided into the current network and the target network;

[0026] (3.2) In the current network of the Actor, with state S t Input, output action A t Action A t Acting on the environment, the state is updated to S. t+1 Return reward R t+1 ; Transfer the dataset <S t A t ,S t+1 ,R t+1 > Store in the experience pool;

[0027] (3.3) Randomly select n datasets <S from the experience pool. i A i ,S i+1 ,R i+1 >, i = 1, 2, ..., n, and use this to calculate the current target Q value y. i The calculation formula is as follows:

[0028] y i =R t+1 +γQ'(S t+1 A t+1 (4)

[0029] In the formula, γ represents the Q-value attenuation factor, which ranges from 0.95 to 0.99;

[0030] (3.4) Based on the current Q value Q(S) t A t ) and target Q value y i Calculate the mean squared error, backpropagate the error to update the current network parameter α1 of the Critic; use the Adam optimizer to update the current network parameter θ1 of the Actor with the current Q-value data.

[0031] (3.5) The network parameters of the Actor and Critic are copied from the current network at regular intervals to achieve soft updates of network parameters.

[0032]

[0033] In the formula, the parameter β is generally set to 0.001.

[0034] In step (4), the parameters of the Actor neural network have converged. The learning performance of the agent is tested under the test wind speed. The inertial support power is output online directly according to the wind turbine speed and the grid frequency deviation, so as to realize the inertial support of the wind turbine to the grid.

[0035] The technical solution of this invention utilizes deep reinforcement learning to integrate deep learning perception capabilities and reinforcement learning decision-making capabilities, enabling end-to-end optimization of nonlinear problems. Through training, the deep reinforcement learning agent autonomously learns to balance active power output with wind turbine stability and frequency regulation effects, thereby fully leveraging the wind turbine's inertia support capabilities and enabling the wind turbine to better suppress grid frequency changes while ensuring its own stability.

[0036] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial effects: (1) Using deep reinforcement learning algorithm, the agent is trained in the complex scenario of turbulent wind speed and fluctuating load. The agent with converged action output strategy is then directly output inertial support power according to the wind turbine speed and grid frequency deviation under the test wind speed, so as to realize end-to-end optimization of nonlinear problems; (2) In the present invention, the agent reward function comprehensively considers the wind turbine speed and grid frequency. During the inertial support process, the wind turbine speed is limited to the range where the speed term reward is 0, which effectively avoids the wind turbine from losing stability and enables the wind turbine to better suppress grid frequency changes while ensuring its own stability; (3) Based on the RSC control of the wind turbine, the present solution obtains the inertial support power of the wind turbine through the deep deterministic strategy gradient reinforcement learning algorithm. Compared with the traditional wind turbine fixed coefficient primary frequency regulation control, the method proposed in the present invention comprehensively considers the wind turbine stability and inertial support effect, so that the wind turbine can make full use of rotor kinetic energy to provide inertial support for the grid while maintaining its own stable operation. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating the wind turbine grid inertia support method based on deep reinforcement learning described in this invention.

[0038] Figure 2 This is a schematic diagram of the power grid frequency response model considering wind power frequency regulation in this invention;

[0039] Figure 3 This is a graph showing the wind speed during agent training in this invention.

[0040] Figure 4 This is a wind speed curve diagram for testing the intelligent agent in this invention;

[0041] Figure 5 This is a training reward curve diagram of the intelligent agent in this invention;

[0042] Figure 6 This is a graph showing the comparison of the fan speeds of the intelligent agent in this invention under training wind speeds, with and without inertial support provided by the fan.

[0043] Figure 7 This is a graph showing the grid frequency deviation of the intelligent agent in this invention under training wind speeds, with and without inertia support provided by the wind turbine.

[0044] Figure 8 This is a graph showing the comparison of the fan speeds of the intelligent agent in this invention under test wind speeds, with and without inertial support provided by the fan.

[0045] Figure 9 This is a graph showing the difference in grid frequency between the wind turbine providing inertial support and not providing inertial support under the tested wind speed, as described in this invention. Detailed Implementation

[0046] The technical solution of the present invention will now be described in detail with reference to specific embodiments and the accompanying drawings.

[0047] like Figure 1 As shown, the wind turbine grid inertia support method based on deep reinforcement learning of the present invention includes the following steps:

[0048] (1) Obtain wind turbine speed and power grid frequency data;

[0049] (2) An improved primary frequency regulation control strategy for wind turbines based on the DDPG algorithm is designed as follows:

[0050] (2.1) Select the system state variables as wind turbine speed and grid frequency deviation, i.e., S=[ω,Δf];

[0051] (2.2) The action output of the intelligent agent is selected as the power of the wind turbine inertia support, i.e., A = [ΔP] vic ];

[0052] (2.3) Design the agent reward function to ensure that the power output supported by the wind turbine's inertia considers both the wind turbine's rotational speed and the effect of inertia support. The reward function is set as follows:

[0053] R=-k1|Δf|-k2|h(ω)| (1)

[0054] In the formula, h(ω) is the reward function for the wind turbine speed term; the reward function is divided into two parts, k1 and k2 being the reward weight factors for the two parts, respectively. The first part of the reward is related to the grid frequency, and the agent's behavior tends to be related to the wind turbine inertia support effect. The second part of the reward is related to the wind turbine speed, and the agent's behavior tends to be related to unit stability.

[0055] The expression for h(ω) is shown in equations (2) and (3):

[0056]

[0057]

[0058] In the formula, ω min This refers to the fan tripping speed. When the fan speed deviates from the set upper and lower penalty boundaries ω... up ω down The agent's action output is penalized, and the greater the deviation, the greater the penalty. In particular, when the rotation speed drops to the cut-off speed, the penalty value becomes infinite, and the current training round ends.

[0059] (2.4) The sum of the active power and the inertia support power under RSC control of the wind turbine is the electromagnetic power of the wind turbine. The electromagnetic power of the wind turbine is as follows:

[0060] P e =P0+ΔP vic (4)

[0061] (3) The DDPG agent is trained under training wind speed, specifically as follows:

[0062] (3.1) Initialize the neural network parameters: θ1, θ2, α1, α2. Where θ1 = θ2, α1 = α2; the neural network consists of two parts: a policy network (Actor) and a value network (Critic). The Actor part is responsible for outputting actions based on the state, while the Critic part outputs Q(S). t A t Each part is further divided into the current network and the target network;

[0063] (3.2) In the current network of the Actor, with state S t Input, output action A t Action A t Acting on the environment, the state is updated to S. t+1 Return reward R t+1 . Set the dataset <S t A t ,S t+1 ,R t+1 > Store in the experience pool;

[0064] (3.3) Randomly select n datasets from the experience pool, <S i A i ,S i+1 ,R i+1 >, i = 1, 2, ..., n. Calculate the current target Q value y based on this. i The calculation formula is as follows:

[0065] y i =R t+1 +γQ'(S t+1 A t+1 (5)

[0066] In the formula, γ represents the Q-value attenuation factor, which ranges from 0.95 to 0.99.

[0067] (3.4) Based on the current Q value Q(S) t A t ) and target Q value y i Calculate the mean squared error, and backpropagate the error to update the current network parameters α1 of the Critic. Update the current network parameters θ1 of the Actor using the Adam optimizer with the current Q-value data.

[0068] (3.5) The network parameters of the Actor and Critic targets are periodically copied from the current network to achieve soft updates of the network parameters. As shown in equation (6):

[0069]

[0070] In the formula, the parameter β is generally set to 0.001.

[0071] (4) Using the intelligent agent trained in step (3), the inertial support power is output online according to the wind turbine speed and the grid frequency under the test wind speed, so as to realize the inertial support of the wind turbine to the grid.

[0072] The deep reinforcement learning-based wind turbine grid inertia support method of the present invention can be applied in computer equipment. Specifically, the computer equipment includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0073] (1) Obtain wind turbine speed and power grid frequency data;

[0074] (2) An improved primary frequency regulation control strategy for wind turbines is designed based on the DDPG algorithm;

[0075] (3) Train the DDPG agent under training wind speed;

[0076] (4) Apply the trained agent in step (3) to output inertial support power online according to the wind turbine speed and grid frequency under the test wind speed, so as to realize the inertial support of the wind turbine to the grid.

[0077] The deep reinforcement learning-based wind turbine grid inertia support method of the present invention can also be applied to a computer storage medium, specifically: a computer program is stored on the computer storage medium, and when the computer program is executed by a processor, it performs the following steps:

[0078] (1) Obtain wind turbine speed and power grid frequency data;

[0079] (2) An improved primary frequency regulation control strategy for wind turbines is designed based on the DDPG algorithm;

[0080] (3) Train the DDPG agent under training wind speed;

[0081] (4) Apply the trained agent in step (3) to output inertial support power online according to the wind turbine speed and grid frequency under the test wind speed, so as to realize the inertial support of the wind turbine to the grid.

[0082] The present invention will now be described in further detail with reference to the embodiments.

[0083] Example 1:

[0084] The proposed method was validated using a wind power grid-connected model built with MATLAB / SIMULINK. The main parameters of the wind turbine are shown in Table 1, and the grid connection parameters are as follows: Figure 2 The Low Order Frequency Response (LFR) model shown is illustrated in Table 2. The DDPG neural network parameters are randomly initialized, and other algorithm parameters are shown in Table 3.

[0085] Table 1 Main parameters of the fan

[0086]

[0087]

[0088] Table 2 Main parameters of LFR model

[0089]

[0090] Table 3 Main parameters of DDPG algorithm

[0091]

[0092] First, determine the relevant parameters in the reward function, with the reward weight factor k1 = 1. Fan switching speed ωmin The upper and lower penalty boundaries ω are 400 rpm. up ω down The speeds are 1300 rpm and 700 rpm, respectively. In complex scenarios involving turbulent wind speeds and fluctuating loads, the agent trains online for 80-second rounds to learn the inertial support power, enabling the turbine's inertial support power output to adapt to changes in wind speed and load. The training and test wind speeds are generated by Bladed, as shown below. Figure 3 , Figure 4 As shown.

[0093] The agent is trained for a set number of rounds, and the reward curve after training is as follows: Figure 5 As shown in the figure. The comparison of wind turbine speed and grid frequency deviations after the agent training is completed and under the condition that the wind turbine does not provide inertial support is shown in the figure. Figure 6 , Figure 7 As shown in the table, the agent's reward curve has essentially converged by round 130. After the agent training is complete, the maximum and minimum grid frequency deviations and the maximum frequency deviations when the wind turbine does not provide inertia support are shown in Table 4. Under the control strategy proposed in this paper, the minimum frequency deviation is improved by 0.1343 Hz, the maximum frequency deviation is improved by 0.1381 Hz, and the inertia support effect is good. Furthermore, the wind turbine speed is limited to the range where the speed term reward is 0, ensuring unit stability.

[0094] Table 4. Maximum and minimum frequency deviations under training wind speeds.

[0095]

[0096] To test the generalization performance of the agent outside the training set, specifically to verify whether the agent's inertial support power output under different wind speed data equally considers the inertial support effect and turbine stability, a pre-trained agent was applied and tested at the test wind speed. Based on the turbine speed and grid frequency deviation, the agent outputs inertial support power online, realizing the inertial support of the wind turbine to the grid. The comparison of grid frequency deviation and turbine speed with and without inertial support is shown below. Figure 8 , Figure 9 As shown in Table 5, the maximum and minimum frequency deviations of the grid under the proposed control strategy and without inertia support are compared at the test wind speed. Calculations show that the minimum frequency deviation improvement under the proposed control strategy is 0.2337 Hz, and the maximum improvement is 0.191 Hz. The intelligent agent also exhibits good inertia support at the test wind speed. The speed comparison curves show that the wind turbine speed is reliably limited within the preset range under the proposed control strategy.

[0097] Table 5. Maximum and minimum frequency deviations under test wind speeds.

[0098]

Claims

1. A method for supporting the grid inertia of wind turbine generators based on deep reinforcement learning, characterized in that, Includes the following steps: Acquire wind turbine speed and grid frequency data; based on An improved frequency regulation control strategy for wind turbines based on algorithm design; Training wind speed The agent is trained; Utilization steps The trained agent outputs inertial support power online based on the wind turbine speed and the deviation of the power grid frequency under test wind speed, thereby realizing the inertial support of the wind turbine to the power grid. step In the middle, the one based on The improved primary frequency regulation control strategy for wind turbines, based on algorithm design, includes the following steps: The system state variables are selected as wind turbine speed and grid frequency deviation, i.e. ; The action output of the intelligent agent is taken as the power supported by the inertia of the wind turbine, that is... ; Design an agent reward function to ensure that the power output of the wind turbine is supported by its inertia while taking into account both the wind turbine speed and the inertia support effect. The sum of the active power under RSC control of the wind turbine and the inertia support power is the electromagnetic power of the wind turbine. step In the design of the agent's reward function, the expression of the reward function is as follows: ； In the formula, The reward function is for the wind turbine speed term; the agent reward function consists of two parts. The reward weighting factors are divided into two parts: the first part of the reward is related to the grid frequency deviation, and the agent's behavior tends to be related to the wind turbine inertia support effect; the second part of the reward is related to the wind turbine speed, and the agent's behavior tends to be related to the unit stability. The The expression is as follows: ； ； In the formula, This indicates the fan tripping speed; when the fan speed deviates from the set upper and lower penalty boundaries... The agent's action output is penalized, and the greater the deviation, the greater the penalty; when the fan speed drops to the cut-off speed, the penalty value becomes infinite, and the training for the current round ends.

2. The wind turbine grid inertia support method based on deep reinforcement learning according to claim 1, characterized in that, step In the text, the description of the training wind speed is as follows: Training an intelligent agent involves the following steps: Initialize the neural network parameters; the neural network is divided into a policy network. and value network Two parts, Part of it is responsible for outputting actions based on the state. Partial output Each part is further divided into the current network and the target network; exist In the current network, state Input, output action ;action Acting on the environment, the state is updated to Return reward ; The dataset Store in the experience pool; Randomly drawn from the experience pool Datasets , To calculate the current target value The calculation formula is as follows: ； In the formula, express Value attenuation factor, with a range of values ​​of ; According to the current value and target value Calculate the mean square error and update it via backpropagation. Current network parameters ; with the current Value data usage Optimizer update Current network parameters ; , The target network parameters are periodically copied from the current network to achieve soft updates of network parameters. 。 3. The method for supporting the grid inertia of wind turbine generators based on deep reinforcement learning according to claim 1, characterized in that, step In the middle, intelligent agents The neural network parameters have converged. The learning performance of the agent was tested under the test wind speed. The inertial support power was output online directly based on the wind turbine speed and the grid frequency deviation, so as to realize the inertial support of the wind turbine to the grid.

Images