High-response wind power plant control strategy design method realized based on reinforcement learning and MPC combination method
By combining reinforcement learning and MPC control strategies, the power distribution and response speed of wind farms are optimized, solving the stability and grid-friendliness issues of wind farms under uncertain environments and achieving the control effect of high-response wind farms.
Patent Information
- Application Number
- CN202410595665.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-14
- Publication Date
- 2024-11-15
AI Technical Summary
Existing wind farm control strategies struggle to balance computational load with control effectiveness, fail to adequately address uncertainties, exhibit poor operational safety under unknown disturbances, and lack dynamic adaptability to environmental uncertainties, leading to grid frequency fluctuations and poor wind turbine performance.
By combining reinforcement learning and model predictive control (MPC), a baseline controller is designed by establishing a nonlinear wind farm model and integrating online control logic to optimize power allocation, improve system response speed and stability, and achieve active scheduling capability.
It improves the wind farm's response speed and stability to environmental changes, optimizes the stability of output power, enhances its grid friendliness, and provides rapid response and proactive support capabilities, thereby improving the overall efficiency and safety of the wind farm.
Smart Images

Figure CN118971168A_ABST
Abstract
Description
Technical Field
[0003] The present invention relates to the field of wind power technology, and in particular to a wind farm control strategy design method based on reinforcement learning and MPC control strategy. Background Art
[0005] As an important part of clean energy, wind farms have achieved remarkable success worldwide. However, challenges have also gradually emerged, one of which is the adverse impact of the wake effect on the performance of wind turbines. The wake effect refers to the residual wind flow after the wind passes through a wind turbine, which affects the wind speed and direction of subsequent wind turbines, causing them to generate less electricity. In order to better deal with this problem, we need to further improve the control strategy of wind farms so that they can maintain efficient operation in complex and changing environments.
[0006] At the same time, friendly wind farms are increasingly becoming the construction standard for wind farms. When new energy generation started, it was once input into the grid as a source of interference. When wind farms started in China in 2011-2012, there were many large-scale wind turbine disconnection accidents. Even a small wind turbine of 1MW could cause a large range of fluctuations in the grid frequency. Therefore, the concept of friendly wind farms was proposed, requiring wind turbines to have grid adaptability: when the grid environment changes, wind turbines should not disconnect from the grid and continue to operate. Although the low voltage ride-through capability of wind farms has been increasingly improved since then, a grid frequency drop accident still occurred in 2016. Therefore, since 2016, high-response wind farms have added new connotations: when the grid fails or is disturbed, in addition to continuing to operate without disconnecting from the grid, wind turbines should be able to provide active support to the grid, which requires wind farms to have the ability to respond quickly.
[0007] The current problems with wind farm control include the difficulty in balancing the amount of calculation and the control effect, and the failure to fully consider how to deal with uncertain factors. In particular, when there are unknown disturbances that threaten the safety of wind turbine operation, the wind turbines are usually hard-switch to safety mode, and there is no ability to perform additional learning on such disturbances. At the same time, wind turbines under safety constraints usually only guarantee basic operation, with poor performance, making it difficult to effectively utilize wind energy.
[0008] In the current research on friendly wind farms, frequency regulation technology has basically matured, but there is less research on reducing grid frequency fluctuations based on prediction or proactive advance booking of future strategies. Existing research usually determines power generation targets based on the rated or historical average power generation of each wind turbine in the wind farm, lacks dynamic judgment and adaptability to environmental uncertainties, which is also the main reason for grid frequency fluctuations. Summary of the invention
[0010] The purpose of the present invention is to solve the problems existing in the prior art. Based on the reinforcement learning collaborative control framework, by combining robust MPC and reinforcement learning, the environmental state is predicted based on historical data, and the predicted state is used for the online controller to predict power distribution in advance, thereby accelerating the system's perception and response speed to environmental changes, ensuring the friendliness of the wind turbine to the power grid, enabling the system to have the ability of active scheduling, and optimizing the output power stability.
[0011] In order to achieve the above-mentioned purpose, the present invention adopts the following technical methods
[0012] accomplish:
[0013] The high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC includes the following steps:
[0014] S1. Establishment of nonlinear wind farm model;
[0015] S2, design benchmark controller;
[0016] S3. Design RL+MPC control framework;
[0017] In step S1, a single wind turbine and a wind farm model are designed for a high-response wind farm target.
[0018] In step S2, a reference controller is established based on the MPC concept according to the wind farm model established in step S1.
[0019] In step S3, based on the reference controller in step S2, online control logic is added to build an online control framework to achieve a high-response wind farm, so that the control target is changed from maximizing power generation to the fastest response to a given target and actively adjusting the power allocation target.
[0020] Compared with the prior art, the beneficial effects of the present invention are:
[0021] 1. Comprehensive control strategy that fully considers uncertainty: Introducing reinforcement learning as a control method to combat uncertainty, in order to cope with uncertainty and external disturbances in wind farms. This comprehensive control strategy enables the system to adapt more flexibly to changes in actual working conditions, improving the stability and efficiency of the entire wind farm;
[0022] 2. Realize high-response wind farms: Integrate reinforcement learning into MPC control to improve the response speed of the entire system, enable the system to have the ability to actively dispatch, optimize the stability of output power, and be friendly to the power grid. On the one hand, pre-dispatching is achieved, and on the other hand, the experience of MPC can also act on the reinforcement learning agent to improve its ability to predict environmental uncertainty in uncertainty; BRIEF DESCRIPTION OF THE DRAWINGS
[0024] AttachedFigure 1 is a flow chart of the present invention;
[0025] Attached Figure 2 This is a schematic diagram of the control framework of the present invention; DETAILED DESCRIPTION
[0027] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.
[0028] Example: As shown in the attached Figure 2 As shown, the present invention describes a high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC, a high-response wind farm model is constructed, and the proposed comprehensive control strategy based on the combination of reinforcement learning and MPC can more flexibly adapt to changes in actual working conditions and improve the stability and efficiency of the entire wind farm.
[0029] Includes steps:
[0030] S1. Establishment of nonlinear wind farm model:
[0031] The system model considering uncertainty and nonlinearity is:
[0032]
[0033] in is the dynamic equation of the nonlinear system; represents the system disturbance, , express The set of perturbations to which it belongs; , is the constraint function.
[0034] S2. Design base controller:
[0035] S21. Prediction model establishment:
[0036] based on At time , the prediction step is N, and the state prediction of the system is as follows:
[0037]
[0038]
[0039]
[0040]
[0041] in, represents a function compounded N times according to the above rule. Then we can have the equation:
[0042]
[0043] in represents the function matrix that contains N steps and satisfies the above equation, and:
[0044]
[0045] Then we have:
[0046]
[0047] in , , is the control sequence to be solved.
[0048] S22, MPC controller design method:
[0049] Assume the cost function is:
[0050]
[0051] in:
[0052] It's every moment Performance target item, indicating the system status and control input the “cost”;
[0053] It's the final moment The performance objective item is usually a terminal constraint;
[0054] is a robustness-related term that takes uncertainty into account The impact of. This can be the norm of uncertainty, performance loss in the worst case, etc.
[0055] is the weight coefficient used to balance the robustness term at each moment.
[0056] Solve Minimization control strategy , and take the first item, which is the current moment , and obtain the benchmark control action of MPC .
[0057] S3. Design RL+MPC control framework;
[0058] After obtaining the baseline control effect of S22, consider using the reinforcement learning PPO algorithm as an auxiliary control effect. PPO uses the value function network Simultaneous estimation and The policy network continuously updates the policy trajectory based on the estimated results to calculate the control effect , and obtain comprehensive control effect It acts on the controlled object and finally continuously updates its own strategy network and value function network by sensing changes in environmental parameters and state errors.
[0059] In summary, the algorithm flow combining MPC and RL is as follows:
[0060] 1) Initialize PPO: Initialize system model parameters , Strategy network parameters , value function network parameters , learning rate , Discount Factor , Hyperparameters in the PPO loss function ; Initialize the structure of the policy network (Actor) and the value function network (Critic).
[0061] 2) In At this moment, measure the actual environmental state variables , according to the current model , calculate the MPC benchmark control action ;
[0062] 3) Policy network based on ,make , let the current strategy Generate the trajectory and get , ,according to Correction time Value, where The discount factor for reinforcement learning control.
[0063] 4) Value Function Network Calculate the advantage estimate: is the strategy trajectory length, estimating the advantage value at each time step :
[0064]
[0065] in ;
[0066] It is set as a function related to the dynamically changing power generation instructions and the operation status of the power grid. The adjustment time becomes an important performance indicator to be taken into consideration. At the same time, the PPO loss is calculated: the current policy network and the old policy network are used to calculate the PPO loss. Including policy loss and entropy regularization term;
[0067] consider With the previous moment ,calculate
[0068] .
[0069] .
[0070] 5) Update the strategy network and value function network. In addition to the update method in the PPO algorithm, let Join the update of network parameter loss function.
[0071] 6) Update the system model in MPC at regular intervals .
[0072] Execute (2) to (6) in a loop.
Claims
1. A high-response wind farm control strategy design method based on a combination of reinforcement learning and MPC, characterized in that: Includes steps: S1. Establishment of nonlinear wind farm model; S2, design benchmark controller; S3, designing an RL+MPC online control framework. In step S1, a single wind turbine and wind farm model are designed for the high-response wind farm target. In step S2, a benchmark controller is established based on the wind farm model established in step S1 and based on the idea of MPC and reinforcement learning. In step S3, based on the benchmark controller in step S2, online control logic is added, and an online control framework is built to achieve a high-response wind farm, so that the control target is changed from maximizing power generation to the fastest response to a given target and actively adjusting the power allocation target.
2. The high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC according to claim 1 is characterized by: In step S1, a nonlinear wind farm model is established.
3. The method for estimating periodic disturbance of a fan based on iterative learning control strategy according to claim 1 is characterized in that: In step S2, the steps of designing a benchmark controller are as follows: first, for the high-response wind farm model in step S1, based on the idea of MPC, a nonlinear state equation of the wind farm is constructed, and then a robust MPC controller is constructed as a benchmark controller to ensure the robustness and safety of the system.
4. The high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC according to claim 1 is characterized in that: In step S2, the steps of designing a baseline controller include: S21. Design an MPC benchmark controller.
5. The high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC according to claim 1 is characterized in that: In step S3, designing the RL+MPC control framework includes the following steps: S31. Design RL+MPC controller. S32. Design an active power regulation and dispatching strategy for wind farms. Based on the controller, add control logic to change the control target from maximizing power generation to the fastest response to a given target and actively adjust the power allocation target.
6. The high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC according to claim 5 is characterized in that: On the basis of establishing a general nonlinear model, the dynamically changing power generation instructions and grid operation conditions are taken into consideration when designing the cost function.
7. The high-response wind farm control strategy design method based on the combination of reinforcement learning and MPC according to claim 5 is characterized in that: The step S21 designs a reference controller based on the MPC concept to ensure the basic dynamic performance and safety of the system.
