Method for controlling a wind farm using a reinforcement learning method

Abstract

The present invention relates to a method for controlling a wind turbine farm, in which a reinforcement learning (RL) method is implemented in a decentralized manner (for each wind turbine), for which the reward (REC) is calculated as a function of a wake propagation time (DEL). Thus, the reward is well representative of the effect of the last action (for example the control of the previous yaw). Figure 2 to be published

Description

Title of the invention: Method for controlling a wind farm using a reinforcement learning method Technical field

[0001] The present invention relates to the field of controlling a wind farm to maximize the power produced.

[0002] A wind farm, also called a wind park or wind power plant, is a site grouping together a plurality of wind turbines producing electricity. This site can be on land or at sea. A distinction is thus made between onshore wind farms and "offshore" wind farms, i.e. at sea.

[0003] The wind turbines on these farms are generally horizontal axis wind turbines that have a system to orient the horizontal axis of rotation in the direction of the wind, in order to maximize the energy recovered by the wind turbine. A wind turbine allows the kinetic energy of the wind to be transformed into electrical or mechanical energy. For the conversion of wind into electrical energy, it consists of the following elements:

[0004] - a mast allowing a rotor to be placed at a sufficient height to allow its movement (necessary for horizontal axis wind turbines) or to place this rotor at a height allowing it to be driven by a stronger and more regular wind than at ground level. The mast may house some of the electrical and electronic components (modulator, control, multiplier, generator, etc.);

[0005] - a nacelle mounted at the top of the mast, housing mechanical components, tire matics, certain electrical and electronic components, necessary for the operation of the machine (modulator, control, multiplier, generator, etc.). The nacelle can rotate to orient the rotor in the right direction;

[0006] - a rotor, fixed to the nacelle, comprising several blades (generally three) and the nose of the wind turbine. The rotor is driven by wind energy, it is connected by a mechanical shaft directly or indirectly (via a gearbox and mechanical shaft system) to an electrical machine (electric generator, etc.) which converts the collected energy into electrical energy. The rotor is potentially equipped with control systems such as variable-angle blades or aerodynamic brakes;

[0007] - possibly a transmission, composed in particular of two axes (shaft rotor mechanics and mechanical shaft of the electric machine) connected by a multiplier (gearbox).

[0008] Since the early 1990s, wind energy has seen a resurgence of interest, particularly in the European Union where the annual growth rate is around 20 %. This growth is attributed to the inherent possibility of carbon-free electricity production. To support this growth, the efficiency of wind turbines and wind farms must continue to improve. The prospect of increasing wind energy production requires the development of efficient production tools and advanced control tools to improve machine performance. Wind turbines are designed to produce electricity as cheaply as possible.

[0009] For this power regulation, controllers are designed for variable speed wind turbines. The objectives of the controllers are to maximize the recovered electrical power, minimize rotor speed fluctuations and minimize fatigue and extreme moments of the structure (blades, mast and platform). Prior art

[0010] Wind farms are subject to a phenomenon commonly referred to as the "wake effect", when disturbances created by turbines located upstream of the wind farm create suboptimal electricity production conditions for other turbines. In fact, downstream of the wind turbine a vortex wake is formed, and in this wake the average wind speed is reduced, because the wind turbine has captured some of the kinetic energy of the wind and the turbulence intensity is increased.

[0011] A common strategy for maximizing the energy production of a wind turbine is to orient its rotor so that it faces the wind. The angle between the rotor and the wind direction, called the yaw angle, is then 0°. [Fig.l] illustrates schematically and in a non-limiting manner the yaw angle. [Fig.l] is a top view of a wind turbine. The wind turbine comprises blades 1 and a nacelle 2, oriented in a direction AA. The wind is represented by the arrow U, having a direction DD. The angle y between the direction AA and the direction DD is the yaw angle. When the turbine of the wind turbine is aligned with the wind direction, this angle y is zero.

[0012] In wind farms, however, applying this strategy (zero yaw angle) to all turbines, in a so-called "greedy" method, makes the farm subject to the so-called wake effect: when a wind turbine extracts energy from the wind, the downstream wind speed decreases and its turbulence increases. This leads to suboptimal conditions for the energy production of the downstream turbines, with losses in total production reaching 40% at sea. A number of controllable actuators can be used to reduce this effect: power capture can be influenced by controlling the blade orientation or the generator torque, the wake of a turbine can be deflected under the downstream turbines by tilting the plane of the rotor, or sideways by changing the yaw, a technique known as wake steering.

[0013] To correct this problem, one strategy is to use yaw actuators to misalign the turbines with respect to the direction of the incident wind: this allows for wake redirection to limit the impact on downstream turbines. Finding the optimal yaw angles (which maximize the total electrical power of the wind farm) is a complex problem.

[0014] For this complex problem, traditional control approaches can be considered: they use a wind propagation model in a wind farm, and optimize the yaw angles with respect to this approximation. Various models using analytical approximations or numerical calculations have been proposed, but such models, however, lack accuracy and ignore turbulent wind dynamics and wake propagation, leading to poor estimation of wake effects in wind farms. There are higher-fidelity models, which take into account wake advection, deflection, meandering and merging, but they are very expensive in terms of time and computational resources, which discourages their use for real-time optimization.

[0015] It is possible to overcome these constraints by using model-free methods. Reinforcement learning (RL) is one example: its methods learn by trial and error, and deduce optimal actions solely by observing a system's responses to changes in input. This online learning approach is particularly interesting because of modeling uncertainties, which make it necessary to forget some suboptimal behaviors learned in modeling in the field. But the combination with a decentralized approach is not obvious: decentralized algorithms limit the observability of the problem for each turbine, making their environment non-stationary.Meanwhile, wind propagation dynamics prevent observation of a controller's impact until the wake has propagated downstream, and make assigning the power variation to the correct controller more difficult due to the cumulative effect of multiple wakes.

[0016] Reinforcement learning methods have also been used for automatic generation control via yaw control for a wind farm: the paper “S. Vijayshankar, P. Stanfel, J. King, E. Spyrou, and K. Johnson, Deep reinforcement learning for automatic generation control of wind farms, in 2021 American Control Conference (ACC), pp. 1796+1802, 2021” uses the centralized deep learning algorithm Deep Deterministic Policy Gradient (DDPG) with the axial induction factor as the control variable. In the paper “H. Dong, J. Zhang, and X. Zhao, Intelligent wind farm control via deep reinforcement learning and high-fidelity simulations, Applied Energy, vol. 292, no. C, 2021”, DDPG is combined with offline supervised learning, resulting in a significant increase in the evaluated output power in high-fidelity wind farm simulations. These centralized methods, however, see their problem dimension increase with the number of turbines, raising the question of scalability to larger wind farms. In particular, in this second paper (DONG et al., 2021), the authors exploit a symmetry specific to the turbine layout in their use case to reduce the problem dimension, but this does not transfer to other farm layouts.

[0017] To solve this problem, decentralized learning methods have been tested and have resulted in a significant increase in total electricity production on several wind farm simulations. But the majority of these methods ignore the wake propagation times in the wind farm (notably in the paper “Z. Xu, H. Geng, B. Chu, M. Qian, and N. Tan, Model-free optimization scheme for efficiency improvement of wind farm using decentralized reinforcement learning, IFAC-PapersOnLine, vol. 53, no. 2, pp. 12103+12108, 2020. 21st IF AC World Congress”). These methods have been tested on simulators that neglect the wake propagation times. However, if we apply these methods while taking into account the wake propagation times, these methods are no longer as efficient, especially for convergence and for the power produced. Furthermore, it is demonstrated in the paper “P. Stanfel, K. Johnson, CJ Bay, and J.King, A distributed reinforcement learning yaw control approach for wind farm energy capture maximization*, in 2020 American Control Conference (ACC), pp. 4065+4070, 2020 » that effective yaw control methods can be rendered completely ineffective when wake propagation time is taken into account in more realistic simulations. .

[0018] Patent applications EP3997335 and US2022186709 also describe reinforcement learning methods. However, neither of these methods allows management of the wake propagation delay in the wind farm. Furthermore, in patent application EP3997335, only a portion of a wind farm is considered, which does not allow optimal control over the entire wind farm. Furthermore, in patent application US2022186709, the resolved objective concerns the stabilization of the output power of the wind farm under varied wind conditions. This method therefore does not allow in particular maximization of the power produced.

[0019] Few strategies have been used to address wake propagation time in wind farm optimization problems. In the aforementioned paper (DONG et al., 2021), the problem is delegated to a learning algorithm that learns to predict the impact of a yaw change from measurements made at constant yaw before deployment in the wind farm. However, this decouples the reward signal from the current output of the wind farm, which limits the adaptability of the algorithm and makes it vulnerable to the failure of an individual turbine. In the aforementioned paper (STANFEL et al., 2020) the wind farm is divided into subsets within which only one turbine at a time is allowed to change yaw, a so-called “locking” strategy. However, this locking strategy excessively slows down convergence for all agents. Summary of the invention

[0020] The aim of the invention is to control a wind farm in real time to maximize the electrical power produced, regardless of the implementation of the wind turbines within the wind farm, and with rapid convergence. For this purpose, the present invention relates to a method for controlling a wind farm, in which a reinforcement learning method is implemented in a decentralized manner (for each wind turbine), for which the reward is calculated as a function of a wake propagation time. Thus, the reward is well representative of the effect of the last action (control of the previous yaw), which makes it possible to maximize the power produced by the wind farm. The implementation of a reinforcement learning method for each wind turbine allows the adaptation of the method for any wind farm, and also allows rapid convergence.

[0021] Furthermore, the invention relates to a wind farm capable of implementing the control method.

[0022] The invention relates to a method for controlling a wind turbine farm, each wind turbine of said wind turbine farm comprising an actuator for modifying an operating point of said wind turbine, in particular the yaw angle of said wind turbine, the yaw angle being the angle formed between a turbine of said wind turbine and a wind direction. For this method, the following steps are implemented: a. We acquire a power produced by each wind turbine, a wind speed and direction, and the yaw angle of each wind turbine; b. For each wind turbine, a wake propagation delay formed by said wind turbine is determined, as a function of said acquired wind speed and direction, as well as an arrangement of said wind turbines within said wind farm; c. For each wind turbine, a value of a reward is determined representing an impact of a control of the wind turbine in particular on the sum of the powers produced by all the wind turbines of said wind farm, said reward being calculated by taking into account said propagation delay of determined wake; d. For each wind turbine, a reinforcement learning method is applied to determine a target operating point of said wind turbine based on said determined reward and the prior yaw angle; and e. The operating point of each wind turbine is controlled by applying said target operating point determined by means of said actuator.

[0023] According to one embodiment, said wake propagation delay is determined by means of a Taylor fixed turbulence hypothesis.

[0024] According to one implementation, said wake propagation delay is determined by taking into account a wake limit distance.

[0025] Advantageously, said reinforcement learning method is a Watkins Q-learning method.

[0026] According to one aspect, for each wind turbine, said reward is determined when the time elapsed since the last check is greater than said determined wake propagation delay.

[0027] According to one configuration, said yaw angle control is a variation of a fixed step of said yaw angle.

[0028] Advantageously, said wind speed and direction are acquired by measurement using a LiDAR sensor, or an anemometer, or by measurement using a real-time data acquisition and control system.

[0029] According to one embodiment, the acquired yaw angle is said prior controlled yaw angle.

[0030] According to one implementation, said reward is determined by taking into account an average of the powers produced during a predetermined time interval.

[0031] Furthermore, the invention relates to a wind turbine farm, for which each wind turbine of said wind turbine farm comprises an actuator for modifying an operating point of the wind turbine, in particular the yaw angle of said wind turbine, the yaw angle being the angle formed between the turbine of said wind turbine and a wind direction. Said wind turbine farm comprises computer means for implementing the method for controlling a wind turbine farm according to one of the preceding characteristics.

[0032] Other characteristics and advantages of the method and system according to the invention will appear on reading the following description of non-limiting examples of embodiments, with reference to the figures appended and described below. List of figures

[0033] [Fig.l]

[0034] [Fig.l], already described, illustrates the yaw angle of a wind turbine.

[0035] [Fig.2]

[0036] [Fig.2] illustrates the steps of the control method according to one embodiment of the invention.

[0037] [Fig.3]

[0038] [Fig.3] illustrates two wind farm implementations for two examples.

[0039] [Fig.4]

[0040] [Fig.4] illustrates, for the two examples of [Fig.3], the wake effect of each wind turbine, with a control method according to the prior art.

[0041] [Fig.5]

[0042] [Fig.5] illustrates, for the first example, curves of total power produced by the wind farm as a function of time, for a method according to the prior art and for the method according to one embodiment.

[0043] [Fig.6]

[0044] [Fig.6] illustrates, for the first example, the curves of the yaw angle of each wind turbine as a function of time.

[0045] [Fig.7]

[0046] [Fig.7] illustrates, for the first example, the curves of power produced by each wind turbine as a function of time.

[0047] [Fig. 8]

[0048] [Fig.8] illustrates, for the second example, curves of total power produced by the wind farm as a function of time, for a method according to the prior art and for the method according to one embodiment.

[0049] [Fig.9]

[0050] [Fig.9] illustrates, for the second example, the curves of the yaw angle of each wind turbine as a function of time.

[0051] [Fig. 10]

[0052] [Fig. 10] illustrates, for the second example, the curves of power produced by each wind turbine as a function of time. Description of the embodiments

[0053] The present invention relates to a method for real-time control of a wind farm. A wind farm, also called a wind park or wind power plant, is a site grouping together a plurality of wind turbines producing electricity. Each wind turbine (also called a turbine) in the wind farm comprises an actuator for modifying an operating point of the wind turbine. An example of an operating point may be the yaw angle of the wind turbine. Other operating points may include restricting the wind turbine, or modifying the power curve of the wind turbine. The position of the wind turbines within the wind farm, also called wind turbine arrangement or wind turbine implementation, is previously known.

[0054] In the remainder of the description, only the yaw angle control is described, however other operating points can be controlled.

[0055] The method according to the invention comprises the following steps:

[0056] 1) Acquisition of the powers produced, the speed and direction of the wind, the angles of lace

[0057] 2) Determination of the propagation time of the wake of each wind turbine

[0058] 3) Determination of a reward for each wind turbine

[0059] 4) Application of a reinforcement learning method for each wind turbine

[0060] 5) Control of each wind turbine

[0061] Steps 2 to 4 can be implemented by computer means, in particular a computer, a processor or a calculator. The steps are detailed in the remainder of the description. The steps are carried out in parallel for each wind turbine, thus the method is decentralized. In addition, the method does not implement any physical model, and relies on real data measured within the farm, which makes it competitive in terms of calculation time, representativeness and accuracy. These steps are carried out continuously in real time; the steps are repeated at each time step.

[0062] [Fig. 2] illustrates, schematically and in a non-limiting manner, the steps of the wind farm control method according to one embodiment of the invention. The powers produced by all the wind turbines in the wind farm, the wind speed and direction, and the yaw angles of the wind turbines (y) are acquired (ACQ). For each wind turbine in the wind farm, a propagation delay (DEL) of the wake of the wind turbine is determined as a function of the wind speed and direction and the positioning of the wind turbines. For each wind turbine, a reward (REC) is determined representing the impact of controlling the wind turbine on the sum of the powers produced, taking into account the propagation delay.For each wind turbine, a reinforcement learning (RL) method is applied to determine a target wind turbine yaw angle (or target operating point), from a previous yaw angle (y) (or previous operating point) and the determined reward. Each wind turbine is controlled (CON) by applying the target yaw angle (or target operating point) to the wind turbine.

[0063] 1) Acquisition of the powers produced, the speed and direction of the wind, the angles of lace

[0064] During this step, the following are acquired by means of measurement or estimation: - The electrical power produced by each wind turbine, - The wind speed, in particular the speed of the undisturbed wind measured at the entrance to the wind farm, - Wind direction, including wind direction at the entrance to the wind farm, and - The yaw angle of each wind turbine.

[0065] According to one embodiment, the wind speed and direction can be measured, in particular by means of a LiDAR sensor (acronym for the English expression "light detection and ranging" which can be translated as remote sensing by Laser), an anemometer, or by measurements using a SCADA (Supervisory Control And Data Acquisition) real-time data acquisition system, or by any similar sensor. A SCADA real-time data acquisition system is a large-scale remote management system making it possible to process a large number of telemetry data in real time and to remotely control technical installations. It is an industrial technology in the field of instrumentation, the implementations of which can be considered as instrumentation structures including a middleware type layer.From these measurements, the undisturbed wind speed at the entrance to the wind farm can be deduced, the entrance to the wind farm being defined as a function of the wind direction.

[0066] According to one implementation, the electrical power produced by each wind turbine may be measured by the real-time control and data acquisition SCADA system, or by measurements of current and voltage delivered by each wind turbine.

[0067] According to one aspect of the invention, the yaw angle of each acquired wind turbine may be the yaw angle monitored at a previous instant. Alternatively, the yaw angle may be measured by a sensor, in particular an angular sensor.

[0068] 2) Determination of the propagation time of the wake of each wind turbine

[0069] During this step, for each wind turbine, a propagation delay of the wake formed by the wind turbine is determined, as a function of the wind speed and direction (acquired in step 1), and as a function of the implementation of the wind turbines within the wind farm. The wake effect corresponds to the disturbances and turbulence formed by a wind turbine in the wind flow. The wake formed by a wind turbine impacts the wind turbines downstream of this wind turbine in the wind direction. The implementation of the wind turbines corresponds to the arrangement of the wind turbines within the farm. In other words, the implementation of the wind turbines corresponds to the relative position of the wind turbines within the wind farm. The implementation of the wind turbines and the wind speed and direction make it possible to determine the wind turbines upstream and downstream of each wind turbine. This step makes it possible to take into account the propagation time of the wake in the optimization problems of wind farms.

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079] According to one embodiment of the invention, the wake propagation delay for each wind turbine can be approximated using Taylor's frozen wake hypothesis. For example, the propagation delay from upstream turbine i to a downstream turbine j can be determined using the following formula: æ . — with ci,j the distance between turbines i and i along the wind direction axis, and the undisturbed wind speed measured at the entrance to the wind farm (obtained in step 1. According to one implementation of this embodiment of the invention, the propagation delay formula may include a multiplier coefficient greater than 1. Alternatively, the propagation delay formula can take into account induction zones upstream of the rotor, which allows the fixed turbulence hypothesis to be relaxed with a more realistic duration taking into account the associated decelerations. According to one aspect of the invention, the determination of the wake propagation delay may take into account a wake limit distance. Indeed, the disturbances in the wind field due to a turbine i become less significant as the distance from its rotor increases and the wind returns to conditions similar to the free flow at the entrance to the farm, its wake effect may therefore be negligible beyond a certain distance. Thus, thanks to this limit distance, the propagation delay to any turbine j located beyond this distance may not be taken into account to calculate the reward reception delay. For example, a wake propagation time matrix D;j can be determined for each pair of turbines (i, j) in the wind farm. This wake propagation time matrix D^j can be written, for all pairs of turbines i, j with 0 < i, j < M, with M the number of wind turbines in the wind farm: mx dj j, if j downstream of i and Cij < dhm ^ij — * 0 otherwise With m a multiplier coefficient greater than 1, dq propagation delay (for example from Taylor's frozen turbulence hypothesis) from the upstream turbine i to a downstream turbine j, e'J a distance between the two turbines i and j along the wind direction axis, diim a wake limit distance. 3) Determination of a reward for each wind turbine In this step, for each wind turbine, a reward value is determined representing the impact of controlling the wind turbine, in particular on the sum of the powers produced by all the wind turbines in the wind farm, the reward being determined according to the wake propagation delay determined in step 2. The reward is a parameter of the learning method by ren forcement, which will be described in step 4. In other words, the reward is a value of a reward function of a reinforcement learning method, which will be described in step 4. The reward associated with a state allows the method to automatically learn an action. If the reward is positive, the action taken previously is favorable, which favors this action by machine learning. Since a change in a yaw angle of a wind turbine (or any other operating point) has a delayed impact on downstream wind turbines, the reward is determined with a delay based on the wake propagation delay determined in step 2.

[0080] According to one embodiment, the reward is determined by measuring the energy production of the turbines downstream of a wind turbine considered at the time of the impact estimated by means of the propagation delay. Thus, the present invention makes it possible to reduce uncertainties. Each wind turbine therefore receives a different reward function depending on its location in the wind farm. Finally, to reduce the influence of the nominal power on the evaluation of the impact of the different turns, percentage increases rather than raw values ​​can be taken into account. The sign of the measured variation can be used as a reward signal and a threshold can be applied to filter the noise. For example, for each turbine i (between 1 and M, M being the number of wind turbines in the wind farm), a reward rijk can be defined for each time step k:

[0081] fi . x l 0 otherwise

[0082] With ô a positive threshold, y 1=^^and y _ D , Pj,k the power produced by wind turbine j at time step k, Pp<+D; । the power produced by wind turbine j at time step k to which we add the wake propagation delay D;j between turbine i and j (determined in the previous step). Thus, the reward is positive if the sum of the powers has increased by a (positive) amount greater than a positive threshold, the reward is negative if the sum of the powers has decreased by an amount less than or equal to a negative threshold, and the reward is zero in the interval.

[0083] Furthermore, to take into account the variation in instantaneous power due to wind turbulence, an average of the power measurements produced by each turbine can be taken: by setting X > 1 the size of the averaging window. In other words, the reward can be determined by taking into account an average of the powers produced by the wind turbines during a predetermined time interval. (average window size X).

[0084] According to an implementation of the invention, the reward can be determined for each wind turbine if the time elapsed since the last prior check is greater than the determined wake propagation delay. Thus, the reward is determined only when sufficient time has elapsed since the last prior check was carried out, so as to be able to clearly observe the impact of the last prior check on the power produced by the wind turbines.

[0085] 4) Application of a reinforcement learning method for each wind turbine

[0086] During this step, for each wind turbine, a reinforcement learning method is applied to determine a target yaw angle (or target operating point) of the wind turbine, based on the reward determined in step 3, and based on the previous yaw angle (or previous operating point). The target yaw angle corresponds to a yaw angle setpoint for the wind turbine. The target operating point corresponds to a setpoint for the operating point. Reinforcement learning consists of an autonomous agent (in this case a wind turbine in the wind farm) learning the actions to take (in this case modifying the yaw angle) from experiences, so as to optimize a quantitative reward over time (in this case the reward determined in step 3 which takes into account the wake propagation delay).The agent is immersed in an environment (in this case the wind farm and the wind), and makes decisions based on its current state (in this case the current yaw angle). In return, the environment provides the agent with a reward, which can be positive or negative. For the invention, the approach is deferred reward. The agent searches through past achievements for an optimal strategy, in the sense that it maximizes the sum of rewards over time. This online learning approach is particularly interesting due to the modeling uncertainties. In addition, this approach allows for real-time determination. Such a learning method is applied to each wind turbine in the wind farm, which allows for decentralized optimization.

[0087] According to one embodiment of the invention, the reinforcement learning method may be a Watkins Q-learning method. Such a reinforcement learning method is described in particular in the document: “C. WATKINS, P. DAYAN, TECHNICAL NOTE - Q-Learning, Machine Learning 8, 279-292 (1992), 1992 Kluwer Academie Publishers, Boston, Manufactured in the Netherlands”. Alternatively, other reinforcement learning methods may be used, in particular so-called “policy-gradient” learning methods of the actor-critic type, or any similar learning method. In these approaches, Each agent has two elements: an actor that tries to predict the optimal control for the current state, and a critic that evaluates the effectiveness of the control predicted by the critic. Algorithms of this type are called, for example: A2C (Advantage Actor Critic), DDPG (Deep Deterministic Policy Gradient), PPG (Proximal Policy Optimization).

[0088] Formally, for the embodiment implementing the reinforcement learning method implementing the Q-leaming method, we can define a Markov decision process (MDP) {S, A, r, P], with S the state space, A an action space, P the matrix of transition probabilities of the environment and r: S x A —> R a reward function. In other words, at time t, an agent located in a state si, chooses an action a< and receives an associated reward rt. In the following, the continuous time step is denoted t, and the discrete time is denoted k.

[0089] The application that associates with each observed state a corresponding action, or a probability distribution over the possible actions in that state, is called policy ir; an agent is said to follow a policy jt. The sequence of states, actions and rewards ST, aT, rr) observed by the agent when interacting with the environment is called a trajectory. The trajectory followed depends on the agent's policy, but also on the dynamics of the environment; for an action taken in a given state, the next state and the reward obtained are not always identical. We speak of transition probabilities from one state to another. In our case, for example, exogenous factors such as wind hazards can increase or decrease the reward. We therefore seek to maximize the expected return.

[0090] We can write A(s) as the subset of actions ae A available in state s. An agent interacts with the environment by following a stochastic policy a ~ ir(s), se S, ae A(s), where ir(als) is the probability of choosing action a when it is in state s. If the policy is deterministic, there exists an action a' for which ir(a'ls) = 1 and we can directly write a' = ir(s). The agent's objective is then to find an optimal strategy ji* that maximizes the expectation E of the sum to infinity of all its discounted rewards, also called discounted return:

[0091] maxE[G] = maxE Pkr(sk, ak)]

[0092] with 0 < [3 < 1 a discount factor (in other words a weighting factor), kVo an initial state, G is the discounted return, i.e. the sum of discounted future rewards, and . fc — Q the trajectory of the agent in the environment under policy jt. For a policy ji, we can define the state-action value function Qn (or q-function) for ir as the expected discounted return for a state-action pair (a,s): Q (s, a ) = E[G|s0= s, a0 = a]. That is, for any state-action pair (s, a), Q(s, a) is the expected value for an agent who chooses action a in state s, then follows policy ir for the rest of the trajectory. We can define an optimal q-function Q* such that: V (s, a), Q*(s, a) = max^ Qn(s, a)-

[0093] To search for the best policy in a given environment, one can directly attempt to learn Q*. Watkins's Q-learning algorithm keeps estimates of the q values ​​for each pair (s, a) - it is said to be tabular -, and iteratively updates an estimate Q of the optimal q function Q* at each time step k: 100941

[0095] Where TDk is the Bellman error estimator which can be defined as:

[0096] TDk = rk + p àjw à) - Ôk(Sk, ak) «eA(sfc+!)

[0097] with PG (0, 1) an update factor and 4 a learning rate at time step k. Q then converges with probability one to Q* under certain reasonably achievable conditions. Following a decentralized approach where each turbine is modeled by an agent, we consider M (M being the number of wind turbines in the wind farm) state spaces g. : S, = Y x R2 (with Y a space of admissible yaw angles, and R is the set of real numbers) such that the state observed by an agent i at each time step k is defined by:

[0098] r ir . . . respectively the limit in- Xk = 1¼ wkJ Yl i Y.. S Yls with yL and Yu lower and upper yaw values, and Yuc the yaw angle of wind turbine i at time step k and wk the wind conditions acquired at time step k, in other words the wind speed and wind direction. As a non-limiting example, the action space can be defined as A = {-1°, 0°, 1°}.

[0099] When there is a known delay c between the time an action is sent to the environment and the time the associated reward is collected by the agent, we say that the environment is reward-delayed. We further consider the time step delay cd' such that with h the sampling period - i.e. the time in seconds between two iterations of the algorithm - we obtain Cd = ^ . is then the number of time steps before the reward becomes available. This delay can be managed by allowing the agent to observe the history of all states received and actions taken during the delay. Alternatively, we can make a modification to Watkins' Q-learning method called dQ(0): at time step k, instead of updating / V. nj updates are made to the action that takes effect at time step k. This flexible approach, easily adaptable to the decentralized case, allows the management of wake propagation times in wind farms.

[0100] The present invention may implement a modification of the method of updating estimators in a reinforcement learning algorithm. For Q-learning that updated at Time Step k, the updates may be performed for the action ak-cd whose impact assessment on the wind farm production - i.e. the associated reward - has just been received. We can thus restore the correspondence by waiting for cd no time before updating Q. This approach uses a definition of a reward delay c for each agent, which should correspond to the time it takes for the impact of an action to be fully observed on downstream turbines. This time differs depending on the location of the turbine in the farm, and the decentralized approach used in step 2 allows for different reward delays to be defined for all agents.

[0101] Thus, according to one embodiment, the reward delay at time step c; for each turbine i can correspond to the number of time steps corresponding to the largest estimated delay for propagation of the wake to another turbine in the park. For this embodiment, the update of the Q-learning learning method for any wind turbine can be written:

[0102] ^(sk-c., Hk-cJ = Ôik(sk„c; ak-cJ + lk-TDdadQk

[0103] TDdadQk = rik + p max Ôk(sij<-c1+i, â)- aLk) ^Alskc J kk

[0104] with (sk, ak) ~ b(a|S],) the policy followed by an agent and guaranteeing a certain degree of exploration in the environment, PG (0, 1) an update factor and a learning rate at time step k, s the current state i.e. the current yaw angle, a is the action i.e. the yaw angle, r the reward, c; the time step corresponding to the largest estimated delay in propagation of the wake to another turbine in the park.

[0105] Once Q is determined, we deduce the action a (i.e., the target yaw angle or the target operating point). To do this, we can choose a according to a so-called exploration policy b(a / s), which chooses argmax^Q^S a') with 'a P^us greatest probability, and all other actions with non-zero probabilities.

[0106] According to one embodiment, one can choose a Boltzmann exploration policy. J \ with a T = 0.1. The higher T is, the more one explores in TF / “ the hope of discovering better actions. The smaller it is, the more we tend to systematically choose the best action according to our current estimates. It is also possible to reduce T as we learn. 5) Control of each wind turbine

[0107] In this step, each wind turbine is controlled by applying the target yaw angle (or target operating point) determined in step 4. For this step, for each wind turbine, the actuator of the operating point of the wind turbine is controlled. In particular, the actuator of the yaw angle of the wind turbine can be controlled.

[0108] According to one embodiment, the control of the yaw angle may correspond to a variation of a fixed step of the yaw angle. In other words, the yaw angle may be increased by a fixed step, or decreased by a fixed step, or may remain constant. The fixed step may be between 0.5 and 5 degrees, preferably between 0.5 and 2 degrees, and may be equal to 1 degree. This embodiment makes it possible to avoid sudden changes in the yaw angle, and consequently, sudden changes in the wake.

[0109] Alternatively, the yaw angle control may correspond to a control at a precise value of the yaw angle.

[0110] Furthermore, the invention relates to a wind turbine farm. Each wind turbine of the wind turbine farm comprises an actuator for modifying the yaw angle of the wind turbine or the operating point of the wind turbine. Furthermore, the wind turbine farm comprises computer means, in particular a computer, a processor or calculator for implementing the control method according to any one of the variants or combinations of variants described below. The wind turbine farm is therefore controlled by the computer means. In particular, the computer means make it possible to: - Acquire the power produced by wind turbines, - Acquire the yaw angles of each wind turbine, - Determine the wake propagation delay of each wind turbine, - Determine a reward for each wind turbine, - Determine a target yaw angle for each wind turbine or a target operating point, and - Control the yaw angle of each wind turbine or the operating point.

[0111] The computing resources can be centralized: the wind farm comprises a single computing unit which implements the steps of the control method, and which communicates with at least all the yaw angle actuators. Alternatively, each wind turbine comprises a computing unit, all the computing units communicating with each other.

[0112] According to one embodiment, the wind farm may comprise a wind measurement sensor, in particular a LiDAR sensor or an anemometer.

[0113] According to one aspect of the invention, the wind farm may comprise means of SCADA measurement.

[0114] According to one embodiment option, the wind farm may comprise communication means, in particular for transmitting the data acquired in step 1 and / or for communicating the target yaw angles to the controllers. Examples

[0115] The characteristics and advantages of the method according to the invention will appear more clearly on reading the application examples below.

[0116] For these examples, we consider two wind farms. The two wind farms are represented, schematically and in a non-limiting manner, in [Fig. 3]. Figure 3A corresponds to the first wind farm, which comprises three identical aligned wind turbines, and denoted T1, T2, T3. These are 5 MW wind turbines with a diameter of 126m. The turbines are separated by a distance of four diameters. For this first example, the wind U is parallel to the alignment of the wind turbines (which corresponds to the worst situation in terms of wake effect). Figure 3B corresponds to the second wind farm, which comprises two parallel lines of three identical aligned wind turbines, the wind turbines are denoted T1, T2, T3, T4, T5, and T6. These are 5 MW wind turbines with a diameter of 126m. Within an alignment, the turbines are separated by a distance of four diameters, and the two alignments are separated by a distance of four diameters.For this second example, the wind U is parallel to the wind turbine alignments (which corresponds to the worst-case situation in terms of wake effect). To reproduce realistic wind conditions, a turbulent wind simulator was used to generate a time series of 3D wind speed vectors in the flow field. The simulated wind has an average free-stream speed of 8 m / s with 8% turbulence. The examples are simulated on a FAST.Farm wind farm simulator developed by the National Renewable Energy Laboratory (NREL), which is a simulator that accounts for the complex dynamics of wake effects, including velocity deficits, advection, deflection, meandering, and wake merging in the presence of multiple turbines. For the simulations, the sampling period is set to three seconds.

[0117] [Fig.4] illustrates, schematically, the wake formed by the two wind farms, in the case where the yaw angle remains zero for all the wind turbines. Figure 4A corresponds to the wind farm of Figure 3A, and Figure 4B corresponds to the wind farm of Figure 3B. In this figure, the darker the area, the greater the wake effect. It can be seen that the wake effect is greater for the last turbine in each alignment (T3 for Figure 4A, and T3 and T6 for Figure 4B). Consequently, the power produced by the turbines, particularly those at the end of the alignments in the wind direction, is greatly disturbed, which has an impact on the total power produced by the wind farm.

[0118] The control method according to the invention was then applied, implementing the Watkins Q-learning method with deferred reward management, as described above, over a period of 600,000 s, corresponding to 230,000 iterations. The action space of each agent is limited to 3 actions: {-1°, 0°, +1°}, for a corresponding increase or reduction in the yaw angle (in other words, the target yaw angle can increase by 1°, remain constant or decrease by 1°). For the first example, we therefore learn 3 Q-tables of dimension 63 x 3, i.e. 189 parameters. For the second example, we learn 6 Q-tables of dimension 63 x 6 for 378 parameters. We use a window for the power average of X = 10 min. To evaluate the performance of the algorithm without any prior knowledge, all values ​​in the Q-table are initialized to qO = 0.15.The yaws are initialized at 0°, which corresponds to a naive and greedy strategy where all the turbines are made to face the wind.

[0119] [Fig.5] illustrates, for the first example, the curves of total power P in kW produced by the wind farm as a function of time T in seconds (this is the sum of the powers produced by all the wind turbines). The curve marked AA corresponds to a method keeping all the yaw angles at 0°, and the curve marked INV corresponds to the method according to the invention. Compared to the power produced by the greedy strategy (where all the yaws are at 0°), the method according to the invention increases the total power produced by the wind farm by 20%.

[0120] [Fig.6] illustrates, for the first example, the yaw angles y in ° of each wind turbine T1 to T3 as a function of time T in seconds. We note that the yaw angles of the first two turbines T1 and T2 in the wind direction vary greatly: these two turbines are therefore strongly misaligned with the wind direction to maximize the power produced. On the contrary, the yaw angle of the last turbine T3 in the wind direction is little modified.

[0121] [Fig.7] illustrates, for the first example, the power produced P in kW for each wind turbine T1 to T3 as a function of time T in seconds. It can be seen that the power of wind turbine T1 is degraded by its misalignment. However, the powers produced by wind turbines T2 and T3 are increased, in particular for wind turbine T3 whose power produced is multiplied by three.

[0122] [Fig.8] illustrates, for the second example, the curves of total power produced by the wind farm P in kW as a function of time T in seconds (this is the sum of the powers produced by all the wind turbines). The curve marked AA corresponds to a method keeping all the yaw angles at 0°, and the curve marked INV corresponds to the method according to the invention. Compared to the power produced by the greedy strategy (where all the yaws are at 0°), the method according to the invention increases the total power produced by the farm by 14%.

[0123] [Fig.9] illustrates, for the second example, the yaw angles y in ° of each wind turbine T1 to T6 as a function of time T in seconds. We note that the yaw angles of the first two turbines T1, T2, T4 and T5 of each alignment in the wind direction vary greatly: these two turbines are therefore strongly misaligned with the wind direction to maximize the power produced. On the contrary, the yaw angle of the last turbines T3 and T6 of the two alignments in the wind direction is little modified.

[0124] [Fig. 10] illustrates, for the second example, the power produced P in kW for each wind turbine T1 to T6 as a function of time T in seconds. It can be seen that the powers produced by wind turbines T1 and T4 are degraded by their misalignments. However, the powers produced by wind turbines T2, T3, T5 and T6 are increased, in particular for wind turbines T3 and T6 whose powers produced are multiplied by three.

[0125] It can be further noted that for both examples, the algorithm converges around 450,000 s, which corresponds to 150,000 iterations. Despite the doubling of the farm size between the two examples, the convergence time remains substantially the same, demonstrating a key advantage over centralized or turbine-locked approaches. Moreover, as expected, pairs of turbines with the same positions in the alignments converge to similar values. These results validate the management of the delayed reward by the present invention, for decentralized and delay-sensitive Q-learning, under realistic turbulent wind conditions and with dynamic wake simulation. Furthermore, these results show that the method according to the invention maximizes the total produced power, adapts to several wind farm configurations, and allows real-time control with fast convergence.

Claims

Claims

1. Method for controlling a wind farm, each wind turbine of said wind farm comprising an actuator for modifying an operating point of said wind turbine, in particular the yaw angle of said wind turbine, the yaw angle being the angle formed between a turbine (1) of said wind turbine and a wind direction (U), characterized in that the following steps are implemented: a. We acquire (ACQ) a power produced by each wind turbine, a wind speed and direction, and the yaw angle of each wind turbine; b. For each wind turbine, a wake propagation delay (WPD) formed by said wind turbine is determined, as a function of said acquired wind speed and direction, as well as an arrangement of said wind turbines within said wind farm; c. For each wind turbine, a value of a reward (REC) is determined representing an impact of a control of the wind turbine in particular on the sum of the powers produced by all the wind turbines of said wind farm, said reward being calculated by taking into account said determined wake propagation delay; d. For each wind turbine, a reinforcement learning (RL) method is applied to determine a target operating point of said wind turbine based on said determined reward and the prior yaw angle; and e. The operating point of each wind turbine is controlled (CON) by applying said target operating point determined by means of said actuator.

2.

3. A method of controlling a wind farm according to claim 1, wherein said wake propagation delay (DEL) is determined by means of a Taylor fixed turbulence hypothesis. Method for controlling a wind farm according to one of the preceding claims, in which said wake propagation delay (DEL) is determined by taking into account a limit distance of wake.

4. A method of controlling a wind farm according to one of the preceding claims, wherein said reinforcement learning (RL) method is a Watkins Q-learning method.

5. Method for controlling a wind farm according to one of the preceding claims, in which, for each wind turbine, said reward (REC) is determined when the time elapsed since the last control is greater than said determined wake propagation delay.

6. A method of controlling a wind farm according to one of the preceding claims, wherein said operating point is the yaw angle, and preferably the control of the yaw angle is a variation of a fixed step of said yaw angle.

7. A method of controlling a wind farm according to claim 6, wherein the acquired yaw angle is said prior controlled yaw angle.

8. Method for controlling a wind farm according to one of the preceding claims, in which said wind speed and direction are acquired by measurement using a LiDAR sensor, or an anemometer, or by measurement using a real-time data acquisition and control system.

9. Method for controlling a wind farm according to one of the preceding claims, in which said reward (REC) is determined by taking into account an average of the powers produced during a predetermined time interval.

10. Wind turbine farm, wherein each wind turbine of said wind turbine farm comprises an actuator for modifying an operating point of the wind turbine, in particular the yaw angle of said wind turbine, the yaw angle being the angle formed between the turbine of said wind turbine and a wind direction, characterized in that said wind turbine farm comprises computer means for implementing the method for controlling a wind turbine farm according to one of the preceding claims.

Images