Offshore wind power plant wake flow optimization control method based on deep reinforcement learning

Abstract

The invention relates to an offshore wind power plant wake flow optimization control method based on deep reinforcement learning, and the method comprises the following steps: constructing a directed graph, and for each node in the directed graph, updating the input feature of the current node through the input features of a graph neural volume accumulation and surrounding neighbor nodes, the digraph takes each offshore wind turbine in an offshore wind plant as a node and the upstream and downstream relationship between the wind turbines as an edge; and taking the input characteristics of each node as the input of the trained Actor-Critic network, obtaining an input characteristic action value, and realizing wake flow optimization control of the offshore wind plant. Compared with the prior art, the method has the advantages of enhanced generalization ability, strong timeliness and dynamic adaptability, efficient wake flow control and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of wind power generation, and in particular to a method for optimizing wake control of an offshore wind farm based on deep reinforcement learning. Background Art

[0002] In recent years, with the transformation of the global energy structure and the continuous growth of demand for renewable energy, offshore wind power has gradually become a key area of ​​wind energy development due to its huge wind energy potential and low land resource consumption. Compared with onshore wind power, offshore wind farms can be deployed in sea areas far away from densely populated areas, which not only has less impact on the environment and human activities, but also can provide higher installed capacity and long-term stable power generation performance. However, when developing offshore wind farms on a large scale, the dense layout of turbines inevitably raises the challenge of wake effect. The wake effect is due to the formation of a low-speed, high-turbulence wind field state in the downstream area after the upstream turbines intercept the wind energy, which leads to a decrease in the incoming wind speed of the downstream turbines and a decrease in power output.

[0003] The negative impact of the wake effect not only directly reduces the overall power generation efficiency of wind farms, but also threatens the economic operation of wind farms and the long-term reliability of turbines. With the expansion of wind farm scale and the increase of turbine capacity, this problem has become more prominent, especially under the complex environmental conditions of offshore wind farms. The dynamic change characteristics of the wake effect make it difficult for traditional control strategies to achieve global optimization.

[0004] In view of the above problems, how to reduce the impact of wake effect and maximize the power generation efficiency of wind farms through scientific and reasonable optimization strategies has become one of the important research directions. The main problems of the current optimization control of offshore wind farm wake can be summarized as follows:

[0005] (1) How to build a more accurate and reasonable wake control strategy. Traditional wake optimization methods usually rely on computational fluid dynamics (CFD) simulations based on physical models. Although such methods are accurate, they are computationally expensive and difficult to apply in real time. Traditional steady-state wake models oversimplify the wake structure and ignore the dynamic characteristics of the wake, making it difficult to cope with complex wind farm layouts.

[0006] (2) How to improve the generalization of the wind farm wake control strategy. As one of the classic data-driven methods, model-free reinforcement learning regards each wind turbine in the wind farm as an independent individual, and the state space and action space of the model are directly related to the scale of the wind farm. Therefore, when the scale of the wind farm (such as the number or layout of wind turbines) changes, the dimensions of the state and action change accordingly, resulting in the original model being unable to adapt to the new wind farm layout. For wind farms of different scales or layouts, traditional model-free reinforcement learning requires retraining the model, which not only increases training time and computational costs, but also may make it difficult to quickly deploy the model in actual scenarios. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide an offshore wind farm wake optimization control method based on deep reinforcement learning, so as to solve or partially solve the problems of inaccurate wake control strategy and low generalization of wake control strategy.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] The present invention provides a method for optimizing wake control of an offshore wind farm based on deep reinforcement learning, comprising the following steps:

[0010] Construct a directed graph, and for each node in the directed graph, aggregate the input features of the surrounding neighboring nodes through graph neural convolution to update the input features of the current node, wherein the directed graph uses each offshore wind turbine in the offshore wind farm as a node, and the upstream and downstream relationships between wind turbines as edges;

[0011] The input features of each node are used as the input of the trained Actor-Critic network to obtain the input feature action value and realize the optimization control of the offshore wind farm wake.

[0012] The process of obtaining the trained Actor-Critic network includes:

[0013] Taking the input features, inflow wind speed and inflow direction of each node as the state, the input feature action value as the action, and maximizing the total power generation of the offshore wind farm as the goal, the deep reinforcement learning problem is modeled based on the graph convolution Actor-Critic network, and the Actor-Critic network is trained through experience replay.

[0014] As a preferred technical solution, the process of updating the input features of the current node by aggregating the input features of surrounding neighboring nodes through graph neural convolution includes the following steps:

[0015] Calculating an adjacency matrix and a degree matrix of the directed graph;

[0016] Calculating a directed edge weight matrix representing the directed graph;

[0017] Based on the adjacency matrix, degree matrix and directed edge weight matrix, the input features of the surrounding neighbor nodes are aggregated through graph neural convolution to update the input features of the current node.

[0018] As a preferred technical solution, the directed edge weight matrix is ​​calculated using the following formula:

[0019]

[0020] in, Indicates that γ max and γ min Offshore wind turbine WT i Maximum and minimum allowed yaw angles, w γ (γ i ) is the yaw angle γ i A non-negative function of δU H (γ i ) is the yaw angle γ i When , the wind speed loss rate under the greedy strategy.

[0021] As a preferred technical solution, the updating of the input features of the current node by aggregating the input features of the surrounding neighbor nodes through graph neural convolution based on the adjacency matrix, degree matrix and directed edge weight matrix is ​​implemented by the following formula:

[0022]

[0023] Among them, A, D, and w are the adjacency matrix, degree matrix, and directed edge weight matrix of the directed graph, σ and θ are the nonlinear activation function and training parameters of the graph neural convolution, I is the unit matrix, X is the input feature of each node, and H is l , H (l+1) They respectively represent the directed graph input feature matrix and the updated directed graph input feature matrix, and I represents the unit matrix.

[0024] As a preferred technical solution, the input characteristics include the yaw angle and axial induction factor of the offshore wind turbine.

[0025] As a preferred technical solution, in the modeling process of the deep reinforcement learning problem, the action, state and control process trajectory are modeled as follows:

[0026]

[0027] Among them, s t 、a t , τ are action, state and control process trajectories respectively, S, A are action space and state space respectively, γ i,t 、ai,t are the yaw angle and axial induction factor of the offshore wind turbine at time t, Γ represents the time set, Δγ i,t , Δa i,t They represent the yaw angle action value and axial induction factor action value of the offshore wind turbine at time t, respectively. t d t are the inflow wind speed and inflow wind direction at time t respectively.

[0028] As a preferred technical solution, the offshore wind turbine yaw angle, axial induction factor, offshore wind turbine yaw angle action value, and axial induction factor action value are all within the respective maximum and minimum value ranges.

[0029] As a preferred technical solution, the experience replay training process of the graph convolutional Actor-Critic network includes the following steps:

[0030] Initialize the discount factor, Actor network parameters, Critic network parameters, number of training rounds, and experience replay buffer;

[0031] Generate input features for each offshore wind turbine using a pre-built state distribution, and get the state based on the inflow wind speed and inflow direction;

[0032] Sampling the actions of each offshore wind turbine, calculating the corresponding reward and updating the state, constructing the sampling data based on the previous state, the current action, the current reward and the next state, and putting it into the experience playback buffer;

[0033] Based on the size of the sampled data in the experience replay buffer, batches of sampled data are randomly selected, the Actor network parameters and the Critic network parameters are updated, and it is determined whether the number of training rounds reaches a maximum value, and if so, the training is terminated.

[0034] As a preferred technical solution, the update of the Actor network parameters and the Critic network parameters is implemented based on the following formula:

[0035]

[0036] Q′(s,w)←τQ(s,w)+(1-τ)Q′(s,w)

[0037] μ′(s,θ)←τμ(s,θ)+(1-τ)μ′(s,θ)

[0038] Where L is the loss function, K is the number of offshore wind turbines, and y k is the action value function Q(s k ,a k ) target value, s k 、ak are the action and state of the th offshore wind turbine respectively, μ(s,θ) and μ′(s,θ) are the strategy functions before and after update respectively, s and θ are the state and Actor network parameters respectively, Q′(s,w) is the updated action value function, w is the Critic network parameter, τ is the smoothing factor and 0≤τ≤1.

[0039] As a preferred technical solution, in the modeling process of the deep reinforcement learning problem, the goal is to maximize the total reward value of the control process trajectory:

[0040]

[0041] Among them, G t is the total reward value, ξ is the discount factor, r t+1 To perform action a t The reward obtained after i,t is the active power of wind turbine i at time t.

[0042] Compared with the prior art, the present invention has at least one of the following beneficial effects:

[0043] (1) Enhanced generalization capability: The graph neural network is used to model the wind farm structure, so that the algorithm can directly migrate the existing model when the scale or layout of the wind farm changes, without the need for retraining, significantly reducing the optimization cost.

[0044] (2) Good temporal and dynamic adaptability: By utilizing the online learning characteristics of deep reinforcement learning, the model can adjust the control strategy according to the real-time wind conditions and demonstrate strong adaptability under non-steady-state wind conditions.

[0045] (3) Efficient wake control: By comprehensively utilizing wind farm operation data and graph structure information, the model can accurately adjust the operating parameters of upstream turbines, reduce the wake losses of downstream turbines, and maximize the overall power generation efficiency of the wind farm. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is a flow chart of offshore wind farm wake optimization control in the embodiment;

[0047] Figure 2 is a directed graph of an offshore wind farm in an embodiment;

[0048] Figure 3 This is a diagram of the Actor-Critic structure based on GCN in the embodiment;

[0049] Figure 4 This is a schematic diagram of the DDPG algorithm framework in the embodiment;

[0050] Figure 5This is a flowchart of DDPG training in the embodiment;

[0051] Figure 6 This is a schematic diagram of the location layout of a 3x3 wind farm in the embodiment;

[0052] Figure 7 Schematic diagram of the DDPG and GCN-DDPG agent training process in the embodiment;

[0053] Figure 8 Flow field diagrams under the greedy strategy, DDPG and GCN-DDPG in the 3x3 wind farm in the embodiment;

[0054] Fig. 9 Flow field diagrams under the greedy strategy and GCN-DDPG in the 3x4 wind farm in the embodiment;

[0055] Fig.10 Schematic diagram of an electronic device in an embodiment. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] Example 1

[0058] The following problems exist in the above prior art:

[0059] (1) How to use data-driven methods to optimize the wind farm wake control strategy based on these empirical data;

[0060] (2) How to improve the generalization of the control strategy so that it can be applied to wind farms of different scales.

[0061] This embodiment provides an offshore wind farm wake optimization control method based on deep reinforcement learning, comprising the following steps:

[0062] Step S1, constructing an offshore wind farm directed graph model.

[0063] Preferably, step S1 may include steps S101-S102:

[0064] Step S101, graph structure representation: According to the principle of graph convolutional neural network (GCN), the wind farm is equivalent to a graph structure, each wind turbine is regarded as a node in the graph, and the connection between the upstream and downstream wind turbines is regarded as a directed edge, so as to characterize the significant wake characteristics between wind turbines.

[0065] Step S102, graph convolution operation: Obtain the corresponding adjacency matrix, degree matrix and other information according to the graph structure, where the adjacency matrix represents the connection relationship between nodes, and the degree matrix is ​​used to describe the strength of association between each node and the surrounding nodes. Use GCN to learn and propagate the feature information of each node, and then update the node's own features.

[0066] In this way, the wake influence relationship between wind turbines can be accurately captured, the coordinated behavior between wind turbines in the reinforcement learning training process can be actively guided, and invalid data sampling in the early stage can be reduced.

[0067] Step S2, constructing a deep reinforcement learning training environment and an intelligent agent training network.

[0068] Preferably, the process of building a deep reinforcement learning training environment and an agent training network is as follows:

[0069] Training environment: In the reinforcement learning framework, a centralized intelligent controller is designed as an intelligent agent to send signals to the wind farm to control the yaw angle γ of each wind turbine. i and axial induction factor a i When the wind farm is subjected to wake optimization control, it is considered as an observable environment. The wind speed and direction of the inflow at the wind farm and the operating status of each wind turbine at time t are observed by the intelligent agent and defined as s t According to s t The agent issues a command to the wind farm to execute action a t , so for the entire control process, there will be a complete trajectory

[0070] From the current state t , take action a t To the next state s t+1 The probability of the transfer function P(s t+1 |s t ,a t ) is determined by the model-free method, which assumes that the function is unknown. The state of the wind farm wake optimization control process s t and action a t Defined as:

[0071]

[0072] Among them, γ i,t and a i,t are the yaw angle and axial induction factor of the i-th wind turbine at time t, v t is the inflow wind speed at time t, d t is the inflow wind direction at time t, Δγ i,t is the action value of the yaw angle of the i-th wind turbine at time t, Δa i,tis the action value of the axial induction factor of the i-th fan at time t.

[0073] In the action space A, you can choose to increase or decrease the yaw angle and axial induction factor of the wind turbine, thereby achieving optimal control of the complete state space. Since the yaw and pitch devices of the wind turbine can be continuously adjusted, the action space is a continuous space, Δγ i,t and Δa i,t is a continuous variable, when Δa i,t ,Δγ i,t When it is positive or negative, it means increasing (decreasing) the axial induction factor and yaw angle. However, in the actual wind farm control task, γ i 、a i There are upper and lower limits. Considering the time delay of wind turbine control, in order to ensure the synchronization of yaw and axial induction factor control actions in each time step, it should be ensured that:

[0074] γ min ≤γ i,t +Δγ i,t ≤γ max

[0075] a min ≤a i,t +Δa i,t ≤a max

[0076] -Δγ max ≤Δγ i ≤Δγ max

[0077] -Δa max ≤Δa i ≤Δa max

[0078] Among them, γ max and γ min is the maximum and minimum yaw angle in wind turbine control; a min and a max are the maximum and minimum values ​​of the axial induction factor -Δγ max and Δa max The maximum action values ​​of the yaw and axial induction factors of the wind turbine in time step t respectively.

[0079] The state space S of reinforcement learning contains necessary system information such as wind farm control parameters and external environment, which is closely related to the decision variables in the action space A. Any mapping from the state space S to the action space A is called a strategy π, that is, π:S→A. In essence, the task of the agent is to find an optimal wind farm wake control strategy so that each wind turbine can make the best action in any state. In order to evaluate the performance of the strategy, r t+1 is defined as performing an action at After the reward, Among them, P i,t is the active power of wind turbine i at time t.

[0080] In the reinforcement learning training environment constructed above, the control goal is to maximize the total reward value obtained by the complete trajectory τ, that is, ξ∈[0,1] is the discount factor. Adjusting this parameter can balance the weights of current rewards and future rewards. When ξ=1, the agent considers current rewards and future rewards with equal importance. On the contrary, when ξ=0, the agent only considers the current rewards, and G t =r t .

[0081] Agent training network: DDPG's Actor-Critic framework is built using a graph convolutional neural network (GCN). GCN is used as the basis for graph structure information transmission. The wind turbine layout structure in the wind farm (relative position between wind turbines, interactions, etc.) is encoded as a directed graph, providing the Actor-Critic network with global interaction information of each wind turbine and accurate environmental modeling, helping DDPG adapt to changes in wind farm structure and scale. The Critic network uses the action value function Q as the evaluation basis, with s as input and w as a parameter, i.e. Q(s,w). The Actor strategy network uses μ π It means that s is the input, θ is the parameter, and a is the output, that is, a = μ π (s,θ), which is used to learn the control strategy of the wind farm.

[0082] Step S3, with the goal of maximizing the power generation of the offshore wind farm, optimize the yaw and axial induction factors of each wind turbine in the offshore wind farm to obtain an optimized control scheme for the wake of the offshore wind farm.

[0083] Preferably, the process of obtaining the control scheme includes steps S301-S306:

[0084] Step S301, parameter initialization: discount factor ξ, Actor network μ π The parameters θ of (s,θ), the parameters w of the Critic network Q(s,w), the number of training rounds and other parameters are initialized, and space is allocated to the experience replay buffer R.

[0085] Step S302, using the state distribution ρ μ Generate the yaw angle γ of each wind turbine at time t t and axial induction factor α t The value of , and the agent observes the wind speed and direction of the wind farm to obtain the state s t .

[0086] Step S303: The agent performs actions a of each wind turbine according to the epsilon-decay strategy. t Sampling is performed, and then a t Get reward value r t , the state is determined by s t Transfer to t+1 . And the data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer R.

[0087] Step S304: According to the amount of data stored in R, randomly select data of batch group size to update the Actor network and Critic network. For the Critic network, the mean square error is used as its loss function: where y k Q(s k ,a k ) target value, Q(s k ,a k ) is the action value function. For the Actor network, two target networks and are designed to calculate y k ,Right now

[0088]

[0089] Step S305 , updating the target network in the following formula, wherein τ is a smoothing factor and 0≤τ<<1.

[0090] Q′(s,w)←τQ(s,w)+(1-τ)Q′(s,w)

[0091] μ′(s,θ)←τμ(s,θ)+(1-τ)μ′(s,θ)

[0092] Step S306, determining whether the number of training rounds has reached a maximum value, if so, the training ends and the wind farm wake optimization control strategy is output.

[0093] In a specific control process such as Figure 1 shown.

[0094] (1) Construct a directed graph model of offshore wind farms.

[0095] According to the GCN principle, the wind farm is equivalent to a graph structure, each wind turbine is regarded as a node in the graph, and the connection between the upstream and downstream wind turbines is regarded as a directed edge, so as to characterize the significant wake characteristics between wind turbines. Then, the corresponding adjacency matrix, degree matrix and other information are obtained according to the graph structure, and graph convolution operations are performed, where the adjacency matrix represents the connection relationship between nodes, and the degree matrix is ​​used to describe the strength of the association between each node and the surrounding nodes. Finally, GCN is used to learn and propagate the characteristic information of each node, and then update the node's own characteristics. In this way, the wake influence relationship between wind turbines can be accurately captured, and the coordination behavior between wind turbines in the reinforcement learning training process can be actively guided, reducing invalid data sampling in the early stage.

[0096] Formula (1) is the process by which each node updates its own features by aggregating the information of surrounding neighboring nodes:

[0097]

[0098] Among them, σ and θ are the nonlinear activation function and training parameters of GCN respectively, w is the weight matrix of the directed edge, which is set according to the strength of the wake effect between the wind turbines, I is the unit matrix, and X is the input feature of each node (yaw angle γ i and axial induction factor a i ),Right now

[0099]

[0100] Figure 2 An example of a directed graph of an offshore wind farm is given. The detailed process of modeling the directed graph of an offshore wind farm based on GCN is as follows:

[0101] Step 1, initialize the wind farm wind turbine control parameters: the axial induction factor a of the wind turbine i and the yaw angle γ i The value range is used to randomly generate the input feature X of each node.

[0102] Step 2: According to the structural layout of wind turbines in the wind farm, a directed graph is established to characterize the wake characteristics of the wind turbines: the adjacency matrix A and degree matrix D in the directed graph are obtained, and the directed edge weight matrix w is calculated by formula (2).

[0103]

[0104] Among them, γ max and γ min are the maximum and minimum yaw angles allowed by the wind turbine WTi; w γ (γ i ) is a non-negative function of the yaw angle, which provides degrees of freedom for the weighted target; δU H (γ i ) is the yaw angle γ iWhen , the wind speed loss rate under the greedy strategy.

[0105] Step 3: According to the connection relationship between the nodes in the directed graph, the graph convolutional neural network is used to aggregate and convolve the information of each node in turn: In the first layer of GCN, node i aggregates the node information of nodes j and i to itself according to formula (1), and nodes j and i aggregate the information of node k. Similarly, in the second layer of GCN, node i aggregates the node information of nodes j and i again.

[0106] (2) Build a deep reinforcement learning training environment and an intelligent agent training network.

[0107] 1) Based on the constructed directed graph model of wind farm, a wind farm controller based on deep reinforcement learning is proposed. The controller consists of two parts: training environment and agent training network. First, a centralized intelligent controller, called agent, is designed to send signals to the wind farm to control the yaw angle γ of each wind turbine. i and axial induction factor a i When the wind farm is subjected to wake optimization control, it is considered as an observable environment. The wind speed and direction of the inflow at the wind farm and the operating status of each wind turbine at time t are observed by the intelligent agent and defined as s t According to s t The agent issues a command to the wind farm to execute action a t , so for the entire control process, there will be a complete trajectory

[0108] From the current state t , take action a t To the next state s t+1 The probability of the transfer function P(s t+1 |s t ,a t ) is determined by the model-free method, which assumes that the function is unknown. The state of the wind farm wake optimization control process s t and action a t Defined as:

[0109]

[0110] Among them, γ i,t and a i,t are the yaw angle and axial induction factor of the i-th wind turbine at time t, v t is the inflow wind speed at time t, d t is the inflow wind direction at time t, Δγ i,t is the action value of the yaw angle of the i-th wind turbine at time t, Δa i,t is the action value of the axial induction factor of the i-th fan at time t.

[0111] In the action space A, you can choose to increase or decrease the yaw angle and axial induction factor of the wind turbine, thereby achieving optimal control of the complete state space. Since the yaw and pitch devices of the wind turbine can be continuously adjusted, the action space is a continuous space, Δγ i,t and Δa i,t is a continuous variable, when Δa i,t ,Δγ i,t When it is positive or negative, it means increasing (decreasing) the axial induction factor and yaw angle. However, in the actual wind farm control task, γ i 、a i There are upper and lower limits. Considering the time delay of wind turbine control, in order to ensure the synchronization of yaw and axial induction factor control actions in each time step, it should be ensured that:

[0112] γ min ≤γ i,t +Δγ i,t ≤γ max (5)

[0113] a min va i,t +Δa i,t ≤a max (6)

[0114] -Δγ max ≤Δγ i ≤Δγ max (7)

[0115] -Δa max ≤Δa i ≤Δa max (8)

[0116] Among them, γ max and γ min is the maximum and minimum yaw angle in wind turbine control; a min and a max are the maximum and minimum values ​​of the axial induction factor -Δγ max and Δa max The maximum action values ​​of the yaw and axial induction factors of the wind turbine in time step t respectively.

[0117] The state space S of reinforcement learning contains necessary system information such as wind farm control parameters and external environment, which is closely related to the decision variables in the action space A. Any mapping from the state space S to the action space A is called a strategy π, that is, π:S→S. In essence, the task of the agent is to find an optimal wind farm wake control strategy so that each wind turbine can make the best action in any state. In order to evaluate the performance of the strategy, r t+1 is defined as performing an action a t After the reward, Among them, P i,t is the active power of wind turbine i at time t.

[0118] In the reinforcement learning training environment constructed above, the control goal is to maximize the total reward value obtained by the complete trajectory τ, that is, ξ∈[0,1] is the discount factor. Adjusting this parameter can balance the weights of current rewards and future rewards. When ξ=1, the agent considers current rewards and future rewards with equal importance. On the contrary, when ξ=0, the agent only considers the current rewards, and G t =r t .

[0119] Action-value function Q π (s t ,a t ) means in state s t Next, take action a according to strategy π t The expected return is shown in formula (9):

[0120]

[0121] Among them, the optimal action value function Q * (s t ,a t ) represents the overall strategy of maximum action value, that is, max π Q π (s t ,a t ). According to the MDP theorem, the optimal strategy is defined as Therefore, the goal of the agent is to maximize the cumulative reward J π :

[0122]

[0123] Where: ρ0 is the initial state s t The probability distribution of .

[0124] 2) Constructing an agent training network: Offshore wind farm wake optimization control is a complex nonlinear problem. In order to solve the optimal strategy π in equation (10), a graph convolutional neural network (GCN) is used to build the Actor-Critic framework of DDPG. GCN is used as the basis for graph structure information transmission. The wind turbine layout structure in the wind farm (relative position and interaction between wind turbines, etc.) is encoded as a directed graph, which provides the Actor-Critic network with global interaction information of each wind turbine and accurate environmental modeling, helping DDPG adapt to changes in wind farm structure and scale. The Critic network uses the action value function Q as the evaluation basis, with s as input and w as a parameter, i.e. Q(s,w). The Actor strategy network uses μ π It means that s is the input, θ is the parameter, and a is the output, that is, a = μ π (s,θ), which is used to learn the control strategy of the wind farm.

[0125] and Figure 4 The Actor-Critic structure diagram based on GCN and the DDPG algorithm framework are given respectively. Figure 4 It can be seen that the agent first interacts with the wind farm environment to obtain sampling data (s t ,a t ,r t ,s t+1 ) and store it in the experience buffer R. Then, the agent randomly extracts small batches of data N*(s t ,a t ,r t ,s t+1 ), train and update the Actor strategy and Critic network.

[0126] (3) With the goal of maximizing the power generation of offshore wind farms, the yaw and axial induction factors of each wind turbine in the offshore wind farm are optimized to obtain an optimized control scheme for the wake of the offshore wind farm.

[0127] Figure 5 A flowchart of DDPG training is given, and the detailed steps are as follows:

[0128] Step 1, parameter initialization: discount factor ξ, Actor network μ π The parameters θ of (s,θ), the parameters w of the Critic network Q(s,w), the number of training rounds and other parameters are initialized, and space is allocated to the experience replay buffer R.

[0129] Step 2, using the state distribution ρ μ Generate the yaw angle γ of each wind turbine at time t t and axial induction factor α tThe value of , and the agent observes the wind speed and direction of the wind farm to obtain the state s t .

[0130] Step 3: The agent takes action a for each wind turbine according to the epsilon-decay strategy. t Sampling is performed, and then a t Get reward value r t , the state is determined by s t Transfer to t+1 . And the data (s t ,a t ,r t ,s t+1 ) is stored in the experience replay buffer R.

[0131] Step 4: According to the amount of data stored in R, randomly select the batch size of data to update the Actor network and Critic network. For the Critic network, the mean square error is used as its loss function: where y k Q(s k ,a k ) target value, Q(s k ,a k ) is the action value function. For the Actor network, two target networks and are designed to calculate y k ,Right now

[0132]

[0133] Step 5: Update the target network in the following formula, where τ is the smoothing factor and 0≤τ<<1.

[0134] Q′(s,w)←τQ(s,w)+(1-τ)Q′(s,w) (13)

[0135] μ′(s,θ)←τμ(s,θ)+(1-τ)μ′(s,θ) (14)

[0136] Step 6, determine whether the number of training rounds has reached the maximum value. If it has reached the maximum value, the training ends and the wind farm wake optimization control strategy is output.

[0137] by Figure 6 The wind farm shown is a typical case. The wind turbine diameter D is 126m. The spacing between the front and rear rows of the wind farm is 5D, the spacing within the group is 3D, the air density is 1.25kg / m3, and the inflow wind turbulence intensity is 5%. The wind turbine models are all NREL 5MW units. The flow field size of the wind farm is 1668.8mx 1158.4m, the flow field grid size is 10m, and the simulation time step is 1s.

[0138] The designed dynamic WFSim platform is used to evaluate the effectiveness and advantages of the proposed wind farm wake control method based on deep reinforcement learning. The reinforcement learning parameters are set as follows: ξ=0.99, T=512, ∈=0.1, λ=0.95, and the mini-batch dataset size is 32. The actor and critic networks are 2-layer GCNs, with 4 input features, 2 output features, and 64 neurons in the Relu hidden layer.

[0139] Figure 7 The agent training process of optimizing the traditional DDPG and the GCN-DDPG combination of this embodiment is given. Figure 7 It can be seen that in the initial stage, since the network weights of the two control strategies DDPG and GCN-DDPG are obtained by random initialization, the cumulative rewards obtained by the agent are low, and oscillate due to the unknown knowledge of the training environment (i.e., the wind farm). As the number of training times increases, the performance of the control strategy is significantly improved, and the reward values ​​converge to their respective stable values, which are called convergence rewards. Finally, both agents obtain the optimal strategy. Compared with the traditional DDPG, the reward value obtained by GCN-DDPG is larger, indicating that the wind farm can generate more electricity under the control strategy of this embodiment.

[0140] In order to more specifically represent the control strategy learned by the agent, Figure 8 The inflow wind speed is plotted as u x =8m / s,u y =0m / s, flow field diagram of the wind farm under the three control strategies. Table 1 gives the comparison of power generation indicators of the wind farm under the three control strategies under this working condition.

[0141] Table 1 Comparison of power generation indicators under different control strategies

[0142]

[0143] Combination Figure 8As shown in Table 1, both DDPG and GCN-DDPG control strategies can effectively reduce the wake effect between upstream and downstream wind turbines by redirecting the wake. Taking wind turbines WT1, WT4 and WT7 as examples, after adopting the GCN-DDPG control strategy, WT1 actively yaws to transfer its wake away from the wind turbine array and reduces its axial induction factor to weaken the wake. Although this control operation reduces its power generation from 2.168MW to 2.113MW, the power generation of WT2 is significantly increased due to the control coordination of WT1. At the same time, WT2 accepts the yaw command assigned by the controller to reduce its wake effect on WT3. Because WT3's wake will not affect other turbines, it minimizes yaw in a greedy manner and collects all available wind energy as much as possible. Compared with the greedy strategy and traditional DDPG, under the GCN-DDPG control strategy, the total power generation of the wind farm is effectively increased by 2.48% and 2.17%.

[0144] In order to verify that the wake control strategy proposed in this embodiment can be applied to wind farms of different scales, the control strategy trained based on GCN-DDPG is applied to a 4x3 wind farm, and the inflow wind speed remains u x =8m / s,u y =0m / s. Since the traditional DDPG control strategy cannot be directly applied to a 4x3 scale wind farm, the wind farm control effects of the GCN-DDPG control strategy and the greedy strategy are compared here. Fig. 9 The flow field diagrams under the two control strategies are given. Compared with the greedy strategy, the control strategy proposed in this embodiment increases the power generation by 1.87%.

[0145] In summary, the method of this embodiment firstly dynamically considers the real-time changes of key parameters such as wind speed, wind direction, wind turbine yaw angle, and axis induction factor in the process of formulating the control strategy, and establishes a dynamic wake optimization model based on deep reinforcement learning to effectively deal with the real-time update problem of wind conditions and wind turbine status. In each training cycle, the model uses the latest wind farm operating status information (such as wind turbine status and environmental characteristics) and combines the wake effect that has not been completely eliminated in the previous cycle to generate a new control strategy for the yaw angle and axis induction factor, thereby achieving real-time dynamic adjustment, improving the optimization effect and ensuring the safe operation of the wind turbine. Secondly, in terms of optimization objectives, the optimization objective function is to maximize the overall power generation of the wind farm. By combining the characteristics of GCN, the wind farm is modeled as a dynamic graph network, the wake interaction between turbines is accurately modeled, and the spatial correlation between wind turbines is captured; at the same time, the exploration and optimization capabilities of deep reinforcement learning are used to generate the optimal control strategy in real time in a dynamic environment, and coordinate the operation of upstream and downstream turbines to reduce wake losses.

[0146] This method has the following characteristics:

[0147] (1) Enhanced generalization capability: The graph neural network is used to model the wind farm structure, so that the algorithm can directly migrate the existing model when the scale or layout of the wind farm changes, without the need for retraining, significantly reducing the optimization cost.

[0148] (2) Timeliness and dynamic adaptability: By utilizing the online learning characteristics of deep reinforcement learning, the model can adjust the control strategy according to the real-time wind conditions and demonstrate strong adaptability under non-steady-state wind conditions.

[0149] (3) Efficient wake control: By comprehensively utilizing wind farm operation data and graph structure information, the model can accurately adjust the operating parameters of upstream turbines, reduce the wake losses of downstream turbines, and maximize the overall power generation efficiency of the wind farm.

[0150] Example 2

[0151] Based on Example 1, this embodiment provides an electronic device, including: one or more processors and a memory, wherein the memory stores one or more programs, and the one or more programs include instructions for executing the offshore wind farm wake optimization control method based on deep reinforcement learning as described in Example 1.

[0152] like Fig.10 As described above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and of course may also include hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above method. Of course, in addition to the software implementation, the present invention does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0153] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for optimizing wake control of offshore wind farms based on deep reinforcement learning, characterized in that: The steps include: Construct a directed graph, and for each node in the directed graph, aggregate the input features of the surrounding neighboring nodes through graph neural convolution to update the input features of the current node, wherein the directed graph uses each offshore wind turbine in the offshore wind farm as a node, and the upstream and downstream relationships between wind turbines as edges; The input features of each node are used as the input of the trained Actor-Critic network to obtain the input feature action value and realize the optimization control of the offshore wind farm wake. The process of obtaining the trained Actor-Critic network includes: Taking the input features, inflow wind speed and inflow direction of each node as the state, the input feature action value as the action, and maximizing the total power generation of the offshore wind farm as the goal, the deep reinforcement learning problem is modeled based on the graph convolution Actor-Critic network, and the Actor-Critic network is trained through experience replay.

2. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 1 is characterized in that: The process of updating the input features of the current node by aggregating the input features of surrounding neighboring nodes through graph neural convolution includes the following steps: Calculating an adjacency matrix and a degree matrix of the directed graph; Calculating a directed edge weight matrix representing the directed graph; Based on the adjacency matrix, degree matrix and directed edge weight matrix, the input features of the surrounding neighbor nodes are aggregated through graph neural convolution to update the input features of the current node.

3. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 2 is characterized in that: The directed edge weight matrix is ​​calculated using the following formula: in, Indicates that γ max and γ min Offshore wind turbine WT i Maximum and minimum allowed yaw angles, w γ (γ i ) is the yaw angle γ i A non-negative function of δU H (γ i ) is the yaw angle γ i When , the wind speed loss rate under the greedy strategy.

4. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 2 is characterized in that: The updating of the input features of the current node by aggregating the input features of the surrounding neighbor nodes based on the adjacency matrix, degree matrix and directed edge weight matrix is ​​implemented by the following formula: Among them, A, D, and w are the adjacency matrix, degree matrix, and directed edge weight matrix of the directed graph, σ and θ are the nonlinear activation function and training parameters of the graph neural convolution, I is the unit matrix, X is the input feature of each node, and H is l , H (l+1) They respectively represent the directed graph input feature matrix and the updated directed graph input feature matrix, and I represents the unit matrix.

5. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 1 is characterized in that: The input characteristics include the yaw angle and axial induction factor of the offshore wind turbine.

6. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 5 is characterized in that: In the modeling process of the deep reinforcement learning problem, the action, state and control process trajectory are modeled as: Among them, s t 、a t , τ are action, state and control process trajectories respectively, S, A are action space and state space respectively, γ i,t 、a i,t are the yaw angle and axial induction factor of the offshore wind turbine at time t, Γ represents the time set, Δγ i,t , Δa i,t They represent the yaw angle action value and axial induction factor action value of the offshore wind turbine at time t, respectively. t ,d t are the inflow wind speed and inflow wind direction at time t respectively.

7. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 6 is characterized in that: The offshore wind turbine yaw angle, axial induction factor, offshore wind turbine yaw angle action value, and axial induction factor action value are all within the respective maximum and minimum value ranges.

8. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 1 is characterized in that: The experience replay training process of the graph convolutional Actor-Critic network includes the following steps: Initialize the discount factor, Actor network parameters, Critic network parameters, number of training rounds, and experience replay buffer; Generate input features for each offshore wind turbine using a pre-built state distribution, and get the state based on the inflow wind speed and inflow direction; Sampling the actions of each offshore wind turbine, calculating the corresponding reward and updating the state, constructing the sampling data based on the previous state, the current action, the current reward and the next state, and putting it into the experience playback buffer; Based on the size of the sampled data in the experience replay buffer, batches of sampled data are randomly selected, the Actor network parameters and the Critic network parameters are updated, and it is determined whether the number of training rounds reaches a maximum value, and if so, the training is terminated.

9. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 8, characterized in that: The update of the Actor network parameters and Critic network parameters is based on the following formula: Q′(s,w)←τQ(s,w)+(1-τ)Q′(s,w) μ′(s,θ)←τμ(s,θ)+(1-τ)μ′(s,θ) Where L is the loss function, K is the number of offshore wind turbines, and y k is the action value function Q(s k ,a k ) target value, s k 、a k are the action and state of the th offshore wind turbine respectively, μ(s,θ) and μ′(s,θ) are the strategy functions before and after the update respectively, s and θ are the state and Actor network parameters respectively, Q′(s,w) is the updated action value function, w is the Critic network parameter, τ is the smoothing factor and 0≤τ<<1.

10. The offshore wind farm wake optimization control method based on deep reinforcement learning according to claim 1, characterized in that: In the modeling process of the deep reinforcement learning problem described above, the goal is to maximize the total reward value of the control process trajectory: Among them, G t is the total reward value, ξ is the discount factor, r t+1 To perform action a t The reward obtained after i,t is the active power of wind turbine i at time t.

Images