A deep reinforcement learning control method for permanent magnet direct-drive wind turbines

Abstract

The present invention discloses a deep reinforcement learning control method for a permanent magnet direct-drive wind turbine, which belongs to the field of wind power. A deep reinforcement learning TD3 algorithm is used to design a speed tracking controller: an intelligent agent based on the TD3 algorithm is constructed according to the environmental state of the permanent magnet synchronous wind turbine, the environmental state includes the generator speed and its reference value, stator current, stator voltage, stator inductance, stator resistance, permanent magnet rotor flux, stator winding temperature, wind speed and its rate of change, a reward function is designed, the wind speed and generator training conditions are determined, the intelligent agent is trained, the trained intelligent agent model is transplanted into the main control chip of the machine-side converter, an actual speed tracking control system is established, and speed tracking control is realized. The present invention can effectively cope with the influence of wind speed fluctuations and changes in the parameters of the generator itself on the control of the wind turbine, and realize rapid, stable and safe control of the permanent magnet synchronous wind turbine.

Description

Technical Field

[0001] The present invention relates to a control method, in particular to a deep reinforcement learning control method for a permanent magnet direct-drive wind turbine generator set, and belongs to the technical field of wind power. Background Art

[0002] In recent years, wind power generation technology has made significant progress. Permanent magnet synchronous generator (PMSG) is widely used in wind power generation systems due to its simple structure, low cost, and no need for speed increase gearbox.

[0003] At present, PID control is still one of the important control methods for PMSG. However, under conditions such as time-varying wind speed and changes in generator parameters, since the control parameters of PID are fixed and cannot adapt to the changing conditions, its control is difficult to maintain good steady-state performance. Model predictive control (MPC) is also a widely used method in PMSG, which realizes tracking control by designing cost functions. However, changes in motor parameters will increase the tracking error of the MPC method, thereby affecting the control effect. In order to reduce the impact of parameter mismatch, the model-free predictive control method is a new idea, but it increases the difficulty of controller design. In short, it is difficult to achieve intelligent control of PMSG through the above control methods.

[0004] Deep reinforcement learning (DRL) is a powerful technology for solving perception problems of complex systems. It effectively combines the perception ability of deep learning and the decision-making ability of reinforcement learning. It can be controlled according to the input information. It is an artificial intelligence control method that is closer to human thinking and has attracted widespread attention in recent years. At present, there are literatures that design DRL agents for loads to adjust the parameters of disturbance observers. Some use current and voltage-based agents to control neutral point clamped (NPC) three-level converters. Some propose output voltage-based DRL agents to solve the voltage interference problem of single-input multi-output DC / DC converters; and some propose a multi-agent DRL solution to solve current stress and power balance problems. However, there is currently little research on the application of deep reinforcement learning control in permanent magnet direct-drive wind turbines. It is worth noting that the double-delay deep deterministic policy gradient (TD3) algorithm, one of the deep reinforcement learning algorithms, has significant advantages in processing continuous action space. This deep reinforcement learning control algorithm, which does not rely on mathematical models, is particularly suitable for wind turbines with unknown time-varying interference (such as volatility and uncertainty in wind speed and wind direction), changes in their own conditions after long-term operation (such as parameter changes caused by aging of generators and converters, etc.), and with nonlinear and strongly coupled characteristics.

[0005] To this end, the present invention proposes a deep reinforcement learning control scheme for a permanent magnet direct-drive wind turbine set, thereby ensuring that the PMSG has excellent steady-state performance under conditions of time-varying wind speed and generator parameter changes. Summary of the invention

[0006] The main purpose of the present invention is to: in view of the shortcomings of the above-mentioned prior art, the present invention provides a deep reinforcement learning control method for a permanent magnet direct-drive wind turbine set, which realizes stable and efficient operation of the generator under time-varying wind speed and changes in its own parameters by utilizing the powerful exploration and interaction capabilities of the DRL algorithm.

[0007] In order to achieve the above objectives, the permanent magnet direct-drive wind turbine set described in the present invention includes: a permanent magnet synchronous wind generator, a machine-side converter, a grid-side converter, etc.; one end of the machine-side converter is connected to the stator of the permanent magnet synchronous wind generator, and the other end is connected to the grid-side converter.

[0008] The present invention provides a deep reinforcement learning control method for a permanent magnet direct-drive wind turbine generator set, comprising the following steps:

[0009] Step 1, determining the environmental state of the permanent magnet synchronous wind turbine:

[0010] The environmental state includes the rotor rotation angular velocity ω(t) of the permanent magnet synchronous wind turbine and its reference value ω m,r , the d-axis component i of the stator current of the permanent magnet synchronous wind turbine d and the q-axis component i q , the d-axis component u of the stator voltage of the permanent magnet synchronous wind turbine d and the q-axis component u q The stator inductance L, stator winding resistance R, rotor permanent magnet flux linkage ψ of the permanent magnet synchronous wind turbine f , the temperature T of the stator winding of the permanent magnet synchronous wind turbine st , wind speed v w and wind speed change rate The environmental state is denoted as S t , then

[0011] Step 2, design the reward function:

[0012] The reward function R t Designed for:

[0013] R t =R fast + R smooth +R T (1)

[0014] In the formula, R fastis the speed tracking reward function, R smooth is the speed fluctuation suppression and distortion rate reward function, R T It is a temperature rise reward function. Its function is that after long-term operation, the generator parameters deteriorate and the capacity needs to be reduced to prevent the generator from being overloaded and burned.

[0015] R fast Defined as:

[0016] R fast =R1|ω m,r -ω(t)|,|ω m,r -ω(t)|≥0.2rad / s (2)

[0017] Where R1 is the speed tracking weight coefficient, which is used to quickly reduce the speed tracking error;

[0018] R smooth Defined as:

[0019]

[0020] In the formula, R2 is the distortion rate quick weight coefficient, which is used to minimize the distortion rate, i d * 、i q * are respectively the d-axis component i of the stator current of the permanent magnet synchronous wind turbine d and the q-axis component i q The reference value, A d1 is the reference value of the current difference; R3 is the stability weight coefficient, which is used to reduce the speed fluctuation of the generator; ω(tl) is the speed of the permanent magnet synchronous wind turbine at the first moment, L is the upper limit of the time for suppressing fluctuations, and l and L are both positive integers;

[0021] R T Defined as:

[0022] R T =R4(T al -T st ) / A d2 (4)

[0023] Where R4 is the stator winding temperature weight coefficient, T st is the temperature of the stator winding of the permanent magnet synchronous wind turbine, T al is the allowable temperature of the stator winding of the permanent magnet synchronous wind turbine, A d 2 is the reference value of the temperature difference.

[0024] Step 3: construct an intelligent agent based on the TD3 algorithm according to the environmental state:

[0025] The agent is composed of a strategy network and a strategy evaluation network; the strategy network is composed of an Actor network and an ActorTarget network, respectively through the corresponding network strategy π(S t |θ) and π ta (S t |θ ta ) realizes the action output A of the agent t (S t |θ); The strategy evaluation network consists of a Critic network and a Critic Target network, which are respectively and Evaluate and improve the policy used to achieve the action output, where θ, θ ta , They represent the weight coefficients of the Actor network, Actor Target network, Critic network, and CriticTarget network respectively.

[0026] Step 4: training the agent:

[0027] A1. Determine the training environment, including wind speed training conditions and generator parameter training conditions, where:

[0028] Wind speed training condition: insert random wind speed noise δv every 1s into the actual wind speed curve w , the training wind speed v' is calculated according to formula (5): w , and at the beginning of a new training round, by δv w Reset v' according to formula (5) w ;

[0029]

[0030] In the formula, δv w Subject to average distribution U; v wn is the rated wind speed; w d is the deviation width, w d ≥1;

[0031] Generator parameter training conditions: In order to ensure that the agent can still have excellent performance when the generator parameters change, at the beginning of each training round, the parameters G' of the permanent magnet synchronous wind turbine are trained according to formula (6) s , where s = 1, 2, 3, respectively representing the stator inductance L, stator winding resistance R and rotor permanent magnet flux ψ of the permanent magnet synchronous wind turbine f , through G s Add random parameter variation additional term δG s , calculated according to formula (6) to obtain G's ; Also at the beginning of a new training round, by δG s Update G' according to formula (6) s , and by the maximum additional factor g max and the minimum additional factor g min Determine G s The upper and lower limits of

[0032]

[0033] In the formula, δG s Subject to average distribution U; w g is the deviation width, w g ≥1.

[0034] A2. Create an interface for the agent to interact with the environment information, initialize the network weight parameters of the agent, and set the relevant parameters for the agent training, as follows:

[0035] Initialize the weight parameters of the Actor main network and the Critic main network And the corresponding target network Actor Target network, target network Critic Target network weight parameters

[0036] Set the learning rate α of the Actor main network;

[0037] Set the discount factor γ for deep reinforcement learning;

[0038] Set the maximum number of training rounds N, the maximum number of steps in each training round T, the update frequency I of the target network Actor Target network and Critic Target network, and the weight of the Critic network Update frequency I p ,I ta ;

[0039] Load actual wind speed v w , wind speed deviation noise δv w , the parameter G of the permanent magnet synchronous wind turbine s and its random parameter variation additional term δG s ;

[0040] Set up and initialize the experience database;

[0041] A3. At the beginning of each training round, first according to the actual wind speed v w , wind speed deviation noise δv w , the parameter G of the permanent magnet synchronous wind turbine s and its random parameter variation additional term δGs Determine the wind speed training condition v' for this training round according to formula (5) and formula (6) respectively wk and generator parameter training condition G' sk ,k=1,2,…,N;v' wk is v' of the kth training round w ; G' sk is the G of the kth training round s .

[0042] A4. Based on the current state S t , in order to prevent the network strategy π(S t |θ) falls into a local optimum, adding action exploration noise N to the output of the Actor main network t , get action A t (S t |θ), providing an opportunity to explore “better”, namely:

[0043] A t (S t |θ)=π(S t |θ)+N t (7)

[0044] In the formula, π(S t |θ) the output of the Actor network; N t The expected value is 0 and the variance is σ 2 The normal distribution of t satisfy:

[0045]

[0046] A5. After each action is executed, the next state of the environment S is obtained t+1 , and the reward function R is calculated according to formula (1) t , and the environmental state S t 、Action A t (S t |θ), reward function value R t , the environmental state S at the next moment t+1 storing as an experience data set in the experience memory;

[0047] A6. Randomly and uniformly sample M small batches of samples from the experience database, that is, M experience data groups (S ti ,A ti ,R ti ,S ti+1 ), where i = 1, 2, 3, …, M;

[0048] A7. Update the weight parameters of the Critic main network in a way that minimizes the error Right now

[0049]

[0050] Where Y ti is the expected target value function, defined as:

[0051]

[0052] In the formula, ε represents the target smooth noise, which is the truncated normal distribution noise, that is:

[0053]

[0054] In the formula, σ clip is the standard deviation and c is the cutoff factor.

[0055] A8. Update the weight parameters θ, θ of the Actor network, Actor Target network, and Critic Target network in a delayed update manner ta and Specifically, the weights of the Critic network Every update I p The weight parameter θ of the Actor network is updated once. The weight of the Critic network Every update I ta Second, the weight parameters θ of the Actor Target network and the Critic Target network ta and Update once; if the number of training steps is the update frequency I of the Critic network p If it is a multiple of , then go to step A9, otherwise go to step A11; p ,I ta is the update frequency, which is a positive integer, and I ta >I p .

[0056] A9. Update the weight parameter θ of the Actor network by performing gradient ascent on the expected value J, that is,

[0057]

[0058] In the formula, the expression of J is:

[0059]

[0060] It means that J is derived with respect to θ:

[0061]

[0062] In the formula, represents π(S ti |θ) with respect to θ.

[0063] A10. If the number of training steps is the update frequency I of the Critic network ta If it is a multiple of , the weight parameter θ of the Actor Target network is adjusted by exponential smoothing according to formula (15). ta and the weight parameters of the Critic Target network To update, that is:

[0064]

[0065] Where τ is the update factor, τ≤1.

[0066] A11. If the number of training steps does not reach the maximum number of steps T for each training round, return to step A4; otherwise, the current training round ends, and the agent model obtained in this training round is saved, and the average reward function value R of this training round is calculated and saved. e ;

[0067] A12. If the average reward function value R for n consecutive training rounds e The absolute value of R set , where R set is a positive real number close to 0, the entire training process ends; otherwise, go to step A13;

[0068] A13. Determine whether the maximum number of training rounds N has been reached. If reached, end the entire training process and start training again; if not reached, return to step A3 and start the next training round.

[0069] Step 5: transplant the intelligent agent model trained in step 4 into the main control chip of the machine-side converter, establish an actual speed control system based on the main control chip of the machine-side converter, realize speed control, and output the stator current q-axis component i of the permanent magnet synchronous wind turbine q The reference value i q * .

[0070] Step 6, adopting the zero d-axis current control strategy and the current feedforward decoupling control strategy, designing the inner loop current tracking controller of the machine-side converter, and realizing the current tracking control of the permanent magnet synchronous wind turbine. The specific process is as follows:

[0071] 61) Output A obtained in step 5 t (S t|θ), i.e., the q-axis component i of the stator current of the permanent magnet synchronous wind turbine q The reference value i q * , and compare it with its actual value i q The output of the first PI controller is decoupled from the q-axis feedforward compensation term -Ri q -ω e L d i d +ω e ψ f Add together to obtain the q-axis component u of the stator voltage of the permanent magnet synchronous wind turbine q The reference value u q * ; Among them, ω e is the rotor electrical angular velocity of the permanent magnet synchronous wind turbine, L d is the d-axis component of the stator inductance;

[0072] 62) Let the d-axis component i of the stator current of the permanent magnet synchronous wind generator be d The reference value i d * =0, and compare it with its actual value i d The second PI controller is used as the differential input, and the second PI controller is decoupled from the d-axis feedforward compensation term -Ri d +ω e L q i q Add together to obtain the d-axis component u of the stator voltage of the permanent magnet synchronous wind turbine d The reference value u d * Among them, L q is the q-axis component of the stator inductance;

[0073] Step 7: The stator voltage reference value u of the permanent magnet synchronous wind turbine generator obtained in step 6 is d * 、u q * After dq / αβ coordinate transformation, we get u α * 、u β * ; Then, u α * and u β * After modulation by the SVPWM module, a driving signal is generated to drive the power switch tube of the machine-side converter to control the operation of the permanent magnet synchronous wind turbine.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] 1. The present invention adopts a deep reinforcement learning control algorithm. Its intelligent agent adjusts the output power automatically according to the wind speed and its rate of change, and fully considers the changes in the generator's own parameters, and monitors the stator winding temperature to prevent the generator's stator winding from being overloaded. It has strong adaptability and robustness to changes in wind speed and changes in the generator's own parameters, and can effectively cope with the impact of wind speed fluctuations and changes in the generator's own parameters on the control of the wind turbine, thereby achieving rapid, safe and stable control of the permanent magnet synchronous wind turbine.

[0076] 2. The present invention can realize optimal control of multi-objective tasks, suppress the stator current harmonic distortion of the wind turbine, reduce the power consumption of the wind turbine and its side converter, and improve the power generation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 This is a schematic diagram of the topological structure of the permanent magnet direct-drive wind turbine unit of the present invention.

[0078] Figure 2 Schematic diagram of deep reinforcement learning control principle.

[0079] Figure 3 Schematic diagram of the training process of the intelligent agent in the deep reinforcement learning TD3 control strategy.

[0080] Figure 4 This is the deep reinforcement learning TD3 control training flow chart of the present invention.

[0081] Figure 5 Training wind speed map for the present invention.

[0082] Figure 6 This is a control structure block diagram of the deep reinforcement learning TD3 control of the present invention.

[0083] Figure 7 This is the control structure block diagram of a permanent magnet synchronous wind turbine generator using PI control strategy.

[0084] Figure 8 is the speed and stator current i under the standard parameters of PMSG, the present invention and the PI control strategy d 、i q Tracking error comparison simulation chart.

[0085] Fig. 9 The speed and stator current i under the PMSG parameter change of the present invention and the PI control strategy are d 、i q Tracking error comparison simulation chart.

[0086] Fig.10The figure shows the stator current waveform and distortion rate of the present invention at different wind speeds under PMSG standard parameters.

[0087] Fig.11 The figure shows the stator current waveform and distortion rate at different wind speeds when PI control is used under PMSG standard parameters.

[0088] Fig.12 The figure shows the stator current waveform and distortion rate of the present invention at different wind speeds when the parameters of the PMSG change.

[0089] Fig.13 The figure shows the stator current waveform and distortion rate at different wind speeds when PI control is used when the PMSG parameters change.

[0090] Fig.14 The figure is a comparison chart of the power generation efficiency of PMSG under the PMSG standard parameters and the PI control strategy.

[0091] Fig.15 The figure is a comparison diagram of the power generation efficiency of PMSG under the present invention and the PI control strategy when the PMSG parameters change.

[0092] Among them, 1-permanent magnet synchronous wind turbine, 2-machine-side converter, 3-grid-side converter; 4-intelligent agent; 5-encoder; 6-speed controller based on deep reinforcement learning TD3; 7-inner loop current tracking controller; 8-first PI controller; 9-second PI controller; 10-outer loop speed tracking controller based on PI control strategy. DETAILED DESCRIPTION

[0093] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0094] like Figure 1 As shown, the permanent magnet direct-drive wind turbine unit of the present invention includes a permanent magnet synchronous wind generator 1, a machine-side converter 2, and a grid-side converter 3. One end of the machine-side converter 2 is connected to the stator of the permanent magnet synchronous wind generator 1, and the other end is connected to the grid-side converter 3; the grid-side converter 3 is connected to the AC power grid through an industrial frequency transformer.

[0095] The part that can reflect the variable information of the permanent magnet synchronous wind turbine 1 and its machine-side converter 2 is defined as the environment; the part that can execute actions and control the operation of the permanent magnet synchronous wind turbine 1 and the machine-side converter 2 is defined as the intelligent agent 4. Figure 2 The figure shows the schematic diagram of the control principle of deep reinforcement learning TD3. The main process is: the agent 4 controls the environment state (State) S t 、Reward function (Reward) R t Output Action (Action) At , Action A t Then it acts on the environment to achieve the optimal environment, thereby effectively controlling the operation of permanent magnet direct-drive wind turbines.

[0096] The present invention provides a deep reinforcement learning control method for a permanent magnet direct-drive wind turbine generator set, comprising the following steps:

[0097] Step 1, determining the environmental state of the permanent magnet synchronous wind turbine 1:

[0098] like Figure 6 As shown, the environmental state includes the rotor rotation angular velocity ω(t) of the permanent magnet synchronous wind turbine 1 and its reference value ω m,r , the d-axis component of the stator current i d and the q-axis component i q , the d-axis component of the stator voltage u d and the q-axis component u q , stator inductance L, stator winding resistance R, rotor permanent magnet flux ψ f , the temperature of the stator winding T st , wind speed v w and wind speed change rate Where ω(t) = ω e / p,ω e is the rotor electrical angular velocity of the permanent magnet synchronous wind generator 1 , measured by the encoder 5 , and p is the pole pair number of the permanent magnet synchronous wind generator 1 .

[0099] Let the environment state be S t , then:

[0100] Step 2, design the reward function:

[0101] Reward function R t Designed for:

[0102] R t =R fast + R smooth +R T (1)

[0103] In the formula, R fast is the speed tracking reward function, whose main goal is to quickly reduce the speed tracking error in a short time, improve the power generation efficiency of PMSG, and achieve efficient control of PMSG. It plays an important role in sudden changes in wind speed, especially when the wind speed changes drastically; R smooth is the speed fluctuation suppression and distortion rate reward function, whose main goal is to improve the steady-state performance of PMSG, reduce the PMSG output current distortion rate, and thus reduce the power generation loss. fast When the speed error cannot be further reduced, it plays a role in stabilizing the actual speed; RT is the temperature rise reward function, ensuring that the parameters of the permanent magnet synchronous wind turbine 1 (such as: L, R, ψ f ) changes, the temperature T of the stator winding st Do not exceed the permissible temperature.

[0104] R fast Defined as:

[0105] R fast =R1|ω m,r -ω(t)|,|ω m,r -ω(t)|≥0.2rad / s (2)

[0106] Where R1 is the speed tracking weight coefficient, which is used to quickly reduce the speed tracking error.

[0107] R smooth Defined as:

[0108]

[0109] In the formula, R2 is the distortion rate quick weight coefficient, which is used to minimize the distortion rate, i d * 、i q * are the d-axis components i and d-axis components i of the stator current of the permanent magnet synchronous wind turbine 1. d and the q-axis component i q The reference value, A d1 is the reference value of the current difference; R3 is the stability weight coefficient, which is used to reduce the speed fluctuation of the generator; ω(tl) is the speed of the permanent magnet synchronous wind turbine 1 at the first moment before, L is the upper limit of the time for suppressing fluctuations, l and L are both positive integers, and L can be set to 10;

[0110] R T Defined as:

[0111] R T =R4(T al -T st ) / A d2 (4)

[0112] Where R4 is the stator winding temperature weight coefficient, T st is the temperature of the stator winding of the permanent magnet synchronous wind turbine 1, T al is the allowable temperature of the stator winding of the permanent magnet synchronous wind turbine 1 (as specified in relevant standards or provided by the manufacturer), A d 2 is the reference value of the temperature difference.

[0113] Step 3: According to the above environmental state, construct the agent 4 based on the TD3 algorithm:

[0114] like Figure 3 As shown, the agent 4 is composed of a strategy network and a strategy evaluation network; the strategy network is composed of an Actor network and an Actor Target network, which are respectively implemented through the corresponding network strategy π(S t |θ) and π ta (S t |θ ta ) realizes the action output A of agent 4 t (S t |θ); The strategy evaluation network consists of a Critic network and a Critic Target network, which are respectively and Evaluate and improve the policy used to achieve the action output, where θ, θ ta , They represent the weight coefficients of the Actor network, Actor Target network, Critic network, and CriticTarget network respectively.

[0115] Step 4, such as Figure 3 , Figure 4 As shown, the deep reinforcement learning TD3 algorithm is used to train agent 4:

[0116] The training process of the intelligent agent is divided into two parts: action exploration and iterative optimization, which rely on each other. In order to avoid the strategy from falling into the local optimum, the neural network is iteratively updated during the process of continuous action exploration, and finally the optimal controller is trained to achieve intelligent and efficient control of PMSG.

[0117] A1. Determine the training environment, including wind speed training conditions and generator parameter training conditions, where:

[0118] Wind speed training conditions:

[0119] Figure 5 The actual wind speed curve v measured every 5 seconds at a certain wind farm is shown in Figure 2. w Used for wind speed training. In order to enrich the training environment based on the real environment and obtain the optimal agent that can adapt to various wind conditions, random wind speed noise δv is inserted every 1s in the actual wind speed curve. w , the training wind speed v' is calculated according to formula (5): w (See Figure 5 ), and at the beginning of a new training round, by δv w Reset v' according to formula (5) w ; Figure 5 Two random wind speed curves v' are also shown w1 、v'w2 .

[0120]

[0121] In the formula, δv w Subject to average distribution U; v wn is the rated wind speed; w d is the deviation width, w d ≥1;

[0122] Generator parameter training conditions:

[0123] In order to ensure that the agent 4 can still have excellent performance when the generator parameters change, at the beginning of each training round, the parameter G' of the permanent magnet synchronous wind turbine is trained according to formula (6): s , where s = 1, 2, 3, respectively representing the stator inductance L, stator winding resistance R, and rotor permanent magnet flux ψ of the permanent magnet synchronous wind turbine. f , through G s Add random parameter variation additional term δG s , calculated according to formula (6) to obtain G' s ; Also at the beginning of a new training round, by δG s Update G' according to formula (6) s . Through the maximum additional factor g max and the minimum additional factor g min Determine G' s The upper and lower limits of max =1.2, g min =0.8, indicating the training process G' s The range is G s Between 80%-120%.

[0124]

[0125] In the formula, δG s Subject to average distribution U; w g is the deviation width, w g ≥1.

[0126] A2. Create an interface for Agent 4 to interact with environmental information, initialize the network weight parameters of Agent 4, and set the relevant parameters for Agent 4 training, as follows:

[0127] Initialize the weight parameters of the Actor main network and the Critic main network And the corresponding target network Actor Target network, target network Critic Target network weight parameters

[0128] Set the learning rate α of the Actor main network;

[0129] Set the discount factor γ for deep reinforcement learning;

[0130] Set the maximum number of training rounds N, the maximum number of steps in each training round T, the update frequency I of the target network Actor Target network and Critic Target network, and the weight of the Critic network Update frequency I p ,I ta ;

[0131] Load actual wind speed v w , wind speed deviation noise δv w , Parameters G of permanent magnet synchronous wind turbine 1 s and its variation additional term δG s ;

[0132] Set up and initialize the experience database.

[0133] A3. At the beginning of each training round, first according to the actual wind speed v w , wind speed deviation noise δv w , Parameters G of permanent magnet synchronous wind turbine 1 s and its variation additional term δG s Determine the wind speed training condition v' for this round according to formula (5) and formula (6) respectively wk and generator parameter training condition G' sk ,k=1,2,…,N;v' wk is v' of the kth training round w ; G' sk is G' of the kth training round s .

[0134] A4. Based on the current state S t , in order to prevent the network strategy π(S t |θ) falls into a local optimum, adding action exploration noise N to the output of the Actor main network t , get action A t (S t |θ), providing an opportunity to explore “better”, namely:

[0135] A t (S t |θ)=π(S t |θ)+N t (7)

[0136] In the formula, π(S t |θ) the output of the Actor network; Nt The expected value is 0 and the variance is σ 2 The normal distribution of t satisfy:

[0137]

[0138] A5. After each action is executed, the next state of the environment S is obtained t+1 , and the reward function R is calculated according to formula (1) t , then agent 4 observes the state S at the next moment t+1 , and the environmental state S t 、Action A t (S t |θ), reward function value R t , the environmental state S at the next moment t+1 storing as an experience data set in the experience memory;

[0139] A6. Randomly and uniformly sample M small batches of samples from the experience database, that is, M experience data groups (S ti ,A ti ,R ti ,S ti+1 ), where i = 1, 2, 3, …, M;

[0140] A7. Update the weight parameters of the Critic main network in a way that minimizes the error Right now

[0141]

[0142] Where Y ti is the expected target value function, defined as:

[0143]

[0144] In the formula, ε represents the target smooth noise, which is the truncated normal distribution noise, that is:

[0145]

[0146] In the formula, σ clip is the standard deviation and c is the cutoff factor.

[0147] A8. Update the weight parameters θ, θ of the Actor network, Actor Target network, and Critic Target network in a delayed update manner ta and Specifically, the weights of the Critic network Every update I pThe weight parameter θ of the Actor network is updated once. The weight of the Critic network Every update I ta Second, the weight parameters θ of the Actor Target network and the Critic Target network ta and Update once; if the number of training steps is the update frequency of the Critic network I p If it is a multiple of , then go to step A9, otherwise go to step A11; p ,I ta is the update frequency, which is a positive integer, and I ta >I p ;

[0148] A9. Update the weight parameter θ of the Actor network by performing gradient ascent on the expected value J, that is,

[0149]

[0150] In the formula, the expression of J is:

[0151]

[0152] It means that J is derived with respect to θ:

[0153]

[0154] In the formula, represents π(S ti |θ) with respect to θ.

[0155] A10. If the number of training steps is the update frequency of the Critic network I ta If it is a multiple of , the weight parameter θ of the Actor Target network is calculated by exponential smoothing according to formula (15). ta and the weight parameters of the Critic Target network To update, that is:

[0156]

[0157] Where, τ is the update factor, τ≤1;

[0158] A11. If the number of training steps does not reach the maximum number of steps T for each training round, return to step A4; otherwise, the current training round ends, and the agent model obtained in this training round is saved, and the average reward function value R of this training round is calculated and saved. e ;

[0159] A12. If the average reward function value R of n consecutive training rounds (for example, n = 10) e The absolute value of R set , where R set is a value close to 0 (for example, let R set =0.01), the whole training process ends; otherwise, it goes to step A13;

[0160] A13. Determine whether the maximum number of training rounds N has been reached. If so, end the entire training process and start training again. If not, return to step A3 and start the next training round.

[0161] Step 5: transplant the intelligent agent model trained in step 4 into the main control chip of the machine-side converter, establish an actual speed control system based on the main control chip of the machine-side converter, realize speed control, and output the stator current q-axis component i of the permanent magnet synchronous wind turbine 1 q The reference value i q * .

[0162] Step 6, such as Figure 6 As shown, the zero d-axis current control strategy and the current feedforward decoupling control strategy are adopted to design the inner loop current tracking controller 7 of the machine-side converter 2 to realize the current tracking control of the permanent magnet synchronous wind turbine 1. The specific process is as follows:

[0163] 61) Output A obtained in step 5 t (S t |θ), i.e., the q-axis component i of the stator current of the permanent magnet synchronous wind turbine 1 q The reference value i q * , and compare it with its actual value i q The difference is input to the first PI controller 8, and the output of the first PI controller 8 is decoupled from the q-axis feedforward compensation term -Ri q -ω e L d i d +ω e ψ f Add together to obtain the q-axis component u of the stator voltage of the permanent magnet synchronous wind turbine 1 q The reference value u q * ; Wherein, R is the stator winding resistance of the permanent magnet synchronous wind turbine 1, ω e is the rotor electrical angular velocity of the permanent magnet synchronous wind turbine 1, measured by the encoder 5, L d is the d-axis component of the stator inductance;

[0164] 62) Let the d-axis component i of the stator current of the permanent magnet synchronous wind turbine 1 be dThe reference value i d * =0, and compare it with its actual value i d The second PI controller 9 is input as a differential input, and the second PI controller 9 is decoupled from the d-axis feedforward compensation term -Ri d +ω e L q i q Add together to obtain the d-axis component u of the stator voltage of the permanent magnet synchronous wind turbine 1 d The reference value u d * Among them, L q is the q-axis component of the stator inductance;

[0165] Step 7: The stator voltage reference value u of the permanent magnet synchronous wind turbine 1 obtained in step 6 is d * 、u q * After dq / αβ coordinate transformation, we get u α * 、u β * ; Then, u α * and u β * After being modulated by the SVPWM module, a driving signal is generated to drive the power switch tube of the machine-side converter to control the permanent magnet synchronous wind turbine 1 to work.

[0166] The present invention is further described below with a preferred embodiment.

[0167] In order to verify the effectiveness of the deep reinforcement learning TD3 control strategy of the present invention, a simulation experiment is carried out on the permanent magnet synchronous wind turbine 1 and its machine-side converter 2 using the control strategy of the present invention, and the simulation experiment results are compared and analyzed with those of the traditional PI control strategy.

[0168] like Figure 7 As shown, the main idea of ​​the control strategy of the permanent magnet synchronous wind turbine based on PI control is: first, the outer loop speed tracking controller 10 based on PI control sets the reference speed ω of the permanent magnet synchronous wind turbine 1 * The difference between the actual speed ω(t) and the actual speed ω(t) is sent to the PI controller, and its output is used as the current reference value i of the inner loop q-axis current tracking control q * Then, this q * Its actual value i q After the difference is made, it is sent to the first PI controller 8, and the output of the controller 8 is u q * At the same time, the d-axis current reference value id * = 0 and its actual value, and then the difference is sent to the second PI controller 9. The output of the controller 9 is u d * Finally, u d * 、u q * After dq / αβ coordinate transformation, we get u α * 、u β * ,u α * 、u β * After modulation by the SVPWM module, a driving signal is generated to drive the power switch tube of the machine-side converter 2, thereby controlling the permanent magnet synchronous wind turbine 1 to work.

[0169] The specific simulation parameters are shown in Table 1, Table 2, and Table 3.

[0170] Table 1 TD3 training related parameters

[0171] parameter Numeric The learning rate α of the Actor network <![CDATA[1×10 -3 ]]> Discount factor γ 0.995 The maximum number of training rounds M 1000 The maximum number of steps N in each training round 1200 The size of the experience memory bank B <![CDATA[2×10 5 ]]> The size of the empirical data set C <![CDATA[2 9 =512]]> <![CDATA[Policy update frequency I p > 3 <![CDATA[Target update frequency I ta > 12 Update factor τ 0.005

[0172] Table 2 Wind turbine related parameters

[0173] parameter Numeric <![CDATA[Rated power P N > 3kW <![CDATA[Cut-in wind speed v wi > 3m / s <![CDATA[Rated wind speed v wn > 9m / s <![CDATA[Optimal tip speed ratio λ opt > 8.0

[0174] Table 3 Permanent magnet synchronous wind turbine generator set related parameters

[0175] parameter Numeric <![CDATA[Rated power P N > 3kW <![CDATA[Rated speed n N > 360r / min Pole pair number p 8 Stator inductance L 8.3mH Stator resistance R 0.53Ω <![CDATA[Rotor permanent magnet flux linkage ψ f > 0.34Wb DC side voltage 650V

[0176] Figure 8 (a) and (b) are respectively the speed ω(t) and the stator side current i of the permanent magnet synchronous wind turbine 1 under the standard parameters of the permanent magnet synchronous wind turbine 1 (see Table 3) under the deep reinforcement learning TD3 control and PI control of the present invention. q 、i d The tracking error of Figure 8 It can be seen that the maximum tracking errors of the three are 0.01 rad / s, 0.19 A, and 0.18 A respectively when the control method of the present invention is used, while they are 0.19 rad / s, 0.50 A, and 0.94 A respectively when the PI control method is used. It can be seen that the control effect of the present invention is better.

[0177] Fig. 9 (a) and (b) are respectively the case where the parameters of the permanent magnet synchronous wind turbine generator 1 are changed (R, L increase by 15%, ψ f Reduce by 10%, that is, let R = 0.61Ω, L = 9.55mH, ​​ψ f=0.306Wb)), the speed ω(t) of the permanent magnet synchronous generator 1 and the stator side current i under the present invention and PI control q 、i d By Fig. 9 It can be seen that the maximum tracking errors of the three are 0.016 rad / s, 0.27 A, and 0.29 A respectively when the control method of the present invention is used, and are 0.28 rad / s, 0.72 A, and 1.24 A respectively when the PI control method is used.

[0178] Depend on Figure 8 , Fig. 9 It can be seen that the control effect of the present invention is better and it has a stronger ability to cope with changes in generator parameters.

[0179] Fig.10 (a) and (b) are the stator current distortion rates of the present invention at wind speeds of 6 m / s and 8 m / s respectively under standard generator parameters; Fig.11 (a) and (b) are the stator current distortion rates at wind speeds of 6m / s and 8m / s respectively when PI control is used under standard generator parameters. Fig.10 , 11 It can be seen that when the wind speed is 6m / s and 8m / s, the current distortion rates using the TD3 control method of the present invention are 0.37% and 0.33% respectively, while the current distortion rates using the PI control method are 3.25% and 2.95% respectively. It can be seen that the control effect of the present invention is better.

[0180] Fig.12 (a) and (b) are respectively the case where the parameters of the permanent magnet synchronous wind turbine generator 1 are changed (relative to the standard parameters, R, L increase by 15%, ψ f Reduce by 10%, that is, let R = 0.61Ω, L = 9.55mH, ​​ψ f =0.306Wb)), the stator current distortion rate of the present invention when the wind speed is 6m / s and 8m / s; Fig.13 (a) and (b) are respectively the case where the parameters of the permanent magnet synchronous wind turbine generator 1 are changed (relative to the standard parameters, R, L increase by 15%, ψ f Reduce by 10%, that is, let R = 0.61Ω, L = 9.55mH, ​​ψ f =0.306Wb)), when PI control is used, the stator current distortion rate at wind speeds of 6m / s and 8m / s. Fig.12 , 13 It can be seen that when the wind speed is 6m / s and 8m / s, the current distortion rates using the control method of the present invention are 0.45% and 0.36% respectively, while the current distortion rates using the PI control method are 4.35% and 4.22% respectively. It can be seen that the control effect of the present invention is better and it has a stronger ability to cope with parameter changes.

[0181] Fig.14 , Fig.15 They are respectively the standard parameters of the permanent magnet synchronous wind turbine 1 (R = 0.53Ω, L = 8.3mH, ψ f =0.34Wb) and parameter changes (relative to standard parameters, R, L increased by 15%, ψ f Reduce by 10%, that is, let R = 0.61Ω, L = 9.55mH, ​​ψ f =0.306Wb)), the power generation efficiency of the permanent magnet synchronous wind turbine 1 under different wind speeds during deep reinforcement learning TD3 control and PI control of the present invention. Power generation efficiency η=P dc / P w , where P dc is the DC side output power, P w Capturing power for wind turbines.

[0182] Depend on Fig.14 It can be seen that the efficiency of the deep reinforcement learning TD3 control method of the present invention reaches 92.9% at rated wind speed, while the efficiency of the PI control method is 89.5%.

[0183] Depend on Fig.15 It can be seen that when the parameters change at rated wind speed, the efficiency of the deep reinforcement learning TD3 control method of the present invention can still reach more than 92%, while the PI control method is 88.2%. It can be seen that the present invention has better control effect and has stronger coping ability in the face of parameter changes.

[0184] In summary, the present invention can achieve optimal control of multi-objective tasks, suppress the stator current harmonic distortion of the wind turbine, reduce power consumption, and improve power generation efficiency; it has strong adaptability and robustness to changes in wind speed and changes in the generator's own parameters, and can effectively cope with the impact of wind speed fluctuations and changes in the generator's own parameters on the control of the wind turbine set, thereby achieving rapid, stable and safe control of the permanent magnet synchronous wind turbine.

Claims

1. A deep reinforcement learning control method for a permanent magnet direct-drive wind turbine generator set, the permanent magnet direct-drive wind turbine generator set comprising: A permanent magnet synchronous wind turbine generator and a machine-side converter; the stator of the permanent magnet synchronous wind turbine generator is connected to the machine-side converter; characterized in that the following steps are adopted: Step 1, determining the environmental state of the permanent magnet synchronous wind turbine: The environmental state includes the rotor rotation angular velocity ω(t) of the permanent magnet synchronous wind turbine and its reference value ω m,r , the d-axis component i of the stator current of the permanent magnet synchronous wind turbine d and the q-axis component i q , the d-axis component u of the stator voltage of the permanent magnet synchronous wind turbine d and the q-axis component u q The stator inductance L, stator winding resistance R, rotor permanent magnet flux linkage ψ of the permanent magnet synchronous wind turbine f , the temperature T of the stator winding of the permanent magnet synchronous wind turbine st , wind speed v w and wind speed change rate The environmental state is denoted as S t , then Step 2, design the reward function: The reward function R t Designed for: R t =R fast + R smooth +R T (1) In the formula, R fast is the speed tracking reward function, R smooth is the speed fluctuation suppression and distortion rate reward function, R T is the temperature rise reward function; R fast Defined as: R fast =R1|ω m,r -ω(t)|,|ω m,r -ω(t)|≥0.2rad / s (2) Where R1 is the speed tracking weight coefficient, which is used to quickly reduce the speed tracking error; R smooth Defined as: In the formula, R2 is the distortion rate quick weight coefficient, which is used to minimize the distortion rate, i d * 、i q * are respectively the d-axis component i of the stator current of the permanent magnet synchronous wind turbine d and the q-axis component i q The reference value, A d1 is the reference value of the current difference; R3 is the stability weight coefficient, which is used to reduce the speed fluctuation of the generator; ω(tl) is the speed of the permanent magnet synchronous wind turbine at the first moment, L is the upper limit of the time for suppressing fluctuations, and l and L are both positive integers; R T Defined as: R T =R4(T al -T st ) / A d2 (4) Where R4 is the stator winding temperature weight coefficient, T st is the temperature of the stator winding of the permanent magnet synchronous wind turbine, T al is the allowable temperature of the stator winding of the permanent magnet synchronous wind turbine, A d2 is the reference value of temperature difference; Step 3: construct an intelligent agent based on the TD3 algorithm according to the environmental state: The agent is composed of a strategy network and a strategy evaluation network; the strategy network is composed of an Actor network and an ActorTarget network, respectively through the corresponding network strategy π(S t |θ) and π ta (S t |θ ta ) realizes the action output A of the agent t (S t |θ); The strategy evaluation network consists of a Critic network and a Critic Target network, which are respectively and h = 1, 2, evaluate and improve the strategy used to achieve the action output, where θ, θ ta , Represent the weight coefficients of the Actor network, Actor Target network, Critic network and CriticTarget network respectively; Step 4: training the agent: A1. Determine the training environment, including wind speed training conditions and generator parameter training conditions, where: Wind speed training condition: insert random wind speed noise δv every 1s into the actual wind speed curve w , the training wind speed v' is calculated according to formula (5): w , and at the beginning of a new training round, by δv w Reset v' according to formula (5) w ; In the formula, δv w Subject to average distribution U; v wn is the rated wind speed; w d is the deviation width, w d ≥1; Generator parameter training conditions: In order to ensure that the agent can still have excellent performance when the generator parameters change, at the beginning of each training round, the parameters G' of the permanent magnet synchronous wind turbine are trained according to formula (6) s , where s = 1, 2, 3, respectively representing the stator inductance L, stator winding resistance R and rotor permanent magnet flux ψ of the permanent magnet synchronous wind turbine f , through G s Add random parameter variation additional term δG s , calculated according to formula (6) to obtain G' s ; Also at the beginning of a new training round, by δG s Update G' according to formula (6) s , and by the maximum additional factor g max and the minimum additional factor g min Determine G s The upper and lower limits of In the formula, δG s Subject to average distribution U; w g is the deviation width, w g ≥1; A2. Create an interface for the agent to interact with the environment information, initialize the network weight parameters of the agent, and set the relevant parameters for the agent training, as follows: Initialize the weight parameters θ of the Actor main network and the Critic main network, And the corresponding target network Actor Target network, target network Critic Target network weight parameter θ ta , h=1,2; Set the learning rate α of the Actor main network; Set the discount factor γ for deep reinforcement learning; Set the maximum number of training rounds N, the maximum number of steps in each training round T, the update frequency I of the target network Actor Target network and Critic Target network, and the weight of the Critic network Update frequency I p ,I ta ; Load actual wind speed v w , wind speed deviation noise δv w , the parameter G of the permanent magnet synchronous wind turbine s and its random parameter variation additional term δG s ; Set up and initialize the experience database; A3. At the beginning of each training round, first according to the actual wind speed v w , wind speed deviation noise δv w , the parameter G of the permanent magnet synchronous wind turbine s and its random parameter variation additional term δG s Determine the wind speed training condition v' for this training round according to formula (5) and formula (6) respectively wk and generator parameter training condition G' sk ,k=1,2,…,N;v' wk is v' of the kth training round w ; G' sk is the G of the kth training round s ; A4. Based on the environmental state S t , in order to prevent the network strategy π(S t |θ) falls into a local optimum, adding action exploration noise N to the output of the Actor main network t , get action A t (S t |θ), providing an opportunity to explore "better", namely: A t (S t |θ)=π(S t |θ)+N t (7) In the formula, π(S t |θ) the output of the Actor network; N t The expected value is 0 and the variance is σ 2 The normal distribution of t satisfy: A5. After each action is executed, the next state of the environment S is obtained t+1 , and the reward function R is calculated according to formula (1) t , then the agent observes the state S at the next moment t+1 , and the environmental state S t 、Action A t (S t |θ), reward function value R t , the environmental state S at the next moment t+1 storing as an experience data set in the experience database; A6. Randomly and uniformly sample M small batches of samples from the experience database, that is, M experience data groups (S ti ,A ti ,R ti ,S ti+1 ), where i = 1, 2, 3, …, M; A7. Update the weight parameters of the Critic main network in a way that minimizes the error Right now Where Y ti is the expected target value function, defined as: In the formula, ε represents the target smooth noise, which is the truncated normal distribution noise, that is: In the formula, σ clip is the standard deviation, c is the cutoff factor; A8. Update the weight parameters θ, θ of the Actor network, Actor Target network, and Critic Target network in a delayed update manner ta and Specifically, the weights of the Critic network Every update I p The weight parameter θ of the Actor network is updated once; the weight parameter θ of the Critic network is updated once. Every update I ta Second, the weight parameters θ of the Actor Target network and the Critic Target network ta and Update once; if the number of training steps is the update frequency I of the Critic network p If it is a multiple of , then go to step A9, otherwise go to step A11; p ,I ta is the update frequency, which is a positive integer, and I ta >I p ; A9. Update the weight parameter θ of the Actor network by performing gradient ascent on the expected value J, that is, θ=θ+μ▽ θ J (12) In the formula, the expression of J is: ▽ θ J represents the derivative of J with respect to θ: In the formula, ▽ θ π(S ti |θ) represents π(S ti |θ) to derive θ; A10. If the number of training steps is the update frequency I of the Critic network ta If it is a multiple of , the weight parameter θ of the Actor Target network is adjusted by exponential smoothing according to formula (15). ta and the weight parameters of the Critic Target network To update, that is: Where, τ is the update factor, τ≤1; A11. If the number of training steps does not reach the maximum number of steps T for each training round, return to step A4; otherwise, the current training round ends, and the agent model obtained in this training round is saved, and the average reward function value R of this training round is calculated and saved. e ; A12. If the average reward function value R for n consecutive training rounds e The absolute value of R set , where R set is a positive real number close to 0, the entire training process ends; otherwise, go to step A13; A13. Determine whether the maximum number of training rounds N has been reached. If reached, the entire training process is terminated and training is restarted. If not reached, return to step A3 and start the next training round. Step 5: transplant the trained intelligent model into the main control chip of the machine-side converter, establish an actual speed control system based on the main control chip of the machine-side converter, realize speed control, and output the stator current q-axis component i of the permanent magnet synchronous wind turbine q The reference value i q * ; Step 6, adopting the zero d-axis current control strategy and the current feedforward decoupling control strategy, designing the inner loop current tracking controller of the machine-side converter, and realizing the current tracking control of the permanent magnet synchronous wind turbine. The specific process is as follows: 61) Output A obtained in step 5 t (S t |θ), i.e., the q-axis component i of the stator current of the permanent magnet synchronous wind turbine q The reference value i q * , and its actual value i q The output of the first PI controller is decoupled from the q-axis feedforward compensation term -Ri q -ω e L d i d +ω e ψ f Add together to obtain the q-axis component u of the stator voltage of the permanent magnet synchronous wind turbine q The reference value u q * ; Among them, ω e is the rotor electrical angular velocity of the permanent magnet synchronous wind turbine, L d is the d-axis component of the stator inductance; 62) Let the d-axis component i of the stator current of the permanent magnet synchronous wind generator be d The reference value i d * =0, and compare it with its actual value i d The second PI controller is used as the differential input, and the second PI controller is decoupled from the d-axis feedforward compensation term -Ri d +ω e L q i q Add together to obtain the d-axis component u of the stator voltage of the permanent magnet synchronous wind turbine d The reference value u d * Among them, L q is the q-axis component of the stator inductance; Step 7: The stator voltage reference value u of the permanent magnet synchronous wind turbine generator obtained in step 6 is d * 、u q * After dq / αβ coordinate transformation, we get u α * 、u β * ; Then, u α * and u β * After modulation by the SVPWM module, a driving signal is generated to drive the power switch tube of the machine-side converter to control the operation of the permanent magnet synchronous wind turbine.

Images