Load frequency control method combining deep reinforcement learning and PI control

Abstract

The present invention discloses a load frequency control method combining deep reinforcement learning with PI control, comprising the following steps: S1, constructing a deep reinforcement learning controller, comprising an actor network, a critic network, an actor target network and a critic target network; S2, processing frequency information into a state signal through a signal processor and inputting the state signal into the actor network and the actor target network respectively; S3, the actor network selects an action according to a strategy, inputs the action into a power system and a critic network, calculates reward information and inputs the reward information into the actor network and the actor target network respectively; S4, updating the state value, calculating the action value, calculating the error of the critic network, calculating the gradient of the critic network, and calculating the gradient of the actor network; S5, comparing the parameters of the actor network and the actor target network, and the parameters of the critic network and the critic target network to determine whether to continue training; S6, connecting the PI controller and the deep reinforcement learning controller to the power system, comparing the frequency change with the set value, and selecting a controller to control the generator speed regulator.

Description

Technical Field

[0001] The present invention belongs to the field of novel power technology, and specifically relates to a load frequency control method combining deep reinforcement learning with PI control. Background Art

[0002] New energy sources will become the primary power source in new power systems. High penetration rates of renewable energy will profoundly alter the form, characteristics, and mechanisms of traditional power systems. These systems will feature a high proportion of renewable energy and power electronics, exhibiting new dynamic characteristics such as low moment of inertia and wide-band oscillations. Frequency stability issues will also become more complex. Current system operation control theories and technologies, based on traditional synchronous generators, are unable to meet the requirements for safe operation of these new power systems. Comprehensive reforms and breakthroughs are urgently needed in fundamental system theory, analysis methods, and control technologies.

[0003] Traditional methods are based on classical electromechanical theory and mathematical models, with large action time constants, and system analysis methods are established on the basis of the electromechanical characteristics of slow systems. However, due to the control effect of power electronic equipment, the new power system with weak inertia has a small action time constant and a wide frequency domain distribution, making it difficult to establish an accurate mathematical model, which brings difficulties to the power system analysis method.

[0004] Traditional methods for system stability analysis and calculation are based on physical models and time-domain simulations, analyzing stability mechanisms by numerically solving algebraic or differential equations. However, due to the massive access to power electronic equipment and the diverse operating modes of new power systems, solutions based on physical models face the curse of dimensionality, which makes it difficult to calculate power system stability.

[0005] (1) Traditional control methods such as PID, fuzzy logic control, and model predictive control require complex mathematical models. However, expert expertise, experience, and knowledge are difficult to obtain. (2) Optimal tracking curve network algorithms optimized by particle swarm optimization, genetic algorithms, and neural networks are usually only effective for specific cycles, lack online learning capabilities, and have limited generalization capabilities. (3) Classic reinforcement learning methods such as Q-learning are prone to the curse of dimensionality problem, have poor generalization capabilities, and are usually only useful for specific tasks. Summary of the Invention

[0006] The purpose of the present invention is to solve the problem in the existing technology that due to the massive amount of power electronic equipment and new energy access and the changeable operating modes, the solution based on the physical model faces the dimensional disaster, which brings difficulties to the stable control of power frequency. A load frequency control method combining deep reinforcement learning and PI control is provided. By combining artificial intelligence and traditional PI control, a better frequency control effect can be achieved.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0008] A load frequency control method combining deep reinforcement learning with PI control includes the following steps:

[0009] S1. Build a deep reinforcement learning controller, including the actor network, critic network, actor target network, and critic target network. Initialize the parameters of the actor network and critic network, and assign the parameters of the actor network and critic network to the actor target network and critic target network respectively.

[0010] S2. The power system processes the frequency information into frequency change, frequency change rate, and total frequency change through a signal processor, and inputs the frequency change, frequency change rate, and total frequency change as status signals of the current state into the Actor network and the Actor target network respectively;

[0011] S3: The Actor network selects an action based on the strategy and uses this action as the action signal of the deep reinforcement learning controller. It is input into the power system and the Critic network respectively. The power system enters the update state and calculates the reward information of the current state based on the state signal. It is also input into the Actor network and the Actor target network respectively.

[0012] S4. Update the state value of the current state through the Critic network, calculate the action value of the action in the current state, calculate the gradient of the Actor network and update the parameters of the Actor network, calculate the error of the Crtic network, calculate the gradient of the Crtic network and update the parameters of the Crtic network;

[0013] S5. Compare the parameters of the Actor network and the Actor target network, as well as the parameters of the Critic network and the Critic target network to see if they are the same. If the parameters are the same, stop training. If the parameters are different, continue training the deep reinforcement learning controller using frequency information.

[0014] S6. Connect the PI controller and the trained deep reinforcement learning controller to the power system, compare the frequency change of the power system with the set switching frequency constant, and select a controller based on the comparison result to control the generator speed regulator of the power system.

[0015] Furthermore, the calculation method of reward information is:

[0016]

[0017] Among them, r t is the reward of the current state, rt-1 is the reward of the previous state, Δf is the frequency change of the system, is the frequency change rate of the system, k1 is a constant that reflects the importance attached to the frequency change, and k2 is a constant that reflects the importance attached to the frequency change rate.

[0018] Reward Function It includes three items, the first item is r t-1 is the previous reward, and the sum of the last two is the reward obtained in this step. In this way, all the reward information can be accumulated to comprehensively evaluate the control effect of the controller.

[0019] Among them, the second term -k1Δf 2 To take into account the frequency change, the greater the frequency change, the greater the negative reward obtained in this step. Taking into account the characteristics of the power system itself, fluctuations within a certain frequency range are allowed, but frequency exceeding the limit is absolutely intolerable. Therefore, a square term is used. When the frequency change is small, this amount can be ignored. The third term To account for the rate of change of frequency, we hope to avoid low-frequency oscillations. In summary, a clever reward function design can better reflect the effectiveness of the controller and can be used in deep reinforcement learning.

[0020] Furthermore, the state value of the current state is updated as follows:

[0021] V(s t )=V(s t )+α[r t+1 +γV(s t+1 )-V(s t )]

[0022] Among them, V(s t ) is the state value of the current state, r t+1 is the reward for the next state, γ is the discount rate, V(s t+1 ) is the state value of the next state, and α is the learning rate.

[0023] The value of the current state V(s t ) is initially an estimated value, not a true value. Through continuous training and correction based on subsequent values, a more accurate state value can be obtained.

[0024] Furthermore, the action value is calculated as:

[0025] Q(s,a)=r+γV(s t+1 )

[0026] Among them, Q(s,a) is the action value of action a in state s, and r is the reward obtained by performing the current action a.

[0027] The action value Q(s,a) is initially an estimate, not a true value. Through continuous training and correction based on subsequent values, a more accurate action value can be obtained.

[0028] Furthermore, the gradient of the Actor network is calculated as follows:

[0029]

[0030] in, is the gradient of the Actor network, N is the number of adjacent states randomly selected for calculation, To find the gradient of the Actor network with respect to action a, Q(s,a| Q ) is the action value of action a in state s calculated based on the parameter Q of the Crtic network, s i is the i-th state, a is the action selected according to the strategy μ, To find the gradient of the policy μ with respect to θ, μ(s| μ ) is the strategy for selecting actions.

[0031] By obtaining the gradient of the Actor network Using the gradient descent method, we can Finding the update direction of the parameter μ of the Actor network can quickly complete the update of the parameter μ of the Actor network.

[0032] Furthermore, the error of the Crtic network is calculated as follows:

[0033] TD-error=r+γV'(s t+1 )-V(s t )

[0034] Among them, TD-error is the error of Crtic network, V'(s t+1 ) is the state value of the next state calculated by the Crtic target network, and r is the reward obtained by performing the current action a.

[0035] The Crtic network starts with an estimated value. By determining the error TD-error between the Crtic network and the Crtic target network, it is used to modify the parameters of the Crtic network. When the error is very small, the result obtained by the Crtic network can be considered to be an accurate result.

[0036] Furthermore, the gradient calculation method of the Crtic network is:

[0037]

[0038] in, is the gradient of the Crtic network, N is the number of adjacent states randomly selected for calculation, To find the gradient of the parameter Q of the Critic network with respect to θ, Q(s,a| Q ) is the action value of action a in state s calculated according to the parameter Q of the Crtic network, s is the state, s i is the i-th state, μ is the strategy for selecting actions, and a is the action selected according to the strategy μ.

[0039] By obtaining the gradient of the Crtic network Using the gradient descent method, we can Finding the update direction of the parameter Q of the Crtic network can quickly complete the update of the parameter Q of the Crtic network.

[0040] Furthermore, the method for comparing the frequency change of the power system with the set switching frequency constant is as follows:

[0041] If the frequency change of the power system is greater than or equal to the set switching frequency constant, the deep reinforcement learning controller is used; if the frequency change of the power system is less than the set switching frequency constant, the PI controller is used.

[0042] The parameters of the Actor network and the Critic network are originally estimated values. The parameters of the Actor target network and the Critic target network are temporarily set to actual values. When there is no error between the estimated value and the actual value, it can be considered that the parameters of the Actor network and the Critic network have converged, the training is completed, and can accurately reflect the results.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention introduces artificial intelligence methods into the field of power system frequency control, designs a reward function that not only considers the current frequency change, but also the speed of frequency fluctuation, can converge quickly and produce better training effects; it also combines deep reinforcement learning methods with traditional PI control, adopts simple traditional PI control when the frequency change is small, and adopts deep reinforcement learning control when the frequency change is large, with good robustness, stability and universal applicability; the present invention can be applied to the load frequency control of new energy power systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Diagram of training a deep reinforcement learning controller. DETAILED DESCRIPTION

[0046] The load frequency control method combining deep reinforcement learning with PI control of the present invention is further described below with reference to the accompanying drawings and specific embodiments.

[0047] See also Figure 1 The present invention discloses a load frequency control method combining deep reinforcement learning with PI control, comprising the following steps:

[0048] S1. Build a deep reinforcement learning controller, including the Actor network, Critic network, Actor target network, and Critic target network. Initialize the parameters of the Actor network and Critic network, and assign the parameters of the Actor network and Critic network to the Actor target network and Critic target network respectively.

[0049] S2. The power system processes the frequency information into frequency change, frequency change rate, and total frequency change through a signal processor, and inputs the frequency change, frequency change rate, and total frequency change as status signals of the current state into the Actor network and the Actor target network respectively.

[0050] S3. The Actor network selects an action based on the strategy and uses this action as the action signal of the deep reinforcement learning controller, which is input into the power system and the Critic network respectively. The power system enters the update state, calculates the reward information of the current state based on the state signal, and inputs it into the Actor network and the Actor target network respectively.

[0051] S4. Update the state value of the current state through the Critic network, calculate the action value of the action in the current state, calculate the gradient of the Actor network and update the parameters of the Actor network, calculate the error of the Crtic network, calculate the gradient of the Crtic network and update the parameters of the Crtic network.

[0052] S5. Compare the parameters of the Actor network and the Actor target network, as well as the parameters of the Critic network and the Critic target network to see if they are the same. If the parameters are the same, stop training. If the parameters are different, continue training the deep reinforcement learning controller through frequency information.

[0053] S6. Connect the PI controller and the trained deep reinforcement learning controller to the power system, compare the frequency change of the power system with the set switching frequency constant, and select a controller based on the comparison result to control the generator speed regulator of the power system.

[0054] The deep reinforcement learning method in the present invention is Deep Deterministic Policy Gradient (DDPG), which is a policy learning method that integrates deep learning neural networks into reinforcement learning. A neural network is used as a simulation of the policy function μ and the Q function, that is, a policy network and a Q network, and then a deep learning method is used to train the above neural network. The double-layer network used can accelerate the convergence faster. The present invention first sets a clever reward to measure the control effect, and then connects the deep reinforcement learning controller to the frequency control of the power system for training. The trained deep reinforcement learning controller is combined with the PI controller, and the switching frequency constant is set. When the frequency change is small, a simple traditional PI control is used, and when the frequency change is large, a deep reinforcement learning control is used.

[0055] Specifically, in step S1, a deep reinforcement learning controller is constructed, which includes an Actor network, a Critic network, an Actor target network, and a Critic target network. Initialize the parameters θ of the Actor network u and the parameters θ of the Critic network Q , and the parameters θ of the Actor network u and the parameters θ of the Critic network Q Assign the Actor target network and the Critic target network respectively to obtain the parameters θ of the Actor target network Q' and the parameters θ of the Critic target network u' .

[0056] In step S2, the power system samples the frequency information f and sends it to the signal processor, which processes the frequency information f into the frequency change Δf and the frequency change rate. The total frequency change ∫Δfdt three signals. The frequency change Δf, frequency change rate The total frequency change ∫Δfdt three signals are used as the state signal S t Input to the Actor network and Actor target network of the deep reinforcement learning controller respectively.

[0057] In step S3, the Actor network selects an action A according to the strategy, namely ΔP c , which is used as the controller action and input into the power system, and the power system enters the state s t+1 , calculate the reward information r t , and feed it to a deep reinforcement learning controller.

[0058] The calculation method of reward information is:

[0059]

[0060] Among them, r t is the reward of the current state, r t-1 is the reward of the previous state, Δf is the frequency change of the system, is the frequency change rate of the system, k1 is a constant that reflects the importance attached to the frequency change, and k2 is a constant that reflects the importance attached to the frequency change rate.

[0061] Reward Function It includes three items, the first item is r t-1 For the previous reward, the last two The sum is the reward obtained in this step, so that all the reward information can be accumulated to comprehensively evaluate the control effect of the controller. 2 To take into account the frequency change, the greater the frequency change, the greater the negative reward obtained in this step. Taking into account the characteristics of the power system itself, fluctuations within a certain frequency range are allowed, but frequency exceeding the limit is absolutely intolerable. Therefore, a square term is used. When the frequency change is small, this amount can be ignored. The third term To account for the frequency change rate, we hope to avoid low-frequency oscillations. The ingenious reward function design of the present invention can better reflect the control effect of the controller and is used in deep reinforcement learning.

[0062] In step S4, after each training round, the parameters θ of the Actor network are adjusted. u and the parameters θ of the Critic network Q Assign the Actor target network and the Critic target network respectively, and obtain the parameter θ of the target network Critic u' and the parameters θ of the target network Q' .

[0063] The update method of state value is:

[0064] V(s t )=V(s t )+α[r t+1 +γV(s t+1 )-V(s t )]

[0065] Among them, V(s t ) is the state value of the current state, r t+1 is the reward for the next state, γ is the discount rate, V(s t+1 ) is the state value of the next state, α is the learning rate. The state value of the current state V(s t) is initially an estimated value, not a true value. Through continuous training and correction based on subsequent values, a more accurate state value can be obtained.

[0066] The action value is calculated as:

[0067] Q(s,a)=r+γV(s t+1 )

[0068] Here, Q(s,a) is the action value of action a in state s, and r is the reward obtained by performing the current action a. The action value Q(s,a) is initially an estimate, not the actual value. Through continuous training and correction based on subsequent values, a more accurate action value can be obtained.

[0069] The gradient of the Actor network is calculated as:

[0070]

[0071] in, is the gradient of the Actor network, N is the number of adjacent states randomly selected for calculation, To find the gradient of the Actor network with respect to action a, Q(s,a| Q ) is the action value of action a in state s calculated based on the parameter Q of the Crtic network, s i is the i-th state, a is the action selected according to the strategy μ (i.e. the parameter μ of the Actor network), To find the gradient of the policy μ with respect to θ, μ(s| μ ) is the strategy for selecting actions (i.e. the parameters of the Actor network). By obtaining the gradient of the Actor network Using the gradient descent method, we can Find the update direction of the Actor network's parameters μ, which can quickly complete the Actor network's parameters θ u Updates.

[0072] The error calculation method of the Crtic network is:

[0073] TD-error=r+γV'(s t+1 )-V(s t )

[0074] Among them, TD-error is the error of Crtic network, V'(s t+1) is the state value of the next state calculated by the Crtic target network, and r is the reward obtained by performing the current action a. The Crtic network initially estimates the value. The error TD-error between the Crtic network and the Crtic target network is determined and used to modify the Crtic network parameters. When this error is very small, the result obtained by the Crtic network can be considered accurate.

[0075] The calculation method of the gradient of the Crtic network is:

[0076]

[0077] in, is the gradient of the Crtic network, N is the number of adjacent states randomly selected for calculation, To find the gradient of the parameter Q of the Critic network with respect to θ, Q(s,a| Q ) is the action value of action a in state s calculated according to the parameter Q of the Crtic network, s is the state, s i is the i-th state, μ is the strategy for selecting actions (i.e., the parameters of the Actor network), and a is the action selected according to the strategy μ (i.e., the parameters of the Actor network μ). By obtaining the gradient of the Crtic network Using the gradient descent method, we can Find the update direction of the Crtic network parameter Q, which can quickly complete the Crtic network parameter θ Q Updates.

[0078] In step S5, the parameters θ of the Actor network are compared respectively. u and the parameters θ of the Actor target network u' , and the parameters θ of the Critic network Q and the parameters θ of the Critic target network Q' If the parameters are the same, stop training; if the parameters are different, continue training the deep reinforcement learning controller through the frequency information f.

[0079] The parameters θ of the Actor network u and the parameters θ of the Critic network Q Originally, these were estimated values. The parameters θ of the Actor target network u' and the parameters θ of the Critic target network Q' Temporarily set it to the actual value. When there is no error between the estimated value and the actual value, it can be regarded as the parameter θ of the Actor network. u and the parameters θ of the Critic network Q Convergence, training is completed, and the results can be accurately reflected.

[0080] In step S6, the PI controller and the trained deep reinforcement learning controller are connected to the power system together, and the frequency change of the power system is compared with the set switching frequency constant Δfref. If the frequency change of the power system is greater than or equal to the set switching frequency constant, the deep reinforcement learning controller is used. If the frequency change of the power system is less than the set switching frequency constant, the PI controller is used. Based on the comparison results, a controller is selected to control the speed regulator of the frequency-modulated generator. This can take advantage of the high robustness of deep reinforcement learning when the frequency fluctuations are large, and the stability of the PI controller when the frequency fluctuations are small.

[0081] In summary, the present invention introduces artificial intelligence methods into the field of power system frequency control, designs a reward function, and takes into account not only the current frequency change but also the speed of frequency fluctuation, which can converge quickly and produce better training effects; it also combines it with traditional PI control, adopts simple traditional PI control when the frequency change is small, and adopts deep reinforcement learning control when the frequency change is large, which has good robustness, stability and universal applicability; the present invention can be applied to the load frequency control of new energy power systems.

[0082] The above description is a detailed description of the preferred embodiments of the present invention, but the embodiments are not intended to limit the scope of the patent application of the present invention. Any equivalent changes or modifications made under the technical spirit disclosed by the present invention should fall within the patent scope covered by the present invention.

Claims

1. A load frequency control method combining deep reinforcement learning with PI control, characterized in that: The following steps are involved: S1. Build a deep reinforcement learning controller, including the actor network, critic network, actor target network, and critic target network. Initialize the parameters of the actor network and critic network, and assign the parameters of the actor network and critic network to the actor target network and critic target network respectively. S2. The power system processes the frequency information into frequency change, frequency change rate, and total frequency change through a signal processor, and inputs the frequency change, frequency change rate, and total frequency change as status signals of the current state into the Actor network and the Actor target network respectively; S3: The Actor network selects an action based on the strategy and uses this action as the action signal of the deep reinforcement learning controller. It is input into the power system and the Critic network respectively. The power system enters the update state and calculates the reward information of the current state based on the state signal. It is also input into the Actor network and the Actor target network respectively. S4. Update the state value of the current state through the Critic network, calculate the action value of the action in the current state, calculate the gradient of the Actor network and update the parameters of the Actor network, calculate the error of the Crtic network, calculate the gradient of the Crtic network and update the parameters of the Crtic network; S5. Compare the parameters of the Actor network and the Actor target network, as well as the parameters of the Critic network and the Critic target network to see if they are the same. If the parameters are the same, stop training. If the parameters are different, continue training the deep reinforcement learning controller using frequency information. S6. Connect the PI controller and the trained deep reinforcement learning controller to the power system, compare the frequency change of the power system with the set switching frequency constant, and select a controller based on the comparison result to control the generator speed regulator of the power system; The calculation method of reward information is: Among them, r t is the reward of the current state, r t-1 is the reward of the previous state, Δf is the frequency change of the system, is the frequency change rate of the system, k1 is a constant that reflects the importance attached to the frequency change, and k2 is a constant that reflects the importance attached to the frequency change rate; Compare the frequency change of the power system with the set switching frequency constant, and select the controller based on the comparison result. Specifically: If the frequency change of the power system is greater than or equal to the set switching frequency constant, the deep reinforcement learning controller is used; if the frequency change of the power system is less than the set switching frequency constant, the PI controller is used.

2. The load frequency control method combining deep reinforcement learning with PI control according to claim 1, characterized in that: The update method of the state value of the current state is: V(s t )=V(s t )+α[r t+1 +γV(s t+1 )-V(s t )] Among them, V(s t ) is the state value of the current state, r t+1 is the reward for the next state, γ is the discount rate, V(s t+1 ) is the state value of the next state, and α is the learning rate.

3. The load frequency control method combining deep reinforcement learning with PI control according to claim 2, characterized in that: The action value is calculated as: Q(s,a)=r+γV(s t+1 ) Among them, Q(s,a) is the action value of action a in state s, and r is the reward obtained by performing the current action a.

4. The load frequency control method combining deep reinforcement learning with PI control according to claim 3 is characterized in that: The gradient of the Actor network is calculated as: in, is the gradient of the Actor network, N is the number of adjacent states randomly selected for calculation, To find the gradient of the Actor network with respect to action a, Q(s,a| Q ) is the action value of action a in state s calculated based on the parameter Q of the Crtic network, s i is the i-th state, a is the action selected according to the strategy μ, To find the gradient of the policy μ with respect to θ, μ(s| μ ) is the strategy for selecting actions.

5. The load frequency control method combining deep reinforcement learning with PI control according to claim 3, characterized in that: The error calculation method of the Crtic network is: TD-error=r+γV'(s t+1 )-V(s t ) Among them, TD-error is the error of Crtic network, V'(s t+1 ) is the state value of the next state calculated by the Crtic target network, and r is the reward obtained by performing the current action a.

6. The load frequency control method combining deep reinforcement learning with PI control according to claim 5, characterized in that: The gradient calculation method of the Crtic network is: in, is the gradient of the Crtic network, N is the number of adjacent states randomly selected for calculation, To find the gradient of the parameter Q of the Critic network with respect to θ, Q(s,a| Q ) is the action value of action a in state s calculated according to the parameter Q of the Crtic network, s is the state, s i is the i-th state, μ is the strategy for selecting actions, and a is the action selected according to the strategy μ.

Images