Microgrid frequency control method and system based on deep deterministic policy gradient

Abstract

The present invention discloses a microgrid frequency control method and system based on deep deterministic policy gradient, which belongs to the field of power system frequency control. It includes taking the frequency deviation of the microgrid system and its integral as training data, and using a double-delay deep deterministic policy gradient algorithm to train an intelligent agent; applying the trained intelligent agent to a microgrid system with new energy, inputting the state information of the current system into an AC framework, selecting the best action, and converting it into actual instructions for the regulator valve opening of the synchronous generator to control the frequency of the microgrid. The present invention uses a model-free deep reinforcement learning algorithm to train an intelligent agent to adaptively learn grid frequency changes. Because microgrids containing new energy have the characteristics of randomness and intermittency, the present invention does not need to rely on an ideal mathematical model that has a large deviation from the real environment, but only requires the input and reward values ​​of the system to continuously learn and iterate, thereby having a better control effect on the microgrid.

Description

Technical Field

[0001] The present invention relates to the field of power system frequency control, and more specifically, to a microgrid frequency control method and system based on deep deterministic policy gradient. Background Art

[0002] With the development of power system frequency control, new energy and their continuous introduction into the power system, new frequency control methods are needed to meet this challenge and achieve frequency stability in the power system within the specified range of 50±0.2Hz. This has inspired a large number of scientific researchers to engage in research on topics related to power system frequency control.

[0003] However, due to the complexity of the power system environment, most researchers use simple linear approximations to simulate the power grid environment when designing controllers, which will lead to some indescribable power grid characteristics being mathematically modeled or linearized, resulting in large errors. On the other hand, with the continuous development of the "dual carbon" strategy, a certain proportion of new energy power generation equipment has been introduced into the microgrid system. The introduction of these devices makes the system random and intermittent, making the frequency control of the system difficult.

[0004] The current frequency control methods use overly simple mathematical modeling of the power system, do not consider the characteristics of new energy in the system, or the traditional control methods cannot adaptively learn changes in system parameters. It is particularly important to realize a new method that does not rely on the system model and can make adaptive adjustments to changes in system parameters.

[0005] The microgrid system containing new energy power generation equipment is composed of wind power generation equipment, photovoltaic power generation equipment, battery energy storage system, AC interconnection line, synchronous generator, power load and other modules. Wind power and photovoltaic power plants are relatively random, so the control effect is not good. Therefore, the synchronous generator is mainly controlled. The frequency signal of the microgrid system can be detected by sensors, and the frequency signal and the nominal frequency are subtracted to obtain the frequency deviation of the power grid and the integral signal as the observable state.

[0006] Therefore, combined with the above points, it is of great significance to develop a new microgrid system frequency control method that does not need to consider the complexity of the model and can effectively cope with changes in system model parameters. Summary of the invention

[0007] In view of the defects of the prior art, the present invention provides a microgrid frequency control method and system based on deep deterministic policy gradient, aiming at the situation where the microgrid system containing wind power, photovoltaic and energy storage equipment with randomness and intermittency cannot be accurately modeled. By not needing to establish an accurate model, but using offline training to allow the intelligent agent to continuously learn, a frequency control method for a new microgrid system based on deep reinforcement learning is provided.

[0008] To achieve the above objectives, the present invention provides a microgrid frequency control method based on deep deterministic policy gradient, in which a double-delay deep deterministic policy gradient algorithm is used to train an intelligent agent to replace a traditional controller for frequency control action decision-making. The method specifically includes the following steps:

[0009] The new energy environment containing wind turbines, photovoltaic power generation and battery energy storage systems is modeled. The purpose of building this environment is to obtain system data as the input of the intelligent agent and obtain the system environment feedback state as the evaluation of the degree of training of the intelligent agent. The method itself is model-free. Define the training environment corresponding to the microgrid simulation model under the AC framework, measure the frequency deviation (the difference between the current frequency value and the nominal frequency value of the system) and the integral of the frequency deviation of the above microgrid system through sensors for state observation, and use it as the observation input of the intelligent agent. Use the double-delay deep deterministic policy gradient algorithm to train the intelligent agent.

[0010] The trained intelligent agent is applied to a microgrid system with renewable energy. The state information of the current microgrid system is input into the performer-critic framework to ensure that the policy network selects the best action under the evaluation of the Q value of the evaluation network, which is converted into actual instructions for the regulator valve opening of the synchronous generator through the backup controller. The microgrid system is controlled to restore balance when the power is unbalanced, and the frequency is maintained within the specified range of 50±0.2Hz to ensure stable operation of the system.

[0011] Furthermore, in order to ensure the correctness of the derivation of the steps of this method, it is established on the following mathematical theoretical basis:

[0012] Markov decision process is the basic mathematical model of reinforcement learning. MDP can be represented as a tuple in is the state space, is the action space, is the transition function, which gives the The probability matrix of transitioning to other states, is the reward function and 0≤γ≤1 is the discount factor.

[0013] The agent’s policy is a mapping from states to the probability of choosing each action: Usually denoted as π. The goal of reinforcement learning is to maximize the long-term reward G t To search for optimization strategy π * ,Right now where k = 1, 2, 3, ..., R t represents the reward at time t, which determines the current and future return values. As long as γ < 1, R(k) is bounded, G t is bounded.

[0014] The microgrid frequency control problem obviously satisfies the Markov assumption, so it can be considered as a Markov decision process. The system frequency deviation and the integral of the system frequency deviation are taken as state observations. The action output by the intelligent agent is the final selected control action. In order to ensure the continuous operation of the entire system, a new microgrid system with new energy power generation equipment is built. The observation value is taken as the input of the intelligent agent, and the output of the intelligent agent is acted on the regulator valve opening of the synchronous generator.

[0015] Furthermore, the present invention adopts a deep reinforcement learning method based on the AC framework, which includes two networks, one is the policy network, represented by π, and the other is the evaluation network, represented by Q π (s,a) means, defined as: in represents the expected value of the random variable given the strategy π, and t represents the time step, which is selected as 0.01s. At this stage, the state-action value function Q(s,a) is calculated by the time difference (TD) method, that is, Q(s t ,a t )←Q(s t ,a t )+δ t ,in is the TD error, The TD error can be used to evaluate the most recently selected action a. Based on the above actor-critic framework and the characteristics of the frequency control problem of the microgrid system as a continuous action space, the present invention uses a double-delayed deep deterministic policy gradient algorithm to train the agent. The specific steps of the algorithm are:

[0016] 1) Initialize the parameters θ of the two evaluation main networks i (i=1,2), and the strategy master network parameter φ;

[0017] 2) Initialize the evaluation target network θ i ′(i=1,2) and the strategic target network φ′;

[0018] 3) Initialize the experience pool (store past learning experience, and use it to sample the data in the experience pool to have more data for subsequent training steps or network parameter updates)

[0019] 4) Circuit training:

[0020] a. The current state of the system s = (Δf, ∫Δf) is observed by the sensor and given to the agent, which then takes action a = (ΔP c ) is selected and applied to the current system, and the state s′=(Δf′,∫Δf′) of the current system at the next moment after the current action is taken is continued to be observed, and the reward value given to the agent by this action is calculated through the reward function.

[0021] b. Perform batch random sampling in the experience pool and update the parameters of the main network by minimizing the loss function. The loss function formula is:

[0022]

[0023] Where γ represents the discount factor, Q′(s′,π(s′|φ)) represents the state-action value that can be obtained by taking action a′=π(s′|φ) through strategy π under the state s′=(Δf′,∫Δf′).

[0024] c. If the current cycle step is a multiple of 2, through the deterministic policy gradient

[0025]

[0026] Update the parameters of the policy network and target evaluation network. The update method is soft update:

[0027] φ′←τφ+(1-τ)φ′

[0028] θ i ′←τθ i +(1-τ)θ i '

[0029] After the training is completed, the convergence of the reward function is observed to determine whether the agent has explored the optimal action. Then the strategy learned by the agent is put into the simulation environment of the new microgrid system, and the frequency changes are observed under various disturbances to verify the control effect of the agent.

[0030] In the microgrid frequency control method based on deep deterministic policy gradient described in the present invention, a frequency control method that does not require a model is provided, and actions are continuously explored and learned through offline training until a suitable action decision method is learned, so that suitable control actions are made when the frequency of the system changes.

[0031] Furthermore, the policy π of the policy network in the performer-critic framework adopts the optimal policy π of the action with the largest evaluation function Q *, defined as:

[0032] π * =argmin(Q1(s,a|θ1),Q2(s,a|θ2))

[0033] Among them, Q i (s,a|θ i ), i = 1, 2 indicates that the maximum state action value is calculated by evaluating the action a taken by the main network under state s, and the argmin(Q1, Q2) function indicates taking the smaller value of Q1 and Q2.

[0034] Furthermore, the attenuated ε-greedy strategy adopts a larger value of ε in the early stage of training to ensure the exploratory nature of the action, and gradually attenuates ε in the later stage of training to ensure the selection of the optimal action. The specific attenuation method is:

[0035] ε←0.99*ε

[0036] ε represents the greedy strategy coefficient;

[0037] The exploration noise is a Gaussian distribution noise with a pruning function:

[0038]

[0039] Among them, clip represents the pruning function, for Gaussian noise Pruning is performed so that the final noise range remains within (-c, c).

[0040] Another aspect of the present invention provides a microgrid frequency control system based on deep deterministic policy gradient, comprising: a computer-readable storage medium and a processor;

[0041] The computer-readable storage medium is used to store executable instructions;

[0042] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the above-mentioned microgrid frequency control method based on deep deterministic policy gradient.

[0043] Compared with the prior art, the above technical solution conceived by the present invention can achieve the following beneficial effects:

[0044] 1. In the traditional frequency control method, it is necessary to mathematically model the power system. Based on the traditional DDPG, the present invention improves the use of a dual evaluation network to avoid overestimation of Q, and uses delayed update of the target network and the action network to ensure the exploratory and usable nature of the action. There is no need to consider complex and difficult-to-model mathematical models containing various power generation equipment, electrical components, etc., making the control method involved more adaptable.

[0045] 2. Compared with the existing control methods, the method proposed in the present invention is based on learning based on the observable state of the system. The algorithm adopts the policy gradient algorithm, which solves the problem of continuous action space very well and can enable the trained intelligent agent to have better dynamic performance when it is put into the power system environment for frequency control.

[0046] 3. Compared with the existing traditional control methods, the present invention can save the computing power of designing the controller, especially when there are great changes in the model in the power grid, there is no need to consider the changes in the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The microgrid system structure diagram containing new energy equipment provided by the present invention;

[0048] Figure 2 A specific structural diagram of the strategy network in the agent training process provided by the present invention;

[0049] Figure 3 A specific structural diagram of the evaluation network in the agent training process provided by the present invention;

[0050] Figure 4 A structural diagram of a double-delay deterministic policy gradient algorithm provided by the present invention;

[0051] Figure 5 A reward function diagram for the agent training process of the present invention;

[0052] Figure 6 It is a frequency variation diagram of the control method invented by the present invention under step disturbance;

[0053] Figure 7 A continuous disturbance diagram of the simulation environment of the present invention;

[0054] Figure 8 This is a frequency variation diagram of the control method provided by the present invention under continuous disturbance. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0056] The present invention provides a microgrid frequency control method based on deep deterministic policy gradient, which uses a double-delay deep deterministic policy gradient algorithm to train an intelligent agent to replace a traditional controller for an action decision method of frequency control. The method specifically includes the following steps:

[0057] The new energy environment containing wind turbines, photovoltaic power generation and battery energy storage systems is modeled. The purpose of building this environment is to obtain system data as the input of the intelligent agent and obtain the system environment feedback state as the evaluation of the degree of training of the intelligent agent. The method itself is model-free. Define the training environment corresponding to the microgrid simulation model under the AC framework, measure the frequency deviation (the difference between the current frequency value and the nominal frequency value of the system) and the integral of the frequency deviation of the above microgrid system through sensors for state observation, and use it as the observation input of the intelligent agent. Use the double-delay deep deterministic policy gradient algorithm to train the intelligent agent.

[0058] The trained intelligent agent is applied to a microgrid system with renewable energy. The state information of the current microgrid system is input into the performer-critic framework to ensure that the policy network selects the best action under the evaluation of the Q value of the evaluation network, which is converted into actual instructions for the regulator valve opening of the synchronous generator through the backup controller. The microgrid system is controlled to restore balance when the power is unbalanced, and the frequency is maintained within the specified range of 50±0.2Hz to ensure stable operation of the system.

[0059] Figure 1 This is the structure diagram of the microgrid system containing new energy equipment. Its specific mathematical model is as follows.

[0060] (1) The wind turbine model is:

[0061]

[0062] Among them G WTG is the transfer function of the wind turbine, K WTG is the gain constant of the wind turbine, T WTG is the wind time constant, ΔP wind and ΔP WTG are the deviations of wind turbine input and output respectively.

[0063] (2) Photovoltaic power generation model:

[0064]

[0065] Among them G PV is the transfer function of photovoltaic power generation, K PV is the gain constant of photovoltaic power generation, T PV is the photovoltaic power generation time constant, ΔP pv and ΔP PV are the deviations of photovoltaic power input and output respectively.

[0066] (3) Battery energy storage model: The battery energy storage system has a charging mode and a discharging mode, and the transfer function is 1 / (1+sT BES) is a simplified description of the battery energy storage system, where ΔP BES is the power deviation.

[0067] (4) Synchronous generator model: The governor-turbine model 1 / (1+sT g )(1+sT t ) indicates that T g and T t are the governor time constant and turbine time constant respectively. The control of the generator is through the control action ΔP c Control the opening of the valve for regulation.

[0068] The specific hyperparameter design in the detailed agent training process is as follows:

[0069] The AC framework based on this invention uses neural networks to approximate the mapping relationship of data. The A strategy network and the C evaluation network refer to Figure 2 and Figure 3 . Figure 2 The main structure of the policy network is shown. The frequency deviation and its integral s = (Δf, ∫Δf) are used as the input value of the input neuron. After passing through 32 hidden layer neurons, one output neuron outputs the control action ΔP. c . Figure 3 The main structure of the evaluation network is shown, with (Δf,∫Δf,ΔP c ) is used as the input state of the input neuron. After passing through 32, 16, 16, and 8 hidden neurons and three layers of relu activation function, the output neuron outputs the corresponding state action value Q. The main training process is as follows Figure 4 As shown:

[0070] 1) Initialize the parameters θ of the two evaluation main networks i (i=1,2), and the strategy master network parameter φ;

[0071] 2) Initialize the evaluation target network parameters θ i ′(i=1,2) and policy target network parameter φ′;

[0072] 3) Initialize the experience pool (used to store past learning experience and for sampling learning);

[0073] 4) Circuit training:

[0074] a. Observe the state of the current system s = (Δf, ∫Δf) through sensors and give it to the agent. Select actions by adding exploratory noise under the strategy π to ensure the exploratory nature of the action. Continue to observe the state of the current system s′ = (Δf′, ∫Δf′) at the next moment after taking the current action, and calculate the reward value given to the agent by this action.

[0075] b. Perform batch random sampling in the experience pool and update the parameters of the main network by minimizing the loss function. The loss function formula is:

[0076]

[0077] Where γ represents the discount factor, Q′(s′,π(s′|φ)) represents the state-action value that can be obtained by taking action a′=π(s′|φ) through strategy π under the state s′=(Δf′,∫Δf′).

[0078] c. If the current training number is a multiple of 2, through the deterministic policy gradient

[0079]

[0080] Update the parameters of the policy network and target evaluation network. The update method is soft update:

[0081] φ′←τφ+(1-τ)φ′

[0082] θ i ′←τθ i +(1-τ)θ i '

[0083] Other hyperparameter designs are shown in Table 1.

[0084] Table 1

[0085] Hyperparameters Suitable value Maximum number of training sessions 500 Maximum training steps / times 1000 Experience pool size 64000000 Experience pool batch size 64 Learning rate of the policy network 0.0001 Evaluate the learning rate of the network 0.0001 Discount Factor 0.98

[0086] Repeat the above training steps until the training is completed, and then observe the reward function and perform convergence analysis, such as Figure 5 It shows that when the number of training steps reaches 450, the reward value has basically converged to near 0. By analyzing the reward function R = -e n|Δf| , we can see that when the control target Δf→0, the reward value is 0, so Figure 5 This indicates that the reward function has converged, that is, the agent strategy has reached the optimal state.

[0087] After the training is completed, the trained intelligent agent model is saved and put into the new microgrid system for real-time control to observe the control effect. Figure 6 It means that when the wind farm, photovoltaic power plant and load-side disturbances occur at 10s, 20s and 30s respectively, the trained agent can restore the frequency of the microgrid system to stability; Figure 7 It represents a continuous disturbance, which is a more practical reflection of the randomness and intermittency of energy storage devices such as wind power and photovoltaics in the new power system. When this continuous disturbance occurs, the frequency change of the system is as follows: Figure 8As shown in the figure, the frequency fluctuates continuously due to continuous disturbances, but under the continuous real-time control of the intelligent agent, it can be guaranteed to be 50±0.1Hz, meeting the frequency requirement range of 50±0.2Hz specified by the power grid; in addition, Figure 8 It also includes a comparison of the control effects of the present invention and other controllers under the same disturbance, which are superior to traditional PI controllers and H∞ controllers in terms of dynamic characteristics such as overshoot, adjustment time and peak time.

[0088] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A microgrid frequency control method based on deep deterministic policy gradient, characterized in that: The following steps are involved: S1. Define the training environment corresponding to the microgrid simulation model under the performer-critic AC framework, use the measured frequency deviation of the microgrid system and the integral of the frequency deviation as the training data of the agent, and use the double-delayed deep deterministic policy gradient algorithm to train the agent; The training environment includes: Set the observation state quantity required by the agent: including the frequency deviation Δf and the integral of the frequency deviation ∫Δf. The state space is expressed as: S = (Δf, ∫Δf); Set the action of the agent: The action space is expressed as: A = (ΔP c ), ΔP c Indicates the valve opening of the synchronous generator regulator; Set the agent's reward function during training: R = -e n|Δf| , where e is the natural exponent, n is the constant term, and |Δf| represents the absolute value of the system frequency deviation; Establish a state action value evaluation function: in, γ is the discount factor, k is the constant term; The method of training an agent using a double-delayed deep deterministic policy gradient algorithm comprises the following steps: S11. Initialize the parameters θ of the two evaluation main networks i And the policy master network parameters φ with random parameters, i = 1, 2, initialize the evaluation target network θ i ′ and the policy target network φ′, initialize the experience pool; the policy network includes the policy main network and the policy target network, the evaluation network includes the evaluation main network and the evaluation target network, the main network includes the evaluation main network 1 and the evaluation main network 2, the evaluation target network includes the evaluation target network 1 and the evaluation target network 2, the evaluation network adopts a dual Q network, and the network with the smaller Q value is selected during calculation; S12. Observe the current state of the system s = (Δf, ∫Δf) and give it to the agent. After adding exploration noise under the strategy π of the policy network in the AC framework, the attenuated ε-greedy strategy is used to perform action a = (ΔP c ), observe the state s′=(Δf′,∫Δf′) of the current system at the next moment after taking the current action, and calculate the reward value of this action. c ,r) is stored in the experience pool, where r is the reward value calculated by the reward function, ΔP c It is the specific action value of the currently selected control valve opening; S13. Perform batch random sampling in the experience pool, and pass the sampled state set s = (Δf, ∫Δf) and action set a = (ΔP c ), as the input of the evaluation main network, and the output Q(s,a|θ) is obtained i ), and by minimizing the error equation Update the parameters of the evaluation main network, where γ represents the discount factor, and Q′(s′,π(s′|φ)) represents the state-action value that can be obtained by taking action a′=π(s′|φ) through strategy π in the state s′=(Δf′,∫Δf′); S14. Through deterministic policy gradient Update the parameters of the strategy main network, thereby updating the parameters of the strategy target network and the evaluation target network: φ′←τφ+(1-τ)φ′ i i ′←τθ i +(1-τ)θ i ′ Where τ is the update parameter; The policy π of the policy network in the performer-critic framework in S12 adopts the optimal policy π of the action with the largest evaluation function Q. * , defined as: p * =argmin(Q1(s,a|θ1),Q2(s,a|θ2)) Among them, Q i (s,a|θ i ), i = 1, 2 means evaluating the action a taken by the main network under state s to calculate the maximum state action value, and the argmin (Q1, Q2) function means taking the smaller value of Q1 and Q2; The attenuated ε-greedy strategy adopts ε in the early stage of training to ensure the exploratory nature of the action. In the later stage of training, ε is gradually attenuated to ensure the selection of the optimal action. The specific attenuation method is: ε←0.99*ε ε represents the greedy strategy coefficient; The exploration noise is a Gaussian distribution noise with a pruning function: Among them, clip represents the pruning function, for Gaussian noise Prune the branches so that the final noise range remains at (-c, c); S2. Apply the trained intelligent agent to the microgrid system with new energy, input the current state information of the microgrid system into the performer-critic framework and ensure that the policy network selects the best action under the evaluation of the Q value of the evaluation network, which is converted into actual instructions for the regulator valve opening of the synchronous generator through the backup controller, controlling the microgrid system to restore balance when the power is unbalanced, and maintaining the frequency within the preset range to ensure stable operation of the system.

2. A microgrid frequency control system based on deep deterministic policy gradient, characterized in that: include: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the microgrid frequency control method based on deep deterministic policy gradient according to claim 1.

Images