TD3-UCB joint learning algorithm-based frequency modulation adaptive control strategy for multi-region power grid containing energy storage

Abstract

The invention discloses a frequency modulation strategy comprising an energy storage multi-region power grid, a twinning delay depth deterministic strategy gradient (TD3-UCB) combined learning algorithm adopting a confidence interval upper limit is used for the frequency modulation strategy comprising the energy storage multi-region power grid, the TD3-UCB algorithm is introduced to control a frequency modulation device, the TD3-UCB algorithm avoids the problem that the Q value is too high by adding two Q value networks, continuous control of the algorithm is more accurate, and the frequency modulation strategy is more stable. The strategy training speed can be increased through priority experience sampling, and higher optimization performance is guaranteed; and a self-adaptive power distribution strategy based on different energy storage SOC intervals is provided for energy storage secondary frequency modulation, and the intelligent agent judges the current interval of energy storage and gives a corresponding frequency modulation instruction to participate in the secondary frequency modulation of the power grid. Compared with a traditional reinforcement learning and power distribution coefficient solving method, the intelligent secondary frequency modulation method for the power grid, provided by the invention, inhibits energy storage oscillation, has better control performance, and can obtain optimal collaboration of a multi-region comprehensive energy system.

Description

Technical Field

[0001] The present invention belongs to the technical field of load frequency control (LFC), and relates to a multi-region power grid secondary frequency modulation method based on deep reinforcement learning. Background Art

[0002] With the proposal of some conventional thermal power generation strategies for energy structure transformation, motors are being replaced by renewable energy. However, renewable energy has disadvantages such as weak inertia, volatility and uncertainty. After they are connected to the power grid, they will cause frequency fluctuations and other problems in the power grid. In recent years, energy storage technology has risen rapidly, especially battery energy storage has been widely used in the field of power systems. Battery energy storage has the characteristics of accurate tracking and fast response speed, and can cooperate well with traditional thermal power units to participate in the frequency regulation of the power grid. Traditional energy storage secondary frequency regulation methods cannot save past experience and knowledge, lack the ability of autonomous learning, and need to be reinitialized every time a new task is performed, which makes the regulation efficiency low, so deep reinforcement learning methods can be introduced. Deep reinforcement learning can update network parameters by receiving rewards from the environment for actions and using historical information. It has strong autonomous learning ability and can face various disturbances in the power grid. It is widely used in AGC systems.

[0003] Judging from the existing literature, the current research on energy storage participating in power system frequency regulation still has the following deficiencies:

[0004] 1. The frequency regulation requirements faced by energy storage when participating in secondary frequency regulation are constantly changing. Depending on the frequency regulation instructions, the adjustment power and working time are different, resulting in differences in the charge state between energy storage, which will greatly affect the frequency regulation performance of energy storage.

[0005] 2. Establishing the active output frequency regulation strategy of the energy storage system requires building a corresponding electrochemical energy storage model. The existing model structure is relatively simple and single. Summary of the invention

[0006] In order to solve the problem of low regulation efficiency when energy storage participates in the secondary frequency regulation of the power grid, the present invention designs a load frequency controller based on TD3-UCB to assist the system in secondary frequency regulation. The TD3-UCB algorithm is an improvement on the continuous reinforcement learning algorithm DDPG. DDPG is only a discrete action learning algorithm based on DQN. It determines the action by evaluating the Q value, and the Q value evaluation is usually too high. TD3-UCB not only solves the problem of overestimation of the Q value of DDPG, but also introduces a priority experience sampling algorithm to speed up the convergence speed of the algorithm. Therefore, it can be better applied to the secondary frequency regulation of wind storage power systems. By dividing the energy storage into different state of charge intervals and energy storage charging and discharging modes, the TD3-UCB intelligent body gives the corresponding frequency regulation power allocation instructions and sends them to the traditional units and energy storage equipment respectively to complete the secondary frequency regulation control of the system area.

[0007] The Critic network of TD3-UCB is used to judge the Actor network, as shown in the following formula:

[0008] actor μ(s|θ), μ t (s|θ t )

[0009] criticQ(s,s|φ),Q t (s,s|φ t )

[0010] TD3-UCB updates the target network in the following ways:

[0011]

[0012] Among them, τ is the soft update parameter, θ is the online policy network parameter, θ′ is the target policy network parameter, W1 and W2 are the network parameters of online Q network 1 and 2 respectively, and W′1 and W2′ are the network parameters of target Q network 1 and 2 respectively.

[0013] The online policy network is updated as follows:

[0014]

[0015] A multi-regional power grid can be established as a learning environment for TD3-UCB agents, and the best frequency regulation method can be obtained through mutual coordination of TD3-UCB agents in multiple regions.

[0016] Establish the transfer function model of the speed regulator, the formula is as follows:

[0017]

[0018] The transfer function model of the reheater is established, and the formula is as follows:

[0019]

[0020] The transfer function model of the steam turbine is established, and the formula is as follows:

[0021]

[0022] The transfer function of a thermal power unit can be expressed as:

[0023]

[0024] Among them, T g1 is the speed regulator time constant, F H is the gain of the reheater model, T R is the reheater model time constant, T t1 is the turbine time constant.

[0025] The transfer function model of the wind turbine is established, and the formula is as follows:

[0026]

[0027] Establish the transfer function model of the energy storage battery, the formula is as follows:

[0028]

[0029] The state of charge (SOC) of energy storage is an important indicator that characterizes the charging and discharging capacity of energy storage. It is defined as the ratio of the remaining capacity E(t) of energy storage to the rated capacity E.

[0030]

[0031] Among them, when S SOC When (t) = 0, it means the remaining energy storage capacity is 0 and the energy storage is fully discharged; when S SOC (t) = 1 means the energy storage is fully charged, S SOC (t0) is the SOC at the initial moment of energy storage.

[0032] The relationship between the remaining capacity of energy storage and the charging and discharging power is as follows:

[0033]

[0034] Among them, E n (t0) is the power of the nth energy storage unit at the initial time t0, P BESS (t) is the charging and discharging power of the energy storage.

[0035] When the energy storage is in different states of charge, the intelligent experience will adaptively give the corresponding frequency modulation method according to the current working range of the energy storage and the ACE of the power system through the above-mentioned iterative method, thereby realizing the adaptive frequency modulation of the energy storage. The SOC interval of the energy storage is divided as follows: Figure 2shown.

[0036] In the actual power grid, it is necessary to comprehensively consider the operation status of the power grid and the charge status of the energy storage, and decompose the demand for the power grid and energy storage. Therefore, it is necessary to divide the intervals according to ACE to determine the output target and action depth of the energy storage. The ACE working state is divided into 4 intervals, such as Figure 3 In each interval, the controller gives corresponding frequency modulation instructions according to the different SOC states of energy storage.

[0037] The TD3-UCB agent in each area coordinates and adjusts the power distribution between the generator set and the energy storage system according to the regional control error of the regional system and the energy storage SOC state, thereby performing adaptive secondary frequency regulation. When the power grid has no secondary frequency regulation demand, it will make full use of the frequency regulation capability of conventional generator sets, restore the SOC of energy storage, and make energy storage have more spare capacity to meet the frequency regulation task of the next stage, thereby improving the utilization rate of energy storage capacity and solving the problem of insufficient energy storage frequency regulation capability.

[0038] The entire multi-regional power system is used as the training environment. The agent selects action a at each step according to the SOC state of the energy storage, the frequency deviation of the system, and the power allocation strategy in the agent. t Acting on energy storage and traditional units for frequency regulation, system feedback is timely rewarded t and the next state S t+1 The current state, action, reward value, and next state vector are stored in the experience pool to form historical data. The historical data is extracted from the experience pool through the method of priority experience sampling and sent to the intelligent agent for learning and training. The state S of the AGC system i The control action of TD3-UCB is a continuous value from 0 to 1, which is composed of environmental variables ACE, Δf and SOC monitored in real time by the regional power grid. i As a control action, the controller outputs the optimal control signal. The agent adjusts the network weights by continuously interacting with the environment to obtain the optimal control strategy, even if the power system frequency deviation is 0 and the SOC remains in a normal state.

[0039] The technical problem to be solved by the present invention is that the speed and accuracy of the learning convergence of the intelligent agent are greatly reduced when facing the continuous state space and action space in the traditional discrete reinforcement learning method. At the same time, the frequency modulation efficiency of energy storage participating in the secondary frequency modulation of the power grid is improved, the energy storage is divided into different working intervals, and the corresponding frequency modulation strategies are adaptively given in different intervals. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a control flow chart of the present invention.

[0041] Figure 2It is the interval division of SOC.

[0042] Figure 3 It is the interval division of ACE.

[0043] Figure 4 It is a two-region integrated energy system model.

[0044] Figure 5 When X=0.01 and Y=15, P essm -SOC curve.

[0045] Figure 6 It is the FM effect of DDPG and TD3-UCB.

[0046] Figure 7 It is the change of energy storage SOC range. Specific implementation methods

[0048] Implementation example:

[0049] Based on the two areas of the IEEE standard, wind power, thermal power, diesel power generation, and battery energy storage were added to establish a Figure 4 The model is a two-region integrated energy system model of region one. Region one and region two respectively add thermal power, diesel, wind power and battery energy storage, and are controlled using deep reinforcement learning TD3-UCB algorithm.

[0050] Before the agent learns online, the system needs to undergo a lot of pre-learning training to optimize the Q function and action set. In the pre-learning phase, a sinusoidal load disturbance (period of 1200s and amplitude of 1000MW) is input to the multi-regional power grid. During the agent exploration process, the initial Q value is 0. After about 350 steps of iterative training, the Q value corresponding to the optimal state action is explored.

[0051] FM effect frequency index:

[0052] (1)V r is the frequency recovery speed;

[0053] (2)Δf m is the maximum frequency deviation;

[0054] (3)Δf a is the root mean square of the frequency deviation;

[0055] (4)Δf s is the steady-state frequency deviation.

[0056]

[0057] The frequency modulation effect of DDPG and TD3-UCB is shown in the figure below. Figure 6 As shown, the energy storage SOC interval changes as follows Figure 7shown.

[0058] The TD3-UCB joint algorithm proposed in the present invention has stronger self-learning ability and stable control results in a multi-regional power system containing energy storage, compared with the traditional fixed-ratio power allocation method and the classical reinforcement learning algorithm. It can respond well to the power allocation strategy under different load disturbances, making the secondary frequency regulation of the energy storage participating in the system faster and more accurate, thus realizing the intelligence of the AGC system.

Claims

1. A load frequency secondary regulation method based on deep reinforcement learning (TD3-UCB), characterized in that: The following steps are involved: S1: Construct a multi-region frequency control model with distributed energy storage; S2: Create an agent for TD3-UCB joint learning, build a network and initialize it; S3: Select the key variables in the power grid environment such as frequency deviation and energy storage SOC as state variables S k ; S4: According to the state variable S k and the policy π in the agent * , select the next action a k , and obtain the total frequency regulation instruction P of thermal power and energy storage ref ; S5: The reward r obtained from the system feedback according to the UCB strategy k , next state S k+1 ; S6: The final output total power instruction P is obtained through continuous updating and iteration ref , and then distribute the joint frequency modulation command to the energy storage P according to the allocation strategy in the intelligent agent bess and thermal power P u ; S7: According to the changes in the interconnection line power and energy storage SOC, an energy storage adaptive smoothing regulation strategy is designed to reduce the power oscillation output of the energy storage device.

2. The load frequency secondary regulation method based on TD3-UCB according to claim 1 is characterized in that: In step S1, the multi-region power system with distributed energy storage includes a speed governor, a reheater, a steam turbine, distributed energy storage, and a load. The model of the energy storage system adopts a distributed energy storage model, which is mainly composed of multiple sets of simplified energy storage unit models. Each set of energy storage models includes the transfer function of power output response and energy storage state. When simulating the grid frequency, the energy storage model can be simplified to a first-order inertia link.

3. The load frequency secondary regulation method based on TD3-UCB according to claim 1 is characterized in that: In step S3, the TD3 algorithm includes an Actor strategy network and a Critic evaluation network. The Actor network takes the state of the power grid as input and outputs actions according to network parameters, and the Critic network is used to judge the quality of the output actions of the Actor network. The UCB strategy is integrated into the deep reinforcement learning TD3 algorithm. The historical information of power grid data is fully utilized when selecting the next action of each intelligent agent. The upper bound of the interval is selected as the value of the action value function of each power frequency modulation based on the optimistic principle to make the optimal action decision, thereby improving the stability of the algorithm.

4. The load frequency secondary regulation method based on TD3-UCB according to claim 1 is characterized in that: In step S4, TD3-UCB selects the system frequency deviation Δf, the integral of the frequency deviation ∫Δf, the energy storage charge state SOC, the tie line power deviation ΔP tie As a state vector, the state space looks like this: Based on LSTM, the millisecond-level time series variable trend extraction and prediction of the power grid state are carried out to obtain the spatiotemporal feature output vector of transient power angle stability situation awareness, which can be used by the intelligent agent to intelligently allocate strategies. k , the agent generates an update strategy π * .

5. The load frequency secondary regulation method based on TD3-UCB according to claim 1 is characterized in that: In step S2, the UCB strategy selects actions based on the average reward value generated during the exploration process and the corresponding confidence level, which can eliminate the influence of parameter initialization to a certain extent. The intelligent system will generate several indicators, among which the expression of the kth indicator value is shown in the following formula: in, is the average reward value of the system frequency deviation value of the k-th action; The confidence level of the thermal storage frequency modulation power allocation action; n k is the number of times the action is selected; n is the number of times all actions are currently selected. The UCB strategy will end up in the state S k Next, select the final action a k , that is, the intelligent agent outputs the thermal power joint frequency regulation instruction. In the kth iteration, there is an error between the estimated Q value and the true Q value, and the error can be limited by the Hodinf inequality: Q k+1 (s k ,a k )-Q k (s k ,a k )≤b τ Among them, b τ is the confidence level, which indicates the degree of certainty of the algorithm for the current state-action pair, i.e. Among them, c is a constant greater than 0, s is the selected multi-regional power grid state vector, p is the frequency modulation power generated by the agent, k is the current number of iterations, q is the confidence factor, and τ is the number of visits to the state-action pair by the agent. According to the TD3-UCB joint algorithm, the next frequency modulation action a is obtained k+1 .

6. The load frequency secondary regulation method based on TD3-UCB according to claim 1 is characterized in that: In the step S5, the experience playback mechanism is adopted to comprehensively consider the action index value of the confidence interval corresponding to the evaluation reward value of the tie line power and the frequency deviation reward value of the thermal power unit in the intelligent agent exploration process to select the action, make full use of historical information, and obtain the optimal action value, that is, the frequency adjustment instruction, after a certain number of iterations, and finally interact with the power grid to obtain the next state S of the power grid. k+1 The reward function that comprehensively considers frequency and energy storage charge state is: Among them, η is the weight coefficient, which is 0.5 here.

7. The load frequency secondary regulation method based on TD3-UCB according to claim 1 is characterized in that: In step S7, when the system receives the frequency modulation instruction from ACE, it obtains the output of the energy storage unit based on the energy storage status and the energy storage power allocation strategy updated by the intelligent agent. The intelligent agent combines the SOC of the distributed energy storage battery and other parameters of itself to construct the output instruction of the energy storage system to achieve rapid power control and regulation. The SOC of energy storage can be divided into different state areas, SOC up With SOC down They represent the maximum SOC state and minimum SOC state of energy storage discharge; SOC 0 With SOC 1 Respectively represent the low and high middle values ​​of SOC; SOC low With sOC high Represents the lower and higher SOC values ​​respectively. The intelligent agent divides each area according to the parameters of the energy storage system. zone Limit the frequency modulation power output by the controller to achieve smooth adjustment of adaptive control and reduce energy storage oscillation. According to the energy storage regulation strategy in the intelligent agent, construct a reasonable P essm -SOC curve to ensure smooth energy storage action. Introduce the Logistic regression function, take the energy storage SOC as the independent variable, and the adaptive factors X and Y as the dependent variables to construct the maximum output limit of energy storage based on SOC. The maximum charge and discharge power P of energy storage c , P d for: When X=0.01 and Y=15, the energy storage can be ensured to work in a safe area and the action of the energy storage can be ensured to be smooth without sudden changes. in, P essm is the maximum output power of energy storage; P dm With P cm Respectively represent the rated discharge and charging power of the energy storage. When the energy storage absorbs excess power according to actual working conditions, the charging strategy of the energy storage is as follows: P c ′=|P u -P ref -P zone | Among them, P u is the actual frequency modulation power of the thermal power unit; P bess is the actual frequency modulation power of energy storage; P ref is the frequency modulation instruction of the intelligent agent; P zone P is the dead zone size of the power system’s dispatch frequency regulation instruction; c ′ is the actual frequency regulation demand of the energy storage system; P c is the maximum charging power limit of the energy storage mentioned above. When the energy storage generates power according to the actual working conditions, the discharge strategy of the energy storage is as follows: P d ′=|P u -P ref -P zone | Among them, P d ′ is the actual frequency regulation and discharge demand of the energy storage system; P d is the maximum discharge power limit of the energy storage mentioned above. Based on the output conditions and grid environment, the intelligent agent predicts whether the SOC is in a state of single cell overcharge or over discharge, and then formulates control instructions for the thermal power unit based on the prediction results and ACE instructions, and outputs the power adjustment instructions to the thermal power unit and the energy storage system respectively. Through the coordination between multiple intelligent agents of the TD3-UCB joint algorithm, the frequency regulation power of each regional energy storage and thermal power unit is obtained, so that the final frequency deviation of the entire power system is reduced to 0.

Images