Power grid frequency cooperative control method and system

Abstract

The invention discloses a power grid frequency cooperative control method and system. The method comprises the steps that a first target model is established, wherein the first target model at least comprises one or more sets of traditional units, wind turbine units or energy storage systems; establishing a target function and a constraint condition based on a first improved algorithm controller; and solving the first target model according to the target function and the constraint condition. According to the method, the advantages of reinforcement learning can be utilized, the control strategy is continuously optimized through interactive learning of the intelligent agent and the environment, and the adaptability and robustness of the power grid are improved. By setting the environment and the intelligent agent based on the Markov decision process, the actual power grid operation condition can be simulated, the intelligent agent can effectively deal with various load disturbances through learning and decision making, and stable operation of the power grid is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of power grid frequency coordinated control, and in particular to a power grid frequency coordinated control method and system. Background Art

[0002] Renewable energy sources such as wind power and photovoltaics have developed rapidly, and their share in the power system has continued to increase. However, the large-scale grid connection of renewable energy will reduce the inertia level of the power system, weaken the frequency regulation capability of the system, and increase the uncertainty of the system, which brings great challenges to the frequency safety and stability of the power system. In order to meet these challenges and ensure the frequency stability of the new power system, it is urgent to research and develop new frequency control methods.

[0003] In the load frequency control (LFC) of power systems, control methods can be divided into model-based methods and data-driven methods. Among the model-based methods, proportional integral differential (PID) control is widely used. However, the tuning of PID parameters mostly relies on experience and repeated debugging, which makes it difficult to achieve the optimal control effect. In response to the problem of PID parameter tuning, some scholars have proposed to use optimization algorithms such as genetic algorithm, particle swarm algorithm, and gray wolf algorithm to optimize PID parameters to improve the control effect. However, these schemes are parameters obtained for specific load disturbances, and the influence of random load disturbances is not considered. With the development of reinforcement learning, some scholars have proposed to apply reinforcement learning to the field of load frequency control. As a data-driven control method, reinforcement learning can continuously optimize control strategies through continuous interaction and learning with the environment. It does not rely on accurate system models and has better adaptability. It can directly apply learned strategies without complex model prediction or optimization calculations. However, the traditional deep reinforcement learning algorithm has the problem of overestimation of Q value, and the obtained strategy may not be the optimal strategy. The DRL algorithm needs to continuously try and error through random strategies in the early stage of training, resulting in a slow convergence of algorithm training. Summary of the invention

[0004] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a method and system for coordinated control of power grid frequency, which can solve the problems mentioned in the background technology.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a method for coordinated control of power grid frequency, comprising:

[0009] Establishing a first target model, wherein the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems;

[0010] Establishing the objective function and constraint conditions based on the first improved algorithm controller;

[0011] The first target model is solved according to the target function and the constraint conditions.

[0012] As a preferred solution of the grid frequency coordinated control method described in the present invention, the first target model includes: the first target model is an arbitrary model whose input is a traditional unit control signal and whose output is a frequency deviation.

[0013] As a preferred solution of the grid frequency coordinated control method described in the present invention, the objective function includes:

[0014] The objective function is an arbitrary function that, when disturbed, the frequency deviation of the system is restored to 0 within a target time period, and the unit output fluctuation is less than a first fluctuation threshold.

[0015] As a preferred solution of the grid frequency coordinated control method of the present invention, the solving of the first target model according to the objective function and the constraint condition includes:

[0016] Converting the first target model into a Markov decision process according to the target function and the constraint conditions;

[0017] The objective function and the constraints are used as the state space, action space and reward function of the Markov decision process, wherein the objective function is the reward function of the Markov decision process, and the constraints are the state space and action space of the Markov decision process.

[0018] As a preferred solution of the grid frequency coordinated control method of the present invention, wherein: solving the first target model according to the objective function and the constraint conditions further includes:

[0019] Setting up an environment and an agent based on the Markov decision process;

[0020] The environment of the Markov decision process is the first target model;

[0021] The intelligent agent of the Markov decision process is a frequency controller.

[0022] As a preferred solution of the grid frequency coordinated control method described in the present invention, the state space of the Markov decision process includes: frequency deviation, integral value of frequency deviation, differential value of frequency deviation and the action value of the previous intelligent agent.

[0023] As a preferred solution of the grid frequency coordinated control method described in the present invention, the action space includes: the control signal of the traditional unit.

[0024] In a second aspect, the present invention provides a power grid frequency coordinated control system, comprising:

[0025] A model building module, used to build a first target model, wherein the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems;

[0026] An improved algorithm module, used to establish an objective function and constraint conditions based on a first improved algorithm controller;

[0027] A solution module is used to solve the first target model according to the objective function and the constraint conditions.

[0028] In a third aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method when executing the computer program.

[0029] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method described above when executed by a processor.

[0030] Compared with the prior art, the present invention has the following beneficial effects: the present invention proposes a method and system for coordinated control of power grid frequency, establishes a first target model, and the first target model includes at least one or more groups of traditional units, wind turbines or energy storage systems; establishes an objective function and constraints based on a first improved algorithm controller; solves the first target model according to the objective function and constraints. By establishing the first target model, the present application can comprehensively consider the coordinated work of various energy forms such as traditional units, wind turbines and energy storage systems, thereby achieving more efficient frequency control. Using the first improved algorithm controller, the present invention can optimize the objective function and constraints to achieve rapid recovery of the frequency deviation to 0 when a disturbance occurs, and limit the unit output fluctuation within an acceptable range. By converting the target model into a Markov decision process, the present application can take advantage of reinforcement learning, continuously optimize the control strategy through interactive learning between the intelligent agent and the environment, and improve the adaptability and robustness of the power grid. The solution module of the present application can automatically adjust the control signal of the traditional unit according to the objective function and constraints to achieve precise control of the power grid frequency. By setting up an environment and intelligent agent based on the Markov decision process, this application can simulate the actual operation of the power grid. Through learning and decision-making, the intelligent agent can effectively respond to various load disturbances and ensure the stable operation of the power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0032] Figure 1 A method flow chart of a method and system for coordinated control of power grid frequency provided by an embodiment of the present invention;

[0033] Figure 2 A single-region power system frequency response model of a power grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0034] Figure 3 A schematic diagram of a GRC model of a thermal power unit of a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0035] Figure 4 A schematic diagram of a GRC model of a hydropower unit of a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0036] Figure 5A TD3 algorithm schematic diagram of a power grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0037] Figure 6 A flowchart of the IL-TD3 algorithm of a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0038] Figure 7 A schematic diagram of an average reward convergence process of a power grid frequency cooperative control method and system provided by an embodiment of the present invention;

[0039] Figure 8 A schematic diagram of frequency deviations of different controllers under random step disturbances of a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0040] Fig. 9 A schematic diagram of renewable energy power variation of a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0041] Fig.10 A schematic diagram of frequency deviations of different controllers under renewable energy disturbances in a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0042] Fig.11 A schematic diagram of a sinusoidal disturbance of a load of a power grid frequency coordinated control method and a system provided by an embodiment of the present invention;

[0043] Fig.12 A schematic diagram of frequency deviations of different controllers under sinusoidal disturbances of a grid frequency coordinated control method and system provided by an embodiment of the present invention;

[0044] Fig.13 An internal structural diagram of a computer device of a power grid frequency cooperative control method and system provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0047] Example 1

[0048] In the existing related technologies, there are some shortcomings in the control strategy, such as slow response speed, low control accuracy and other problems. This application provides a method that can effectively solve the above-mentioned problems. Next, we will combine multiple embodiments to explain in detail how to implement the grid frequency coordinated control method;

[0049] In the existing related technologies, there are some shortcomings in the control strategy, such as slow response speed, low control accuracy and other problems. This application provides a method that can effectively solve the above-mentioned problems. Next, we will combine multiple embodiments to explain in detail how to implement the grid frequency coordinated control method;

[0050] Figure 1 A method flow chart of a method and system for coordinated control of power grid frequency is shown, including:

[0051] S101, establishing a first target model, where the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems;

[0052] In the embodiment of the present application, the first target model is an arbitrary model whose input is a conventional unit control signal and whose output is a frequency deviation.

[0053] In an optional embodiment, there are many ways to establish the first target model, for example, a physical rule-based modeling method or a data-driven modeling method can be used. In the physical rule-based modeling method, the physical characteristics of various devices in the power system, such as the rotor dynamic characteristics of the generator, the static characteristics of the load, etc., can be used to build the model. In the data-driven modeling method, historical operating data can be used to identify and model the behavior of the system through machine learning algorithms. In addition, the first target model can also combine the advantages of these two methods to form a hybrid model to improve the accuracy and generalization ability of the model.

[0054] In the embodiment of the present application, the following is established: Figure 2 The single-region power system frequency response model shown is used as the first target model, wherein the first target model has a conventional unit control signal as input and a frequency deviation as output. The structure of the single-region power system frequency response model constructed in this application is as follows: Figure 2 As shown, R1, R2, and R3 are the primary frequency regulation coefficients of thermal power units, hydropower units, and gas units, respectively. Δu h , Δu s , Δu q They are the control signals of thermal power unit, hydropower unit and gas unit respectively. ΔP L is the load disturbance. K p is the power system gain, T p k is the power system time constant. df is the virtual control coefficient, kpf is the droop control coefficient. Δf is the frequency deviation.

[0055] In an optional embodiment, the first target model includes at least one or several groups of traditional units, wind turbines or energy storage systems. When there are only traditional units in the power system, only the number of traditional units needs to be considered. Traditional units may include thermal power units, hydropower units and gas units. When there are wind turbines or energy storage systems in the power system at the same time, the first target model needs to consider the dynamic characteristics of these new energy sources to ensure the accuracy of frequency control. For example, the output power of wind turbines is greatly affected by wind speed, so the impact of wind speed changes on frequency needs to be considered when establishing the model. The energy storage system can be used as a means of rapid adjustment to assist traditional units in maintaining system frequency stability through its charging and discharging behavior.

[0056] In the embodiment of the present application, the traditional units selected are thermal power units, hydropower units and gas units, and wind power units or energy storage systems are also considered, so the following is established: Figure 2 The first target model (i.e., single-region power system frequency response model) is shown.

[0057] It should be noted that no matter how many specific traditional units, wind turbines or energy storage systems there are, they can be controlled as a whole. Therefore, in this application, there is no detailed division of units of different numbers, and the overall structure is considered when establishing the first target model.

[0058] In an optional embodiment, different units are of different types. For example, when selecting a thermal power unit, you can select a thermal power unit without reheat or a thermal power unit with reheat. The hydropower unit can be a pumped storage unit or a conventional hydropower unit. The gas unit can be a gas turbine or a combined cycle unit. Each type of unit has its own specific dynamic response characteristics, which need to be taken into account when establishing the first target model. For example, the response speed of a thermal power unit with reheat is usually slower than that of a thermal power unit without reheat because the reheating process adds additional delay. Pumped storage units can provide fast power regulation capabilities, which are a valuable resource for frequency control.

[0059] In the embodiment of the present application, the present application adopts a non-reheat thermal power unit, and its frequency response model is usually composed of a governor model and a steam turbine model. There are mechanical friction and gaps in the hydraulic device mounted on the governor, which will cause the governor to not respond within a specific frequency deviation. This phenomenon is called the frequency regulation dead zone (GDB). The frequency regulation dead zone will increase the frequency fluctuation and reduce the stability of the power system, so it should not be ignored during modeling. The governor transfer function considering the frequency regulation dead zone is:

[0060]

[0061] Among them, T g is the time constant of the thermal power unit speed regulator, s is the Laplace operator, N1 and N2 are Fourier coefficients, and for a gap of 0.5%, N1=0.8 and N2=-0.2 / π.

[0062] It should be noted that the steam turbine adjusts the output mechanical power by changing the valve position. Since the change of mechanical power is usually slower than the change of valve opening, it is often represented by a first-order inertia link. However, due to the physical limitations of the turbine, the rate of change of generated power is also limited. This phenomenon is called the generation rate constraint (GRC). The nonlinear characteristics of the generation rate constraint have a significant impact on the power system, so it should be considered when modeling. This application adopts a closed-loop model of GRC. The model is as follows Figure 3 As shown in the figure, T t is the turbine time constant, δ is the generator rate constraint, and this application considers a constraint of 10% / min (0.0017 pu / s).

[0063] In the embodiment of the present application, the frequency response model of the hydroelectric generator set is composed of a speed governor and a turbine. Since the non-minimum phase characteristic of the turbine will cause system instability, a transient droop compensator is added to the speed governor to improve system stability. The transfer function of the speed governor is:

[0064]

[0065] Among them, T gh is the time constant of the hydropower unit speed regulator.

[0066] Furthermore, the transfer function of the transient droop compensator is:

[0067]

[0068] Among them, T rs , T rh is the transient droop compensator time constant.

[0069] Furthermore, the turbine transfer function is:

[0070]

[0071] Among them, T hy is the turbine time constant.

[0072] This application considers the GRC impact of hydropower units and adopts the GRC open-loop model, such as Figure 4 As shown in the figure, δ h and δ lThese are the upper and lower limits of GRC respectively. This application considers an upper limit of 270% / min (0.045 pu / s) and a lower limit of 360% / min (0.06 pu / s).

[0073] In the embodiment of the present application, the frequency response model of the gas generator set is composed of valve position, gas speed governor, combustion system and compressor exhaust system. The transfer function of the valve position is:

[0074]

[0075] Among them, B and C are the valve position constants of the gas generator set.

[0076] Furthermore, the gas governor transfer function is:

[0077]

[0078] Among them, X and Y are the gas generator set speed governor constants.

[0079] Furthermore, the combustion system transfer function is:

[0080]

[0081] Among them, T cr 、T f is the combustion system time constant.

[0082] Furthermore, the compressor exhaust system transfer function is:

[0083]

[0084] Among them, T cd It is the time constant of the gas generator set compressor exhaust system.

[0085] In an embodiment of the present application, the wind turbine can utilize the rotor kinetic energy to participate in a frequency modulation. When the wind turbine is operating normally, a certain amount of kinetic energy is stored on its rotor, which can be released to support the frequency modulation. In the present application, the wind turbine group uses the rotor kinetic energy to quickly respond to frequency changes by adding a virtual inertia control module. The wind turbine model of the present application is represented by a simplified first-order lag transfer function, which includes a frequency detection model and a wind turbine generator model. Among them, the transfer function of the frequency detection model is:

[0086]

[0087] Among them, T r is the frequency detection time constant.

[0088] Furthermore, the transfer function of the wind turbine is:

[0089]

[0090] Among them, T W is the wind turbine generator time constant.

[0091] Furthermore, the reference power for wind turbine frequency regulation is:

[0092]

[0093] Among them, k df is the virtual control coefficient.

[0094] In the embodiment of the present application, the present application adopts a power-type energy storage system, which has the advantages of fast response speed, high control accuracy and bidirectional regulation. The large-scale access of new energy sources reduces the inertia and primary frequency regulation capability of the power system, and the energy storage system can participate in the primary frequency regulation of the power system and enhance the primary frequency regulation capability of the system. Therefore, in the present application, the energy storage system participates in the primary frequency regulation through droop control. The energy storage system model can be represented by a simplified first-order transfer function, and the frequency response model of the energy storage is:

[0095]

[0096] Among them, T b is the energy storage time constant.

[0097] The reference power of the energy storage system frequency regulation is:

[0098] ΔP b =-k pf Δf

[0099] Among them, k pf is the droop control coefficient.

[0100] It should be noted that the first target model is established, and the first target model includes at least one or several groups of traditional units, wind turbines or energy storage systems that can simulate the dynamic response of the power grid under different operating conditions. The establishment of the first target model is based on an in-depth analysis of the stability of the power grid frequency, which can reflect the changing trend of the power grid frequency under conditions such as new energy access and load fluctuations. Through the first target model, the dynamic behavior of the power grid frequency can be predicted and evaluated, thereby providing a theoretical basis and practical operation guidance for the coordinated control of the power grid frequency. In addition, the first target model can also help power grid operators identify potential frequency deviation problems and take measures to make adjustments in advance to ensure the stability and reliability of power grid operation.

[0101] S102, establishing an objective function and constraint conditions based on the first improved algorithm controller;

[0102] In an embodiment of the present application, the objective function includes: the objective function is an arbitrary function that, when disturbed, the frequency deviation of the system returns to 0 within a target time period, and the unit output fluctuation is less than a first fluctuation threshold.

[0103] In an optional embodiment, the objective function can be established by different methods, for example, linear programming, nonlinear programming or mixed integer programming. However, these methods have their own advantages and disadvantages, so it is necessary to select a suitable method according to the characteristics of the actual power grid. For example, the linear programming method has a fast calculation speed and is suitable for linear systems, but it may not accurately describe the dynamic behavior of nonlinear systems; the nonlinear programming method can handle more complex system models, but the calculation amount is large and the solution time is long; the mixed integer programming method combines the advantages of linear and nonlinear programming and is suitable for systems containing discrete variables, but it also has the problem of high computational complexity.

[0104] In an optional embodiment, the objective function can also be established by a method other than linear programming, for example, a two-level programming method is used, which divides the problem into two layers, the upper layer is responsible for the optimization of decision variables, and the lower layer handles the constraints. The two-level programming method can consider the stability and economy of the system at the same time, and is suitable for complex systems that need to meet multiple goals at the same time.

[0105] In an optional embodiment, a multi-objective optimization method can be used, which can simultaneously consider multiple objective functions, such as minimizing frequency deviation and minimizing unit output fluctuation, etc. Through multi-objective optimization, an optimal solution set that meets all objectives, namely, the Pareto frontier, can be found, thereby providing a series of feasible control strategies for power grid operators.

[0106] In the embodiment of the present application, the objective function and constraint conditions based on the first improved algorithm controller are selected;

[0107] In an optional embodiment, the objective functions and constraints established by different means need to satisfy the objective function that when disturbed, the frequency deviation of the system recovers to 0 within the target time period, and the unit output fluctuation is less than the first fluctuation threshold. In other words, when relevant technical personnel design any function that satisfies the objective function that when disturbed, the frequency deviation of the system recovers to 0 within the target time period, and the unit output fluctuation is less than the first fluctuation threshold, it should be within the scope of protection of the present application.

[0108] It should be noted that choosing objective functions and constraints established in different ways will lead to changes in the subsequent solution steps. For example, when the two-level programming method is selected, the subsequent solution method can be the second-order cone optimization algorithm, while when the multi-objective optimization method is adopted, the genetic algorithm or the particle swarm optimization algorithm can be used.

[0109] In the embodiment of the present application, the smaller the target time period is, the better. The design hopes that the frequency deviation of the system will be restored to 0 in the shortest time. This application uses 0.1s as the target time period. Relevant technical personnel can adjust it according to actual needs. No numerical limitation is made here.

[0110] In the embodiment of the present application, the smaller the first fluctuation threshold is, the better. In the design, it is hoped that the frequency regulation cost of the system will be taken into consideration and the unit output fluctuation will be smaller. Therefore, a minimum function can be designed at this position or a specific numerical threshold can be given, which is not limited here.

[0111] In the embodiment of the present application, the first improved algorithm is an improved algorithm IL-TD3 based on the DDPG algorithm.

[0112] It should be noted that DDPG is a deep reinforcement learning algorithm that combines policy gradient and Q learning. It can handle the problem of multi-dimensional continuous action control, greatly expanding the application scope of reinforcement learning. In addition, DDPG's deterministic strategy can output actions based only on the state of the environment, and can be learned without a detailed mathematical model.

[0113] Furthermore, DDPG's strategy function and value function both use neural networks. The Actor network outputs actions based on the state s, and the Critic network evaluates the value of the current action. The DDPG algorithm estimates the value of the current action through the Bellman equation.

[0114] y i = r(s i ,a i )+γQ θ' (s i+1 ,μ φ' (s i+1 ))

[0115] Among them, y i ,s i , a i ,r(s i ,a i ) are the estimated value, state, action and reward at the current moment, where a i =μ φ (s i ), γ is the discount factor, Q θ' (s i+1 ,μ φ' (s i+1 )),s i+1 , μ φ' (s i+1), which are the Q value calculated by the target Critic network at the next moment, the state at the next moment, and the action calculated by the target Actor network at the next moment.

[0116] Furthermore, the Critic network approximates the Q function by optimizing the neural network parameters. Its optimization goal is to minimize the loss function and update it using the gradient descent algorithm:

[0117]

[0118] Among them, N is the minimum batch of sampling, Q θ (s i ,a i ) is the Q value output by the main critic network based on the current state and action.

[0119] Furthermore, the optimization goal of the Actor network is to maximize the action value Q, which is updated using the gradient ascent algorithm:

[0120]

[0121] Among them, μ φ (s i ) is the action value output by the main Actor network based on the current state.

[0122] It should be noted that in order to increase the stability of the learning process, DDPG uses the target network to estimate the value of the next action. The target network has the same network structure as the main network, but its neural network parameters are slowly approached to the main network parameters through soft updates:

[0123] θ′=τθ+(1-τ)θ′

[0124] φ′=τφ+(1-τ)φ′

[0125] Among them, θ′, φ′ are the parameters of the target Critic network and the target Actor network respectively; θ, φ are the parameters of the main Critic network and the main Actor network respectively.

[0126] It should be noted that the DDPG algorithm's Critic network tends to overestimate actions, resulting in the strategy failing to converge to the optimal state. In order to overcome this problem and improve algorithm performance, the TD3 algorithm has been improved on the basis of the DDPG algorithm, specifically in the following three aspects:

[0127] (1) Truncated dual Q learning. The TD3 algorithm simultaneously learns two Q functions by minimizing the mean square error. The two Q functions use the same objective and select the smaller value of the two Q functions as the target Q value. The principle is as follows: Figure 5 As shown, the changed formula is:

[0128]

[0129] (2) Delayed policy updates. Experimental results show that synchronous training of the Actor network and the Critic network will lead to an unstable training process. Therefore, the TD3 algorithm updates the Actor network at a lower frequency and the Critic network at a higher frequency. Usually, the Actor network is updated once every two Critic networks are updated.

[0130] (3) Target strategy smoothing. TD3 adds noise to the target action and smoothes the change of Q value along the action, making it more difficult for the strategy to use the error of Q function to improve performance. Its working principle is:

[0131] a(s')=μ(s')+ε

[0132] Among them, ε is sampled from a normal distribution, that is, ε~N(0,σ).

[0133] It should also be noted that in the initial stage of training, the agent learns from scratch and needs to explore new actions in the action space, but these new actions may not necessarily get more rewards. The agent needs to accumulate experience through continuous "trial and error" to get more rewards, which causes reinforcement learning to take a long time to converge in the initial stage. Therefore, in order to reduce the time for training convergence, this application combines imitation learning with the TD3 algorithm.

[0134] Imitation learning is a method that allows an intelligent agent to learn from expert examples so that it can make decisions like an expert. The most common imitation learning frameworks are behavioral cloning (BC) and inverse reinforcement learning (IRL). BC uses expert data to train neural networks. Expert data is a collection of state-action pairs. During training, the state is used as input and the action is used as a label for supervised training to form a mapping from state to action. ILR, on the other hand, learns a reward function through expert data, and then uses the reward function to guide the agent to learn strategies. Compared with IRL, BC is simpler and more direct, and it is easier to learn expert strategies. Therefore, the imitation learning of this application is implemented through BC. The IL-TD3 algorithm proposed in this application is divided into three stages: pre-learning, offline training, and online application. Its process is as follows Figure 6As shown. In the pre-learning stage, the agent performs pre-learning through imitation learning. This application uses the optimized PID controller as an expert to collect data on the interaction between PID and the power system. The collected expert data is used to train the neural network so that it imitates the expert's actions, and the trained neural network is used as the Actor network in the TD3 agent. This can reduce unnecessary action exploration of the agent during the training process and help the Actor network learn effective strategies faster. In the offline training stage, the pre-learned agent continuously interacts with the power system environment. In each training round, the agent will receive the state information s of the environment, and the Actor network will output action a to the environment based on s. These interaction data are placed in the experience buffer. After a certain amount of data is stored, a certain batch of data is sampled from the buffer and the parameters of the Actor network and the Critic network are updated by the gradient descent algorithm and the gradient ascent algorithm. In the online application stage, the trained agent can generate control signals for each unit based only on the observed state of the power system.

[0135] In the embodiment of the present application, the frequency control method based on the first improved algorithm (IL-TD3 algorithm in the present application) transforms the power system frequency control problem into a Markov decision problem, takes the power system frequency control model as the environment of reinforcement learning, and the intelligent agent as the frequency controller, and learns the optimal strategy through the continuous interaction between the intelligent agent and the environment to achieve the goal of frequency control. The state space, action space (the state space and action space are the constraints of the present application) and the reward function (that is, the objective function of the present application) are the core components of the Markov decision process. These elements define the way the intelligent agent interacts with the environment and how to learn the optimal strategy based on these interactions. The intelligent agent selects actions according to its strategy in each state, and then the environment transfers to a new state according to these actions and gives corresponding immediate rewards. In this way, the intelligent agent gradually learns which actions to choose in a given state can obtain the greatest long-term return. Therefore, the reasonable design of the state space, action space and reward function is crucial to the development of an effective frequency control strategy.

[0136] It should be noted that the objective function and constraints based on the first improved algorithm controller are set based on the actual operation of the power system and the frequency control requirements. The state space includes key parameters such as the frequency deviation, frequency change rate, and output power of the generator set of the power system, which can fully reflect the operating status of the system. The action space defines the control actions that the agent can take, such as adjusting the output power of the generator set in response to the frequency deviation. The reward function is designed to guide the agent to take actions that can stabilize the system frequency and approach the target frequency, while taking into account the stability and economy of the system. Through such a design, the agent can continuously optimize its strategy during the learning process to achieve the best frequency control effect.

[0137] S103, solving the first objective model according to the objective function and the constraint conditions.

[0138] In an embodiment of the present application, solving the first target model according to the target function and the constraint conditions includes:

[0139] The first objective model is transformed into a Markov decision process according to the objective function and the constraints;

[0140] The objective function and constraints are taken as the state space, action space and reward function of the Markov decision process. The objective function is the reward function of the Markov decision process, and the constraints are the state space and action space of the Markov decision process.

[0141] In the embodiment of the present application, solving the first target model according to the objective function and the constraint conditions further includes:

[0142] Set up an environment and agent based on a Markov decision process;

[0143] The environment of the Markov decision process is the first objective model;

[0144] The agent of the Markov decision process is the frequency controller.

[0145] In the embodiment of the present application, the state space of the Markov decision process includes: frequency deviation, integral value of frequency deviation, differential value of frequency deviation and action value of the last intelligent agent. Action space includes: control signal of the traditional unit.

[0146] Specifically, in the frequency control model environment, the observable state quantities include frequency deviation, the action of the agent, the output value of each unit, etc. The state space selected in this application is the frequency deviation, the integral value of the frequency deviation, the differential value of the frequency deviation, and the action value of the last agent, that is:

[0147]

[0148] After receiving the state information, the agent will select an action from the action space according to its own strategy. This application sets the actions of the agent to be the control signals of thermal power, hydropower and gas units:

[0149] a i =[Δu h-i ,Δu s-i ,Δu q-i ]

[0150] The reward function plays a role in guiding the learning and decision-making of the intelligent agent in reinforcement learning. Reasonable reward function design can help the intelligent agent learn better strategies. In this application, the main control goal is to restore the frequency deviation of the system to 0 as quickly as possible when the system is disturbed. At the same time, considering the frequency regulation cost of the system, it is hoped that the unit output fluctuation will be smaller. Therefore, the reward function of this application takes into account the frequency deviation Δf and the power change of each unit, that is:

[0151] r i =-λ1Δf i 2 -λ2|Δf i |-λ3|δP h-i |-λ4|δP s-i |-λ5|δP q-i |

[0152] Among them, λ1, λ2, λ3, λ4, and λ5 are weight coefficients. |δP h-i |、|δP s-i |、|δP q-i | are the power changes of thermal power, hydropower and gas units respectively.

[0153] In summary, the present invention proposes a method for coordinated control of power grid frequency, establishes a first target model, and the first target model includes at least one or several groups of traditional units, wind turbines or energy storage systems; establishes an objective function and constraints based on a first improved algorithm controller; and solves the first target model according to the objective function and constraints. By establishing the first target model, the present application can comprehensively consider the coordinated work of various energy forms such as traditional units, wind turbines and energy storage systems, so as to achieve more efficient frequency control. Using the first improved algorithm controller, the present invention can optimize the objective function and constraints to achieve rapid recovery of the frequency deviation to 0 when a disturbance occurs, and limit the unit output fluctuation within an acceptable range. By converting the target model into a Markov decision process, the present application can take advantage of reinforcement learning, continuously optimize the control strategy through interactive learning between the intelligent agent and the environment, and improve the adaptability and robustness of the power grid. The solution module of the present application can automatically adjust the control signal of the traditional unit according to the objective function and constraints to achieve precise control of the power grid frequency. By setting up an environment and intelligent agent based on the Markov decision process, this application can simulate the actual operation of the power grid. Through learning and decision-making, the intelligent agent can effectively respond to various load disturbances and ensure the stable operation of the power grid.

[0154] This embodiment also provides a power grid frequency coordinated control system, including:

[0155] A model building module, used to build a first target model, the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems;

[0156] An improved algorithm module, used to establish an objective function and constraint conditions based on a first improved algorithm controller;

[0157] The solution module is used to solve the first target model according to the target function and the constraint conditions.

[0158] The above-mentioned unit modules may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above-mentioned modules.

[0159] This embodiment also provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Fig.13As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for coordinated control of power grid frequency is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0160] This embodiment further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0161] Establishing a first target model, wherein the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems;

[0162] Establishing the objective function and constraint conditions based on the first improved algorithm controller;

[0163] The first objective model is solved according to the objective function and the constraints.

[0164] Example 2

[0165] Reference Figure 7-Figure 12 , which is an embodiment of the present invention, provides a grid frequency coordinated control method and system. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0166] In order to verify the control performance of the algorithm proposed in this application, simulation experiments were carried out using MATLAB software. The relevant parameters of the single-region power system model are referenced in Table 1. This application uses the PID and DDPG controllers optimized by the gradient-based optimization algorithm (Gradient-BasedOptimizer, GBO) as controls, where the sampling time of the controller is set to 1s, and the Actor network learning rate, Critic network learning rate, experience buffer size and discount factor of DDPG and IL-TD3 are set to 0.0001, 0.001, 1e6 and 0.99 respectively. The Actor network and the Critic network contain 2 hidden layers, with 32 nodes in each layer.

[0167] Table 1 System parameters

[0168]

[0169] In order to take into account the uncertainty of load disturbances in the real power system as much as possible, so that the trained intelligent agent can show good control effect in different disturbance scenarios, random disturbances are applied to the system at the beginning of each training round. In addition, this application also sets three disturbances: random disturbance, renewable energy disturbance and sinusoidal disturbance.

[0170] The control performance and robustness of the frequency control strategy proposed in this application are verified by using the following scenarios, and four performance indicators are set to evaluate the frequency regulation performance under various disturbance scenarios. The selected indicator is the average absolute frequency deviation. Integrated squared error (ISE), integrated absolute error (IAE), and integrated time absolute error (ITAE):

[0171]

[0172] In order to compare the convergence speed between different deep reinforcement learning algorithms, the average reward of each algorithm is recorded. When the average reward value reaches a stable state, the algorithm is considered to have converged. Figure 7 The changes in average rewards of the DDPG algorithm, TD3 algorithm, and the algorithm proposed in this application during training are shown. Among them, the DDPG algorithm and TD3 algorithm are trained from scratch with random initial neural network parameters, which leads to a long time to converge. They reach convergence after about 185 and 100 training rounds. However, the algorithm proposed in this application only needs about 60 rounds to converge because it has pre-learned the expert's knowledge, which greatly shortens the time for the algorithm to converge. In addition, since the problem of overestimation of the Q value of the DDPG algorithm has been solved, the algorithm proposed in this application can converge to a higher reward value and learn a better strategy.

[0173] This application applies random disturbances in the system to test the control effects of GBO optimized PID, DDPG and IL-TD3 proposed in this application in the system. The frequency deviation change curve of the test is shown in Figure 8 ,The comparison of different performance indicators is shown in Table 2.

[0174] from Figure 8 It can be seen that compared with the other two controllers, the IL-TD3 algorithm proposed in this application can effectively reduce the fluctuation of frequency deviation when facing random step disturbances, and can quickly restore the frequency deviation to a state close to 0. The control overshoot based on DDPG is large, which may be because it overestimates the Q value of a certain action, resulting in the learned strategy being suboptimal.

[0175] In terms of frequency modulation performance comparison, IL-TD3 is better than GBO optimized PID. Compared with DDPG, IL-TD3 decreased by about 38%, ISE decreased by about 29.84%, IAE decreased by about 36.07%, and ITAE decreased by about 31.39%. The frequency modulation performance of the proposed algorithm is better than that of the other two methods in the random step disturbance scenario, and a better frequency modulation effect can be achieved.

[0176] Renewable energy has strong randomness. In order to verify the frequency control performance of the algorithm proposed in this application in the scenario of renewable energy access, Fig. 9 The disturbance signal shown is used to simulate the sudden change of renewable energy.

[0177] Fig.10 The frequency deviation change curves of GBO optimized PID, DDPG and IL-TD3 under renewable energy disturbance scenarios are shown. It can be seen from the figure that compared with the other two controllers, the controller based on IL-TD3 reduces the overshoot of frequency deviation and has smaller fluctuations. The comparison of frequency regulation performance indicators is shown in Table 2. Compared with GBO optimized PID, the controller based on IL-TD3 has The ISE, IAE and ITAE indicators decreased by 63.04%, 80.25%, 59.89% and 61.22%, respectively.

[0178] Compared with the DDPG-based controller, the IL-TD3-based controller The ISE, IAE and ITAE indicators were reduced by 35.41%, 47.65%, 31.82% and 28.60% respectively. The results show that the frequency regulation effect of GBO optimized PID is the worst when facing sudden changes in renewable energy. This may be because the fixed parameter PID cannot adapt to the rapid changes of renewable energy.

[0179] Table 2 Performance indicators under different disturbance scenarios

[0180]

[0181] A sinusoidal disturbance with an amplitude of 0.035 (pu) and a period of 50s is applied to the system. The waveform of the disturbance signal is as follows: Fig.11 The test results are shown in Fig.12 and Table 2.

[0182] from Fig.12It can be seen from the frequency deviation change curves of the three controllers that the controller based on IL-TD3 effectively reduces the fluctuation amplitude of the frequency deviation. In the comparison of frequency modulation performance indicators, the control method proposed in this application has also achieved significant improvement. Compared with GBO optimized PID and DDPG, IL-TD3 has The indicators were reduced by 57.66% and 48.73%, the ISE indicators were reduced by 79.88% and 70.47%, the IAE indicators were reduced by 55.17% and 46.75%, and the ITAE indicators were reduced by 55.89% and 46.61%.

[0183] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

[0184] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code. The scheme in the embodiments of the present application can be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0185] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0188] Although the preferred embodiments of the present application have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0189] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for coordinated control of power grid frequency, characterized in that: include: Establishing a first target model, wherein the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems; Establishing the objective function and constraint conditions based on the first improved algorithm controller; The first target model is solved according to the target function and the constraint conditions.

2. The grid frequency coordinated control method according to claim 1, characterized in that: The first target model includes: the first target model is an arbitrary model whose input is a traditional unit control signal and whose output is a frequency deviation.

3. The grid frequency coordinated control method according to claim 2, characterized in that: The objective function includes: The objective function is an arbitrary function that, when disturbed, the frequency deviation of the system is restored to 0 within a target time period, and the unit output fluctuation is less than a first fluctuation threshold.

4. The grid frequency coordinated control method according to claim 3, characterized in that: The solving the first target model according to the target function and the constraint condition comprises: Converting the first target model into a Markov decision process according to the target function and the constraint conditions; The objective function and the constraints are used as the state space, action space and reward function of the Markov decision process, wherein the objective function is the reward function of the Markov decision process, and the constraints are the state space and action space of the Markov decision process.

5. The method for coordinated control of power grid frequency according to claim 4, characterized in that: The solving the first target model according to the target function and the constraint condition further includes: Setting up an environment and an agent based on the Markov decision process; The environment of the Markov decision process is the first target model; The intelligent agent of the Markov decision process is a frequency controller.

6. The method for coordinated control of power grid frequency according to claim 5, characterized in that: The state space of the Markov decision process includes: frequency deviation, integral value of frequency deviation, differential value of frequency deviation and last action value of the intelligent agent.

7. The method for coordinated control of power grid frequency according to claim 6, characterized in that: The action space includes: control signals of traditional units.

8. A power grid frequency coordinated control system, characterized in that: include: A model building module, used to build a first target model, wherein the first target model includes at least one or more groups of traditional units, wind turbine units or energy storage systems; An improved algorithm module, used to establish an objective function and constraint conditions based on a first improved algorithm controller; A solution module is used to solve the first target model according to the objective function and the constraint conditions.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images