A method of coordinated power grid frequency control based on knowledge-data hybrid driven algorithm

Abstract

A method for coordinated control of power grid frequency based on a knowledge-data hybrid driven algorithm, which establishes a frequency control model of a multi-machine system including energy storage equipment according to the transfer function of each frequency regulation unit of a new power system in the control system; adopts an actor-critic mechanism, takes a deep policy network as an actor, builds a mapping relationship between local information and controller action, and constructs a critic with a policy gradient analytical expression of the system frequency deviation and the frequency regulation controller output on the controller output action, to assist in offline training of the deep policy network parameters. The present invention establishes a knowledge-data hybrid driven algorithm for a new power system frequency regulation control system including energy storage equipment, significantly reduces the number of iterations and computational overhead required for algorithm training, and takes into account the economic factors of frequency regulation of the new power system, so that each frequency regulation controller of the new power system can achieve coordinated control among frequency regulation units of the entire network by relying only on its own local state information in distributed execution.

Description

Technical Field

[0001] The invention relates to a method for coordinated control of power grid frequency. Background Art

[0002] Under the national energy strategy background of "carbon peak" and "carbon neutrality", conventional synchronous generators driven by fossil fuel combustion are being replaced by generators driven by clean renewable energy such as wind power and photovoltaics at an accelerated pace. However, the increasing penetration rate of renewable energy is prone to power fluctuations on the power supply side, which poses a challenge to the balance of power supply and demand, and significantly weakens the power system's ability to control its frequency. At the same time, the gradual replacement of conventional synchronous generators by new energy generators connected to the grid with power electronic converters as interfaces will also significantly reduce the level of rotational inertia of the entire system, thereby greatly exacerbating the system frequency fluctuations caused by power disturbances.

[0003] The increasingly fragile frequency stability of the power system is gradually becoming a bottleneck problem that restricts the continued promotion of renewable energy generation substitution. Low inertia levels and continuously fluctuating power supply side power will be the new normal for the operation and control of future power grids with high penetration of renewable energy. Renewable energy generation systems have the potential to assist in frequency regulation. For renewable energy generation systems equipped with energy storage, the frequency response characteristics of the entire system can be simulated by reasonably adjusting the energy storage output.

[0004] The deep reinforcement learning algorithm that has emerged in recent years has provided a new solution to the real-time frequency control problem of low-inertia power systems. With its powerful search and learning capabilities, the deep reinforcement learning algorithm has the potential to optimize online decisions when facing complex nonlinear frequency control problems. The Google DeepMind team has extended deep reinforcement learning to the continuous action domain and proposed an actor-critic algorithm framework based on the Deep Deterministic Policy Gradient (DDPG), which greatly expanded the application scope of reinforcement learning. Summary of the invention

[0005] In order to achieve real-time control of grid frequency under renewable energy penetration, the present invention provides a grid frequency collaborative control method based on a knowledge-data hybrid driven algorithm.

[0006] Based on traditional deep reinforcement learning, the present invention establishes a knowledge-data hybrid driven real-time decision-making algorithm for online coordinated frequency response control of generators and energy storage systems.

[0007] In order to achieve the above object, the technical solution of the present invention is:

[0008] A method for coordinated control of power grid frequency based on a knowledge-data hybrid driven algorithm, the method comprising the following steps:

[0009] S1: Establish a grid frequency response control model. Determine the model parameter settings (such as Figure 1 As shown): Determine the droop control coefficient R j (j∈[1,m+n]); respectively determine the time constant T of the frequency regulating unit speed governor and the steam turbine unit a 、T b ; Determine the time constant and frequency modulation participation factor T of the energy storage system respectively B , k B,j (j∈[m+1,m+n]); respectively determine the system disturbance power, system inertia time constant and load damping coefficient ΔP L , H and D; determine the maximum / minimum output power constraints of the frequency regulation generator and energy storage system participating in the frequency dynamic response.

[0010]

[0011]

[0012] In the formula, the output power change ΔP of the i-th generator and the j-th energy storage system is determined respectively G,i and ΔP B,j ; Determine the upper and lower limits of the output power of the i-th generator respectively Determine the maximum charge / discharge power representing the jth energy storage system

[0013] S2: Establish a real-time decision-making algorithm driven by knowledge-data hybrid, using an actor-critic architecture. The actor is responsible for providing the optimal action strategy π in each step, which is essentially a deterministic function that maps the state s to the action A, and passes it through the policy network u(s; θ u ) to approximate it. In addition, in order to avoid the algorithm from falling into the local optimum too early, this algorithm performs an approximation on the policy network u(s; θ) during the training process. u ) is superimposed with the random autocorrelation coefficient ξ. Therefore, during the training process, the output action A of controller j in state s is j It can be expressed as:

[0014]

[0015] In the formula, Representing the Actor Network network parameters.

[0016] The controllers (actor networks) distributedly installed in different frequency modulation units and energy storage systems can obtain a large amount of experience sample data by interacting with the environment. These sample data will first be injected into the critic model during the parameter training process of the actor network. After the "data" stream carrying the original experience sample information is combined with the frequency response model knowledge, the actor network parameters are finally updated centrally in the form of policy gradients. The established critic is a parameter derived based on model knowledge that can be used to assist the actor neural network parameters. Policy gradients for training updates.

[0017] S3: Established training process, training knowledge-data hybrid driven algorithm.

[0018] Preferably, the process of practicing the knowledge-data hybrid driven algorithm described in step S3 is as follows;

[0019]

[0020]

[0021] Furthermore, in step S2, the construction of the knowledge-data hybrid driven real-time decision algorithm includes the following process:

[0022] S2-1: Establish control objectives and reward functions, take the minimum value of the weighted sum of system frequency deviation and system frequency modulation total cost as the algorithm control objective, and establish the state action value function Q u (s,A) is used to quantify the overall performance of the system under the collaborative control of all agents in the future T time:

[0023]

[0024] In the formula, A represents the set of output actions of all agents at the current moment, that is, A = [A1, A2, ···A m ,···,A m+n ]; G and B are the collections of frequency-regulated generator sets and energy storage systems respectively; m and n are the numbers of frequency-regulated generator sets and energy storage systems respectively; Q u The superscript u in (s,A) indicates that the state action value function Q is determined under the current state s. u The action set A of size (s,A) is composed of the outputs of m+n policy networks u=[u1,u2,...,u m ,...,u m+n ]; Δt and T represent the time step size and the length of the empirical trajectory respectively; s is the state information of the system at time t; is the unit output cost of the generator set; is the unit power dispatch cost of the energy storage system, where the energy storage is in the charging state Energy storage in discharge state λ is the weighted coefficient for balancing the frequency deviation of the power grid and the total cost of system frequency regulation; f0, P0, and C0 represent the normalized reference values ​​for unifying the dimensions. In order to eliminate the system frequency deviation as quickly as possible, the first term on the right side of equation (4) adds a discount factor γ∈(0,1), so that the current Δf has a greater weight than the future Δf.

[0025] S2-2: Establish an actor network parameter update method by maximizing the expectation E of the state action value function s～ρ [Q u (s, A)] can obtain the optimal control action with it as the evaluation index. Therefore, based on the idea of ​​gradient ascent algorithm, the present invention calculates E in sequence through chain derivation rule s～ρ [Q u (s,A)] for the parameters The gradient of θ is thus updated through iterations u Maximize E along the gradient direction s～ρ [Q u (s,A)], that is, we can obtain the strategy network Output the network parameters of the optimal action

[0026] Taking the jth controller as an example, E s～ρ [Q u (s,A)] for network parameters The gradient and corresponding parameters The update formula is:

[0027]

[0028]

[0029] Where η∈(0,1) is the policy network parameter The learning rate; N is the number of samples in a small batch, and by obtaining N state action value functions Q u (s,A) Gradient The average value of E is used to approximate s～ρ [Q u (s,A)] To achieve the parameter update in equation (6), we must first find and because is a strategy network with multiple hidden layers, and its corresponding function expression is therefore,

[0030]

[0031] In the formula, p represents the number of neural network layers; state s is the input of the neural network; Represents the activation function of the lth layer of neurons in policy network j.

[0032] S2-3: Establish the policy gradient of the state action value function, in the network parameter E s～ρ [Q u (s,A)], in order to obtain the expectation E of the state action value function s～ρ [Q u (s,A)] The key to the gradient of the state action value function is to know For controller action A j Gradient According to the chain derivation rule, for the i∈Gth frequency modulation unit, we can obtain:

[0033]

[0034] For the j∈Bth energy storage system, we can obtain:

[0035]

[0036] According to the frequency response control model, the system frequency deviation Δf(t) and the frequency regulation unit / energy storage system controller output command A i∈G / B The transfer function between can be expressed as follows:

[0037]

[0038] Compared with the generator set, the time constant T of the energy storage system B It can be almost ignored. For the convenience of calculation, T B Approximately 0. In addition, the transfer function between the total output of each frequency regulation unit and the energy storage system and the system frequency can be expressed as:

[0039]

[0040] 1) Derivation

[0041] Perform the inverse Laplace transform on equation (10) and ignore the After the high-order terms of A i∈G The gradient of the system frequency deviation Δf(t) can be obtained after sorting to obtain the output action A of the controller of the i-th frequency modulation unit i∈G The policy gradient

[0042]

[0043] It represents the system frequency change caused by the unit action output by the controller of the i-th frequency regulating unit at the current moment at the subsequent moment t. Therefore, solving the first-order differential equation of formula (12) in the zero state, we can get:

[0044]

[0045] 2) Derivation

[0046] Perform inverse Laplace transform on equation (11) and calculate the control action A i∈G Find the gradient and rearrange it to get:

[0047]

[0048] Substitute equation (12) into equation (14) and rearrange it to obtain:

[0049]

[0050] In the formula It is given by formula (13). Substituting formula (13) and formula (15) into formula (8), we finally get The analytical expression of .

[0051] 3) Derivation

[0052] Similarly, find A on both ends of equation (10) j∈B The gradient is:

[0053]

[0054] (16) represents the unit action quantity output by the j-th energy storage system controller at the current moment, and the system frequency change caused at the next moment t. Therefore, solving the first-order differential equation of formula (16) in the zero state, we can get:

[0055]

[0056] The first term on the right side of formula (17) represents the unit control instruction A j∈B The resulting frequency change is the final steady-state value.

[0057] 4) Derivation

[0058] Perform inverse Laplace transform on equation (11) and calculate the control action A j∈B Find the gradient and rearrange it to get:

[0059]

[0060] Substitute equation (16) into equation (18) and rearrange it to obtain:

[0061]

[0062] In the formula It is given by formula (17). Substituting formula (17) and formula (19) into formula (9), we finally get The analytical expression of .

[0063] S2-4: Establish output power constraints for frequency regulation units and energy storage systems

[0064] The actual active output of the frequency-regulating unit and the energy storage system is not only regulated by their respective droop control coefficients and controller outputs, but also needs to always meet their maximum and minimum output power constraints. The power change of the frequency-regulating unit / energy storage system in the frequency response can be expressed as:

[0065]

[0066] Where ΔP G / B,j represents the power change of the jth frequency regulation unit or energy storage system. Therefore, the maximum / minimum output power constraints of the generator and energy storage system are expressed as:

[0067]

[0068] In the formula They represent the reference operating point and maximum / minimum output power constraints of the jth frequency regulation unit / energy storage system. Also consider Then P G / B,j About network parameters The gradient of can be expressed as follows:

[0069]

[0070] From formula (22), we can see that under the same system state s, if the policy network parameters If the update is in the following direction, the corresponding generator / energy storage system power output will increase, otherwise the power output will decrease.

[0071]

[0072] Combined with the equation (6), along E s～ρ [Q u (s, A)] The direction of the gradient rise, and finally the strategic network parameters considering the maximum / minimum limit of the generator / energy storage system output power can be obtained The update rule is:

[0073]

[0074] Formula (24) shows that if Direction Update Parameters This will cause the generator / energy storage system output power to violate its maximum / minimum limit. In the next iteration, Will be along the vector In vector The projection direction on the plane of the normal continues to be updated, so that The update trajectory of is always within the feasible domain described by the frequency control model.

[0075] The working principle of the present invention is:

[0076] The effect of the present invention is to establish a control model of an actual new power system according to step S1, establish the Python program of the algorithm through step S2, combine the actual photovoltaic output fluctuation data, and train the algorithm through step S3 to achieve the control effect.

[0077] The beneficial effects of the present invention are:

[0078] 1. After being fully trained with global information, the deep strategy network can learn the different characteristics of each generator and energy storage system in the frequency response process, so that each fully trained intelligent agent can achieve coordinated control among the generators / energy storage systems of the entire network by relying only on its own local state information in distributed execution.

[0079] 2. Compared with existing deep reinforcement learning methods (such as DQN and DDPG), the algorithm proposed in this invention has obvious advantages in performance indicators such as system frequency control, total frequency modulation cost and algorithm training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 This is a frequency coordinated control model diagram of a multi-machine system with energy storage according to the present invention.

[0081] Figure 2 This is a schematic diagram of the frequency coordinated control of a multi-machine system with energy storage based on a knowledge-data hybrid driven algorithm of the present invention.

[0082] Figure 3 (a) is a diagram showing the photovoltaic power variation of a high photovoltaic penetration power system according to the present invention, Figure 3 (b) is a diagram showing load power changes in the high photovoltaic penetration power system of the present invention.

[0083] Figure 4 It is a diagram showing the change of the disturbance power of the present invention.

[0084] Figure 5 It is a comparison chart of the frequency control effect between the algorithm proposed in the present invention and the PID controller.

[0085] Figure 6This is a comparison chart of the frequency control effects of the algorithm proposed in the present invention and DQN and DDPG.

[0086] Figure 7 (a)~ Figure 7 (c) is a comparison chart of the iterative convergence process of the cumulative reward value of the algorithm proposed in the present invention and different enhanced algorithms, where Figure 7 (a) is the iteration cycle of the DDPG algorithm, Figure 7 (b) is the DQN algorithm iteration cycle, Figure 7 (c) is the iteration period of the algorithm proposed in the present invention.

[0087] Figure 8 This is a comparison chart of the algorithm frequency control effect before and after adding the economic control target.

[0088] Fig. 9 It is a power change diagram of the coordinated response of different generator sets and energy storage to grid frequency deviation after adding the economic control target.

[0089] Fig.10 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0090] The present invention will be further described below in conjunction with the accompanying drawings.

[0091] Reference Figure 1 , Figure 2 , a power grid frequency coordinated control method based on a knowledge-data hybrid driven algorithm, the method comprising the following steps:

[0092] S1: Establish a grid frequency response control model to determine the number m, n of frequency-regulated generators and energy storage devices in the actual single-region power system; determine the droop control coefficient R of each frequency-regulated generator j (j∈[1,m+n]); determine the time constants of the frequency-regulated unit speed governor and the steam turbine unit respectively T a 、T b ; Determine the time constant and frequency modulation participation factor T of the energy storage system B , k B,j (j∈[m+1,m+n]); determine the system inertia time constant and load damping coefficient H, D; determine the upper and lower limits ΔP of the power constraints of each generator set and energy storage device involved in frequency regulation G,i , ΔP B,j .

[0093] S2: Establish a knowledge-data hybrid driven algorithm based on the grid frequency response control model, determine the control objectives and reward functions while considering economic indicators; determine the actor network parameter update; and determine the policy gradient of the action value function.

[0094] S3: Establish a knowledge-data hybrid driven algorithm training process, determine the algorithm parameters according to the knowledge-data hybrid driven algorithm established in S2, and then determine the knowledge-data hybrid driven algorithm training process according to Table 1;

[0095] S4: Apply the well-trained knowledge-data hybrid drive to the actual single-region power system frequency regulation task.

[0096] In step S2, the construction of the knowledge-data hybrid driven algorithm includes the following process:

[0097] S2-1: Determine the action set A according to the number of generators and energy storage devices involved in frequency regulation, that is, A = [A1, A2, ···A m ,···,A m+n ]; G and B are the sets of frequency-regulated generators and energy storage systems; m and n are the number of frequency-regulated generators and energy storage systems. Determine the unit output cost of the generator set Unit power dispatch cost of energy storage system Cost of energy storage in charging state Cost of energy storage in discharged state The weighting coefficient λ for balancing the grid frequency deviation and the total cost of system frequency regulation;

[0098] S2-2: Establish the actor network parameter update method, and the function fitter used is a deep neural network:

[0099]

[0100] Determine the number of neural network layers p; is the input state s of the neural network; the activation function of the lth layer of neurons in the strategy network j

[0101] S2-3: Establish the critic model parameter update method based on the policy gradient of the state action value function;

[0102] Strategic network parameters considering maximum / minimum constraints on generator / storage system output power Create update rules:

[0103]

[0104] In order to enable those skilled in the art to better understand the present invention, the example analysis includes the following components:

[0105] 1. Case Description

[0106] This example uses Figure 3 A typical high photovoltaic penetration power system is taken as the research object, and the net load fluctuation power (the difference between the load active power and the photovoltaic output) at noon on a certain day is selected as the system disturbance power ΔPL Perform case simulation. Due to the influence of drastic fluctuations in photovoltaic power (such as Figure 4 As shown), the disturbance power ΔP within 10 minutes during the noon period L The peak and valley values ​​are 1800MW and 1215MW respectively, with a variation of 48%. If effective control strategies are not adopted, the stability of system frequency and even the safe operation of the power grid will face severe challenges.

[0107] 2. Analysis of Actual Control Behavior

[0108] The example uses a multi-machine power system consisting of 8 generators (numbered G1-G8) and 1 energy storage system to verify the effectiveness of the proposed real-time decision-making algorithm in system frequency control and system frequency regulation total cost optimization. A dynamic simulation model of the frequency response of a multi-machine power system with energy storage is established in the Python environment and used as the training environment for the actor network in the algorithm proposed in this invention. In addition, the output power benchmark value of each generator set is Maximum output power Minimum output power and unit electricity generation cost As shown in Table 1: Output power benchmark value of energy storage system Maximum charge / discharge power And the unit power dispatch cost As shown in Table 2. The parameter settings of the algorithm proposed in the present invention, such as the number of experience trajectories (episodes) M, the capacity of the experience replay pool D, the number of small batch sampling samples N, the learning rate η, the discount factor γ, the number of agents g, etc., are shown in Table 3.

[0109]

[0110]

[0111] Table 1

[0112]

[0113] Table 2

[0114]

[0115] Table 3

[0116] 3. Case Comparative Analysis

[0117] The example first compares the real-time control performance of the system frequency between the traditional PID controller, the PID controller optimized by the PSO algorithm parameters, and the algorithm established by the present invention. Figure 5It can be seen that under the real-time adjustment of the traditional PID controller, there is a system frequency deviation of about ±0.042Hz during the noon period when the disturbance power fluctuates violently. Compared with the traditional PID controller, after the PSO algorithm parameter optimization, the frequency effect of the PID controller has improved, but the effect is still not ideal, and the system frequency deviation fluctuates within the range of (-0.030Hz, 0.032Hz). In the same period, the real-time decision-making algorithm proposed in the present invention can significantly narrow the system frequency deviation range to (-0.015Hz, 0.018Hz), thereby greatly improving the system frequency stability of the power grid in the scenario of large-scale new energy grid connection.

[0118] The algorithm proposed in this invention and the DDPG algorithm are both deep reinforcement learning algorithms based on continuous action space. Figure 6 As shown in the figure, the real-time frequency control effects of both algorithms are significantly better than the DQN algorithm designed based on discrete action space. After integrating the respective advantages of knowledge-driven and data-driven algorithms, the algorithm proposed in the present invention has further improved the real-time frequency control effect in the continuous action domain compared to the fully trained DDPG algorithm. Figure 7 The iterative convergence process of the cumulative reward value of the DQN algorithm, the DDPG algorithm and the algorithm proposed in the present invention is compared. Among them, after the DQN algorithm and the DDPG algorithm have experienced about 400 and 650 iteration cycles respectively, the changes in their cumulative reward values ​​tend to be stable and no longer continue to increase. Each iteration cycle of this example is an empirical trajectory containing 200 iterations. That is, the two algorithms require 80,000 and 130,000 iteration updates respectively to converge. The algorithm that adopts knowledge-data hybrid drive only needs about 150 iteration cycles (30,000 iteration updates) to complete the parameter training of the deep neural network, and the convergence curve of the algorithm has the smallest oscillation amplitude.

[0119] The algorithm proposed in the present invention balances the frequency deviation control effect of the power grid and the total cost of system frequency regulation by introducing a weighting coefficient λ into the objective function. The frequency and economic indicators of the algorithm under different weighting coefficients are shown in Table 4.

[0120]

[0121] Table 4

[0122] in, are the absolute value of the frequency deviation and the average value of the total system frequency regulation cost during the simulation period. As shown in Table 4, a larger weighting coefficient λ increases the total system frequency regulation cost while making the algorithm proposed in the present invention have a better frequency control effect; on the contrary, a smaller weighting coefficient λ further improves the economy of system operation at the expense of increasing the system frequency fluctuation amplitude. The dispatching operator can set the weighting coefficient accordingly according to the relative importance of the frequency deviation and the total system frequency regulation cost under the current system operation state.

[0123] Figure 8 The frequency control effects of the algorithm proposed in the present invention when the weighted coefficient λ is 1 and 0.6 are compared. Since the economic index is added to the control target, the penalty weight of the algorithm for frequency deviation is relatively reduced, resulting in a slight increase in the maximum frequency deviation of the system compared to before the addition of the economic index. Therefore, under the control of the proposed algorithm, the reduction in the overall operating cost of the system does not come at the expense of significantly sacrificing the system frequency control effect, and thus can better balance the system frequency deviation control and the total cost of system frequency regulation.

[0124] This example compares the output power changes of each online generator set and energy storage system in the scenario of a sudden increase in the net load (disturbance power) of the power grid from 11:15:00 to 11:15:30, in order to demonstrate the coordinated response of different entities to the system frequency deviation under the control of the algorithm proposed in this invention. Figure 4 As shown in the figure, since 11:15:00, due to the violent fluctuation of photovoltaic power, it first experienced a rapid rise process; then it entered a relatively stable stage from 11:15:10 to 11:15:20; finally, from 11:15:20 to 11:15:30, the system net load began to drop slightly. In order to quickly compensate for the system frequency drop caused by the sudden increase in net load, Fig. 9 As shown, the output power of the generator set and the energy storage system will increase rapidly in the first stage after the sudden increase in net load. In this stage, the offset of the system frequency is large, and the first item (frequency deviation) in the control target equation (4) plays a dominant role. Since the energy storage system is almost unconstrained by the ramp rate, its discharge power reaches the maximum instantaneously, and maintains its maximum discharge power in the first stage to assist the generator set with slower ramping to perform frequency regulation. However, as the system frequency deviation is gradually compensated, in the second stage after about 11:15:10, the role of the second item (economic indicator) in the control target (4) begins to emerge. From Fig. 9 It can be seen that in the second stage (the system net load is relatively stable), due to the impact of frequency regulation costs, the generator sets G7 and G8 with higher unit output costs gradually reduce their power output during this stage, and the energy storage system reduces the discharge power to near zero, thereby improving the overall frequency regulation economy of the system; while the generator sets G1 and G2 with lower unit output costs will continue to increase their respective output power during this stage to continue to compensate for the system frequency deviation and the power shortage caused by the gradual withdrawal of energy storage and high-cost units. It is worth noting that due to the constraints of the output power upper limit, such as Fig. 9As shown, G1 reaches output saturation in the second stage. Entering the third stage, affected by the slow decline in net load power, the output power of generators with higher output (such as G7) and generators with medium output cost (such as G4) has decreased to varying degrees. It can be seen that after adding the economic control index, the algorithm proposed in the present invention can reasonably allocate power output according to the real-time frequency deviation of the current system and the unit output cost of each frequency regulation subject as well as the upper and lower limits of the output, thereby achieving an economic coordinated response to the system frequency deviation.

[0125] The contents described in the embodiments of this specification are merely an enumeration of implementation forms of the inventive concept. The protection scope of the present invention should not be regarded as limited to the specific forms described in the embodiments. The protection scope of the present invention also includes equivalent technical means that can be thought of by those skilled in the art based on the inventive concept.

Claims

1. A method for coordinated power grid frequency control based on a knowledge-data hybrid driven algorithm, characterized in that: The following steps are involved: S1: Establish the grid frequency response control model and determine the model parameter settings, including: Determine the droop control coefficient R j , j∈[1,m+n]; respectively determine the time constant T of the frequency regulating unit speed governor and the steam turbine unit a , T b ; Determine the time constant and frequency modulation participation factor T of the energy storage system respectively B , k B,j , j∈[m+1,m+n]; respectively determine the system disturbance power, system inertia time constant and load damping coefficient ΔP L , H and D; determine the maximum / minimum output power constraints of the frequency-regulated generator and energy storage system in participating in the frequency dynamic response; In the formula, the output power change ΔP of the i-th generator and the j-th energy storage system is determined respectively G,i and ΔP B,j ; Determine the upper and lower limits of the output power of the i-th generator respectively Determine the maximum charge / discharge power representing the jth energy storage system S2: Establish a real-time decision-making algorithm driven by knowledge-data hybrid, including: adopting an actor-critic architecture; the actor provides the optimal action strategy π in each step, which is essentially a deterministic function that maps the state s to the action A, and passes through the strategy network u(s; θ u ) to approximate it; in order to avoid falling into the local optimum too early, the policy network u(s; θ) is approximated during the training process. u ) is superimposed with the random autocorrelation coefficient ξ; therefore, during the training process, the output action A of the controller j in state s j It can be expressed as: In the formula, Representing the Actor Network Network parameters; The controllers of the distributed frequency regulation units and energy storage systems can obtain a large amount of experience sample data by interacting with the environment; these sample data will first be injected into the critic model during the parameter training of the actor network; after the "data" stream carrying the original experience sample information is combined with the frequency response model knowledge, the actor network parameters are finally updated centrally in the form of policy gradients; the established critic is a parameter derived based on the model knowledge that can be used to assist the actor neural network parameters Policy gradients for training updates; S3: Established training process, training knowledge-data hybrid driven algorithm.

2. A method for coordinated power grid frequency control based on a knowledge-data hybrid driven algorithm as claimed in claim 1, characterized in that: The training knowledge-data hybrid driving algorithm described in step S3 specifically includes:

3. A method for coordinated power grid frequency control based on a knowledge-data hybrid driven algorithm as claimed in claim 1, characterized in that: The construction of the knowledge-data hybrid driven real-time decision algorithm described in step S2 includes: S2-1: Establish control objectives and reward functions, take the minimum value of the weighted sum of system frequency deviation and system frequency modulation total cost as the algorithm control objective, and establish the state action value function Q u (s,A) is used to quantify the overall performance of the system under the collaborative control of all agents in the future T time: In the formula, A represents the set of output actions of all agents at the current moment, that is, A = [A1, A2, ···A m ,···,A m+n ]; G and B are the collections of frequency-regulated generator sets and energy storage systems respectively; m and n are the numbers of frequency-regulated generator sets and energy storage systems respectively; Q u The superscript u in (s,A) indicates that the state action value function Q is determined under the current state s. u The action set A of size (s,A) is composed of the outputs of m+n policy networks u=[u1,u2,...,u m ,...,u m+n ]; Δt and T represent the time step size and the length of the empirical trajectory respectively; s is the state information of the system at time t; is the unit output cost of the generator set; is the unit power dispatch cost of the energy storage system, where the energy storage is in the charging state Energy storage in discharge state λ is the weighting coefficient for balancing the frequency deviation of the power grid and the total cost of system frequency regulation; f0, P0, and C0 represent the normalized reference values ​​for unifying the dimensions; in order to eliminate the system frequency deviation as quickly as possible, the first term on the right side of equation (4) adds a discount factor γ∈(0,1), so that the current Δf has a greater weight than the future Δf; S2-2: Establish an actor network parameter update method by maximizing the expected value of the state action function s～ρ [Q u (s, A)] can obtain the optimal control action with it as the evaluation index; therefore, based on the idea of ​​the gradient ascent algorithm, the present invention calculates E in sequence through the chain derivation rule s～ρ [Q u (s,A)] for the parameters The gradient of θ is thus updated through iterations u Maximize Ε along the gradient direction s～ρ [Q u (s,A)], that is, we can obtain the strategy network Output the network parameters of the optimal action Taking the jth controller as an example, E s～ρ [Q u (s,A)] for network parameters The gradient and corresponding parameters The update formula is: Where η∈(0,1) is the policy network parameter The learning rate; N is the number of samples in a small batch, and by obtaining N state action value functions Q u (s,A) Gradient The average value of E is used to approximate s～ρ [Q u (s,A)] To achieve the parameter update in equation (6), we must first find the gradient of equation (5) and because is a strategy network with multiple hidden layers, and its corresponding function expression is therefore, In the formula, p represents the number of neural network layers; state s is the input of the neural network; represents the activation function of the lth layer of neurons in the policy network j; S2-3: Establish the policy gradient of the state action value function, in the network parameter E s～ρ [Q u (s, A)], in order to obtain the expected value of the state action function Ε s～ρ [Q u (s,A)] The key to the gradient of the state action value function is to know For controller action A j Gradient According to the chain derivation rule, for the i∈Gth frequency modulation unit, we can obtain: For the j∈Bth energy storage system, we can obtain: According to the frequency response control model, the system frequency deviation Δf(t) and the frequency regulation unit / energy storage system controller output command A i∈G / B The transfer function between can be expressed as follows: Compared with the generator set, the time constant T of the energy storage system B It can be almost ignored; for the convenience of calculation, T B is approximately 0; in addition, the transfer function between the total output of each frequency regulation unit and the energy storage system and the system frequency can be expressed as: 1) Derivation Perform the inverse Laplace transform on equation (10) and ignore the After the high-order terms of A i∈G The gradient of the system frequency deviation Δf(t) can be obtained after sorting to obtain the output action A of the controller of the i-th frequency modulation unit i∈G The policy gradient It represents the unit action quantity output by the controller of the i-th frequency regulating unit at the current moment, and the system frequency change caused at the next moment t; therefore, solving the first-order differential equation of formula (12) in the zero state, we can get: 2) Derivation Perform inverse Laplace transform on equation (11) and calculate the control action A i∈G Find the gradient and rearrange it to get: Substitute equation (12) into equation (14) and rearrange it to obtain: In the formula It is given by formula (13); Substituting formula (13) and formula (15) into formula (8), we finally get The analytical expression of ; 3) Derivation Similarly, find A on both ends of equation (10) j∈B The gradient is: (16) represents the unit action quantity output by the j-th energy storage system controller at the current moment, and the system frequency change caused at the next moment t; therefore, solving the first-order differential equation of formula (16) in the zero state, we can obtain: The first term on the right side of formula (17) represents the unit control instruction A j∈B The final steady-state value of the frequency change caused; 4) Derivation Perform inverse Laplace transform on equation (11) and calculate the control action A j∈B Find the gradient and rearrange it to get: Substitute equation (16) into equation (18) and rearrange it to obtain: In the formula It is given by formula (17); Substituting formula (17) and formula (19) into formula (9), we finally get The analytical expression of ; S2-4: Establish output power constraints for frequency regulation units and energy storage systems The actual active output of the frequency-regulating unit and the energy storage system is not only regulated by their respective droop control coefficients and controller outputs, but also needs to always meet their maximum and minimum output power constraints; the power change of the frequency-regulating unit / energy storage system in the frequency response can be expressed as: Where ΔP G / B,j represents the power change of the jth frequency regulation unit or energy storage system; therefore, the maximum / minimum output power constraints of the generator and energy storage system are expressed as: In the formula Respectively represent the reference operating point and maximum / minimum output power constraints of the jth frequency regulation unit / energy storage system; Also consider Then P G / B,j About network parameters The gradient of can be expressed as follows: From formula (22), we can see that under the same system state s, if the policy network parameters If the update is in the following direction, the corresponding generator / energy storage system power output will increase, otherwise the power output will decrease; Combined with the equation (6), the s～ρ [Q u (s, A)] The direction of the gradient rise, and finally the strategic network parameters considering the maximum / minimum limit of the generator / energy storage system output power can be obtained The update rule is: Formula (24) shows that if Direction Update Parameters This will cause the generator / energy storage system output power to violate its maximum / minimum limit. In the next iteration, Will be along the vector In vector The projection direction on the plane of the normal continues to be updated, so that The update trajectory of is always within the feasible domain described by the frequency control model.

Images