Energy storage auxiliary thermal power generating unit deep reinforcement learning load frequency control method

Through the deep reinforcement learning method combined with Markov decision model and Actor-Critic architecture, DDPG algorithm and random network distillation technology are used to solve the shortcomings of SOC status and grid frequency regulation requirements in the load frequency control of energy storage-assisted thermal power units, and realize energy storage The coordinated operation with the thermal power unit simplifies the controller design and optimizes the grid frequency control effect and energy storage SOC status.

CN119051070AActive Publication Date: 2024-11-29NANJING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202410946519.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-29
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing technology failed to fully consider the SOC status of the energy storage system and the grid frequency regulation requirements in the load frequency control of energy storage auxiliary thermal power units, resulting in insufficient energy storage participation in secondary frequency regulation, high control strategy complexity, and difficulty in solving multi-objective reward function .

Method used

Design a deep reinforcement learning load frequency control method, combines Markov decision model and Actor-Critic architecture, adopts DDPG algorithm and random network distillation technology, and realizes energy storage and thermal power through multi-objective dynamic adaptive reward function and dynamic weight adjustment. The coordinated operation of the unit simplifies controller design and optimizes energy storage output strategy.

Benefits of technology

The coordinated operation of energy storage and thermal power units is realized, the ability of energy storage to participate in secondary frequency regulation is improved, the controller design is simplified, the energy storage output strategy is dynamically adjusted, and the grid frequency control effect and energy storage SOC status are optimized.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119051070A_ABST
    Figure CN119051070A_ABST
Patent Text Reader

Abstract

The invention discloses an energy storage assisted thermal power generating unit deep reinforcement learning load frequency control method, which comprises the steps of establishing a deep reinforcement learning environment of a power system load frequency control problem so as to simulate characteristics and control requirements of each component in an actual power system; aiming at a lithium ion battery energy storage system in an electric power system assembly, designing an energy storage system output control strategy considering the frequency deviation of the electric power system and the charge state of the energy storage system; the method comprises the following steps: modeling a load frequency control problem of a power system as a Markov decision model, and constructing a deep reinforcement learning load frequency control framework based on an Actor-Critic architecture in combination with the Markov decision model and an energy storage system output control rule; a depth deterministic strategy gradient (DDPG) algorithm and a random network distillation technology are combined, and a dual-output load frequency controller (agent) is designed, including agent state, action and a multi-target dynamic adaptive reward function; and designing an intelligent agent solving algorithm based on dynamic weight adjustment. The load frequency control method provided by the invention does not depend on a specific model, has high environmental adaptability, and can realize complementary coordination control of the energy storage system and the thermal power generating unit and effectively improve the frequency modulation effect of the power system compared with a traditional control strategy of constant energy storage output.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of load frequency control, and specifically relates to a deep reinforcement learning load frequency control method for energy storage-assisted thermal power units. Background Art

[0002] Automatic generation control (AGC) is an important technical means to achieve active power balance and system frequency stability in the power grid. At present, the traditional AGC system is mainly composed of thermal power units, which have the disadvantages of long response time lag, low unit ramp rate, and insufficient ability to track AGC instructions; in addition, with the large-scale access of renewable energy to the power grid, its weak inertia, volatility and uncertainty have put forward higher requirements on the speed of grid frequency regulation. Therefore, it is of high academic significance and engineering application value to introduce better frequency regulation resources to solve the problem of strong random disturbance caused by large-scale grid connection of renewable energy.

[0003] In recent years, energy storage technology represented by electrochemical batteries has developed rapidly, and its large-scale application in power systems is increasing rapidly. At present, a large number of literatures have conducted a series of studies on the application of energy storage batteries in secondary frequency regulation of power systems. The literature Secondary frequency control strategy for BESS considering their degree of participation (Energy Reports, 2020, 594-602) proposed a storage output control strategy based on participation to dynamically adjust the output size of energy storage participating in secondary frequency regulation. The literature Fuzzy logic control and battery energy storage system based power system secondary frequency regulation (2023 7th International Conference on Green Energy and Applications, 2023) comprehensively considered the regional control deviation (ACE) and energy storage SOC signal, and proposed a fuzzy control strategy for energy storage output based on ACE and SOC to smooth the output of energy storage system participating in secondary frequency regulation. However, the energy storage output function of existing literature is mostly expressed by fuzzy control and piecewise function. To a certain extent, it alleviates the problem of overcharging and over-discharging of energy storage, but it also introduces new problems. First, it weakens the rapid response characteristics of energy storage, and second, it is easy to cause secondary disturbances to the system frequency at the critical point of output.

[0004] Therefore, it is very important to design a reasonable energy storage output function. Based on existing research, the present invention selects the sigmoid function as the basic curve of energy storage output, takes SOC as the independent variable, and the energy storage output power as the dependent variable, and constructs the charge and discharge output function curve of the energy storage system. This function can ensure the smoothness of the energy storage output while taking into account the rapid response characteristics of energy storage.

[0005] With the grid connection with a high proportion of distributed renewable energy, the control performance of traditional controllers in the above-mentioned power grid needs to be further improved when facing complex working conditions such as a large number of random disturbances, system parameters and structural changes. In recent years, reinforcement learning algorithms have been widely used in the field of load frequency control due to their strong decision-making ability, high environmental adaptability, continuous exploration and trial and error in the environment, and accumulated experience to obtain the optimal control strategy. The design of load frequency controller based on reinforcement learning has become a hot topic in AGC research. The literature A frequency control strategy of large grid with energy storage based on multi-agent algorithm (2023 IEEE 3rd International Conference on Power, Electronics and Computer Applications, 2023, 181-184) proposes a load frequency control technology for energy storage power grid based on DDPG algorithm for load frequency control systems including thermal power, renewable energy and energy storage batteries, so as to achieve the purpose of energy storage-assisted grid frequency regulation. The paper A safe policy learning-based method for decentralized and economic frequency control in isolated networked-microgrid systems (IEEE Transactions on Sustainable Energy, 2022, 1982-1993) proposes a data-driven decentralized economic frequency control method for isolated microgrid systems. This control method integrates the energy storage output strategy as part of the controller design into the reinforcement learning controller, and limits the energy storage output in different scenarios through constraints. From the existing literature, the current research on energy storage participating in secondary frequency regulation still has the following deficiencies:

[0006] (1) The control strategy does not take into account the SOC state of the energy storage system. Too high or too low SOC will reduce the bidirectional frequency regulation capability of the energy storage battery, and there is no SOC self-recovery function of the energy storage system:

[0007] (2) The energy storage output control strategy is formulated based on the unilateral frequency regulation demand of the power grid, without fully considering the energy storage frequency regulation capability and the power grid frequency regulation demand, and failing to fully utilize the complementary advantages between energy storage and the power grid.

[0008] (3) In the existing methods, the output of energy storage is used as a constraint condition for controller design, which increases the complexity of controller design. In addition, the energy storage output limit considered by the existing methods is relatively simple, and the output characteristics of different energy storage SOC ranges are not considered.

[0009] (4) Existing methods only use a simple linear weighting method to solve the multi-objective reward function, which makes it difficult to determine reasonable multi-objective weights and, therefore, unable to solve the optimal strategy. Summary of the invention

[0010] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units. The method can comprehensively consider the power grid frequency regulation demand and energy storage frequency regulation capability when giving frequency regulation actions, and realize the complementary and coordinated operation of energy storage and conventional units. At the same time, through dynamic weight adaptive adjustment, the coordination problem of multiple optimization objectives is solved.

[0011] To achieve the above object, the present invention provides a method for deep reinforcement learning load frequency control of an energy storage-assisted thermal power unit, comprising the following steps:

[0012] S1. Establish a deep reinforcement learning environment for power system load frequency control problems to simulate the characteristics and control requirements of each component in the actual power system;

[0013] S2. Design an output control strategy for the lithium-ion battery energy storage system in the power system components that takes into account the power system frequency deviation and the state of charge (SOC) of the energy storage system;

[0014] S3. Model the load frequency control problem of the power system as a Markov decision model, and combine the Markov decision model with the output control law of the energy storage system to build a deep reinforcement learning load frequency control framework based on the Actor-Critic architecture;

[0015] S4. Combining the deep deterministic policy gradient (DDPG) algorithm and random network distillation technology, a dual-output load frequency controller (agent) is designed, including the agent input state, the agent output action and the multi-objective dynamic adaptive reward function.

[0016] S5. Based on the external rewards and internal rewards in the multi-objective dynamic adaptive reward function, design an agent (controller) solution algorithm based on dynamic weight adjustment.

[0017] S6. A computer device, characterized in that it comprises a memory and a processor; the memory is used to store computer programs; the processor is used to operate according to the instructions to execute the steps of the method described in any one of claims S1 to S5.

[0018] S7. A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method described in any one of claims S1 to S5 are implemented.

[0019] Furthermore, a general deep reinforcement learning environment for load frequency control model is constructed, which can accurately simulate the characteristics of a real power frequency regulation system. The model components of the load frequency control model in this environment include thermal power units, energy storage systems, photovoltaic power generation, wind power generation and loads.

[0020] Furthermore, the output control strategy of the energy storage system includes two modes: normal adaptive frequency regulation and abnormal energy storage self-recovery. The normal adaptive frequency regulation mode comprehensively considers the grid frequency regulation demand and the energy storage frequency regulation capability when the system has a frequency regulation demand, and outputs active power to assist the power system frequency regulation. The abnormal energy storage self-recovery mode uses the remaining frequency regulation reserve capacity of the power system to restore the energy storage SOC to a state when the system frequency deviation is small and the energy storage SOC is not good, providing more frequency regulation capacity for subsequent frequency regulation tasks.

[0021] 1) Normal adaptive frequency modulation mode: When the system frequency deviation exceeds the frequency dead zone, it indicates that the system has a frequency modulation demand at this time. The charging and discharging rules of the energy storage at this time should be determined according to the energy storage operation characteristics and the real-time SOC state of the energy storage. c and P d .

[0022]

[0023]

[0024] Among them, P r is the rated charge and discharge power of energy storage; SOC l , SOC h , SOC0 and SOC1 represent different partition boundaries of energy storage charge state, which are obtained through parameter identification according to the operating characteristics of the energy storage system; n is an adjustable factor determined according to the operating characteristics of the energy storage battery.

[0025] 2) Abnormal energy storage self-recovery mode: When the system frequency deviation is in the frequency regulation dead zone, and the energy storage SOC state is not good at this time, the energy storage takes the recovery of SOC as the first output target, and makes full use of the remaining frequency regulation reserve capacity of the conventional units, taking into account the operation state restrictions of the power grid, and recovers the energy storage SOC state. The following SOC self-recovery charging and discharging law P is designed c1and P d1 :

[0026]

[0027] Among them, n1 and n2 are adjustable factors, which can be determined according to the operating characteristics of the energy storage battery.

[0028] Furthermore, the steps of modeling the power system load frequency control problem as a Markov decision model are as follows:

[0029] The Markov decision process consists of a four-tuple<S,A,P,R> It can be described as follows: S represents the state set; A represents the action set; P represents the state transition function; and R represents the reward function. In each interaction between the agent and the environment, the power system is regarded as the environment that the agent interacts with. First, the agent observes the system state s t , based on the current strategy, give action a t ; Then, the system performs action a t And update the system's operating status to s t+1 , and returns the agent’s reward R for the action performed t The state transition process composed of the intelligent agent and the power system environment has the Markov property. Therefore, the state transition process of load frequency control can be described by the Markov decision process. As shown in the following formula, the probability of the current state transferring to the next action is only related to the current state and action.

[0030] P(s t+1 |s t , a t )=E(s t+1 |s t , a t )

[0031] Among them, P(s t+1 |s t , a t ) indicates that according to the state s t and action a t , so that the state is transferred to s t+1 The probability of E(s t+1 |s t , a t ) indicates that based on the state s t and action a t The observation function represents the probability distribution based on the observed values.

[0032] During the state transfer process, the agent receives a reward R t And the executed strategy π satisfies the following formula:

[0033]

[0034] Among them, the discount factor γ∈(0,1), π * represents the expected strategy to obtain the maximum return.

[0035] Furthermore, a load frequency control framework is constructed. The framework uses the power system model as the environment, and the system frequency deviation signal and energy storage system SOC signal given by the environment as the controller input. According to the input information, the intelligent agent outputs two frequency modulation actions a1 and a2, which act on the thermal power unit and the energy storage system respectively. The a1 signal is superimposed with the adaptive factor and acts on the thermal power unit model. The a2 signal acts on the energy storage system output strategy module to generate energy storage active output to balance the load changes and realize the load frequency control of the power system. The energy storage output strategy module can independently control the active output and SOC self-recovery of the energy storage system. It avoids the use of energy storage output as a constraint condition for controller design in existing methods, which greatly simplifies the design of the controller. In addition, the output restrictions considered in existing methods are relatively simple, only including the upper and lower bounds of SOC, and do not consider the output characteristics of different energy storage SOC intervals.

[0036] Furthermore, the design of the dual-output load frequency controller includes:

[0037] State S: In the power system, the state observed by the intelligent agent includes the real-time SOC of the energy storage system and the frequency deviation Δf.

[0038] Action A: For the agent, its task is to control the power output of the energy storage system and the thermal power unit. The action set is defined as:

[0039] A=[a1,a2]

[0040] Among them, a1 acts on the thermal power unit and a2 acts on the energy storage system.

[0041] Multi-objective dynamic adaptive reward function: The primary goal of the load frequency control system is to quickly suppress system frequency oscillations and ensure the safe and stable operation of the system. Secondly, make full use of the rapid response characteristics of energy storage to participate in system frequency regulation, while ensuring that the SOC of the energy storage system is in an appropriate range. Finally, on the basis of rapid frequency regulation, the intelligent agent should fully explore the unknown state of the power system environment and improve the environmental adaptability of the controller. Therefore, the following multi-objective dynamic adaptive reward function is designed:

[0042]

[0043] in, It represents the external reward value designed according to the real-time status feedback of the power grid, which is a true reflection of the system frequency deviation. represents the internal reward value, that is, the evaluation value of the random network distillation algorithm on the novelty of the current state. λ1 and λ2 represent the two objectives respectively. and The weight coefficient is a positive number. Defined as:

[0044]

[0045] Among them, when |Δf| is in the frequency modulation dead zone [0, α1], the system frequency meets the minimum error requirement for normal operation, so the maximum reward given to the agent is 0; when |Δf| is in the interval [α1, α2], [α2, α3] and the interval (α3, +∞), it indicates that the agent has not found the optimal strategy and there is a power shortage in the power system. At this time, the agent will receive a corresponding negative reward, that is, a penalty term; among them, β1>0, β2>0 and β3>0 are the weights corresponding to the reward functions of each control area; C is the end of the round mark, and γ>0 is the weight coefficient.

[0046] In order to ensure that the energy storage state of charge is always maintained at a good level and to ensure the effectiveness of the training round, the round end penalty γC is introduced into the reward function. It means that during the round training process, when the system frequency deviation is too large or the energy storage system SOC is low, a large negative reward value is given to the agent, ending the round training and entering the next round. The round end flag is defined as:

[0047] C=1,|Δf|>α4 or SOC<α5

[0048] Among them, α4 and α5 are the values ​​of the constraint conditions.

[0049] Furthermore, the internal reward of the multi-objective dynamic adaptive reward function It is obtained by the random network distillation algorithm, which includes the following steps:

[0050] S51, Environmental state feature extraction based on random network distillation:

[0051] Assume that the state of the power system at time t is s t , the state at time t+1 is s t+1 When state s t+1 Input to the random network, the prediction network gives the state prediction features The target network gives the state target feature f(s t+1 ). The random network consists of a prediction network and a target network. Each network consists of two fully connected layers and two RELU layers, and their parameters are and θ f .

[0052] S52, internal reward solution based on state characteristics:

[0053] Status prediction features For state s t+1 The novelty prediction value of the state target feature f(s t+1 ) is state s t+1 The prediction network is trained to predict the output of the target network. The mean squared error between the predicted and target values ​​represents the internal reward term:

[0054]

[0055] Furthermore, the weight coefficients λ1 and λ2 of the two objective functions are expressed as follows:

[0056] System frequency deviation Δf ave It is an intuitive reflection of the frequency modulation effect of the control strategy. If there is a large frequency error, then it is particularly important to choose appropriate control measures to reduce it. On the other hand, if there is a small deviation, then the correction of this indicator is not so important and other goals can be given priority. External reward goals The weight coefficient is defined as:

[0057]

[0058] Among them, gain1 represents the dynamic gain of the weight, represents a positive constant weight, which represents the relative priority of the corresponding target. gain1 is determined by the following formula:

[0059]

[0060] Where Δf max Denotes the maximum acceptable frequency deviation of the power system in steady state, Δf ave represents the system average frequency deviation at the end of the DDPG agent round training, and h1 represents the gain change rate.

[0061] The error between the predicted value and the target value of the random network distillation model is an objective evaluation of the network prediction effect. Control measures should be taken to reduce it when the error is large, and other goals should be given priority when the error is small. Internal reward target The weight coefficient is defined as:

[0062]

[0063] Among them, gain2 represents the dynamic gain of the weight, represents a positive constant weight, which represents the relative priority of the corresponding target. gain2 is determined by the following formula:

[0064]

[0065] in, represents the maximum acceptable error between the predicted value and the target value of the random network distillation model, represents the average error between the random network distillation model prediction value and the target value at the end of the DDPG agent round training, and h2 represents the gain change rate.

[0066] Furthermore, based on the external rewards and internal rewards in the multi-objective dynamic adaptive reward function, a multi-objective deep reinforcement learning solution algorithm based on dynamic weight adjustment is designed, including:

[0067] Initialize constant weights for multi-objective dynamic adaptive reward functions Deep reinforcement learning network parameters, random distillation network parameters θ f , select an appropriate maximum acceptable steady-state frequency deviation Δf max and the maximum acceptable error of the random network distillation model

[0068] The agent interacts with the environment, the action network is used to output control actions a1 and a2 to act on the power system, and the evaluation network evaluates the reward function and The action value q e and q i , and the system gives the reward value R of the current action e and R i ;

[0069] Based on the evaluation action value, the multi-objective weighted loss function is calculated, the evaluation network parameters are updated by minimizing the weighted loss function, the action network parameters are updated by the gradient descent method, and the random distillation network is updated by minimizing the mean square error between the predicted value and the target value; the multi-objective weighted loss function is expressed as follows:

[0070]

[0071] in It represents the target evaluation value of the evaluation network for the action and state of the next time step, and N represents the number of random sampling samples.

[0072] The agent repeatedly interacts with the environment until the round ends, and the average frequency deviation Δf in the round is calculated ave , the average error between the predicted value and the target value of the random network distillation model And determine Δf ave and If the error is large, adjust the increase rate of dynamic gains h1 and h2, calculate and update the values ​​of weight coefficients λ1 and λ2. If the error is small, fine-tune the weight coefficients to obtain the best frequency modulation performance.

[0073] The agent continues to interact with the power system environment in the next round until the multi-objective dynamic adaptive reward function R t The cumulative reward converges to 0, which means that the agent has solved the optimal control strategy.

[0074] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the above-mentioned deep reinforcement learning load frequency control method for energy storage-assisted thermal power units when executing the program.

[0075] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned deep reinforcement learning load frequency control method for energy storage-assisted thermal power units.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] (1) The present invention considers the energy storage SOC and the self-recovery of the energy storage SOC when providing the energy storage system output rule. The energy storage output control strategy considers both the grid frequency regulation demand and the energy storage frequency regulation capability.

[0078] (2) The energy storage output strategy of the present invention is independent of the load frequency controller, which simplifies the design of the controller and reduces the complexity of solving the controller strategy. At the same time, the design of the energy storage output strategy can consider more complex constraints to achieve accurate energy storage system output control.

[0079] (3) The present invention introduces random network distillation technology to generate new evaluation indicators for reward function design, which encourages the intelligent agent to extensively explore the unknown state space and thus solve the optimal control strategy.

[0080] (4) The present invention can better coordinate the proportion of external rewards and internal rewards through a dynamic weight adaptive adjustment method, thereby achieving the best frequency modulation effect and the optimal state of charge level of the energy storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is an overall flow chart of the load frequency control method according to an embodiment of the present invention;

[0082] Figure 2 This is a load frequency control framework diagram of an embodiment of the present invention;

[0083] Figure 3is a curve of dynamic gain gain1 change in an embodiment of the present invention;

[0084] Figure 4 This is the dynamic gain gain2 change curve of the embodiment of the present invention:

[0085] Figure 5 is a load variation diagram of an embodiment of the present invention;

[0086] Figure 6 is a frequency deviation comparison curve diagram of an embodiment of the present invention;

[0087] Wherein, curve 1 represents the method of the present invention, and curve 2 represents the fixed ratio PI method;

[0088] Figure 7 1 is a graph showing a comparison of output power of a thermal power unit according to an embodiment of the present invention;

[0089] Wherein, curve 1 represents the method of the present invention, and curve 2 represents the fixed ratio PI method;

[0090] Figure 8 is a graph comparing the output power of the energy storage system according to an embodiment of the present invention;

[0091] Wherein, curve 1 represents the method of the present invention, and curve 2 represents the fixed ratio PI method;

[0092] Fig. 9 is a SOC comparison curve diagram of the energy storage system according to an embodiment of the present invention;

[0093] Wherein, curve 1 represents the method of the present invention, and curve 2 represents the fixed ratio PI method; DETAILED DESCRIPTION

[0094] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0095] Embodiment 1

[0096] like Figure 1 to Figure 2 As shown, a deep reinforcement learning load frequency control method for energy storage-assisted thermal power units includes the following steps:

[0097] S1. Establish a deep reinforcement learning environment for power system load frequency control problems to simulate the characteristics and control requirements of each component in the actual power system;

[0098] S2. Design an output control strategy for the lithium-ion battery energy storage system in the power system components that takes into account the power system frequency deviation and the state of charge (SOC) of the energy storage system;

[0099] S3. Model the load frequency control problem of the power system as a Markov decision model, and combine the Markov decision model with the output control law of the energy storage system to build a deep reinforcement learning load frequency control framework based on the Actor-Critic architecture;

[0100] S4. Combining the DDPG algorithm and random network distillation technology, a dual-output load frequency controller (agent) is designed, including the agent input state, the agent output action and the multi-objective dynamic adaptive reward function.

[0101] S5. Based on the external rewards and internal rewards in the multi-objective dynamic adaptive reward function, design an agent (controller) solution algorithm based on dynamic weight adjustment.

[0102] S6. A computer device, characterized in that it comprises a memory and a processor; the memory is used to store computer programs; the processor is used to operate according to the instructions to execute the steps of the method described in any one of claims S1 to S5.

[0103] S7. A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the method described in any one of claims S1 to S5 are implemented.

[0104] Furthermore, in step S1, the load frequency control system transfer function model is constructed using the MATLAB toolbox. This example simulates the random uncontrollability brought about by the grid connection of new energy by introducing thermal power units, energy storage systems, photovoltaic power generation and wind power generation systems.

[0105] S11: Establish the transfer function model of the thermal power unit as follows:

[0106]

[0107] Among them, T g is the time constant of the thermal power unit speed regulator, T CH , T RH , T HP They are the turbine time constant, reheater time constant and reheater gain respectively.

[0108] S12: The transfer function model of the energy storage system is expressed as:

[0109]

[0110] Among them, T e is the energy storage output response time constant.

[0111] S13: Establish a transfer function model for the photovoltaic system, the formula is as follows:

[0112]

[0113] Among them, K PVi and T PVi represent the gain and time constant of the photovoltaic system respectively.

[0114] S14: Establish a transfer function model of the wind power generation system, the formula is as follows:

[0115]

[0116] Among them, K Wi and T Wi Represent the gain and time constant of the wind power generation system respectively.

[0117] S15: Establish the transfer function model between the generator and the load, the formula is as follows:

[0118]

[0119] Among them, M and D are the system rotational inertia and damping coefficient respectively.

[0120] Furthermore, the method for constructing the output control law of the energy storage system is as follows: first determine the position of the system frequency deviation. When the system frequency deviation crosses the frequency dead zone, it indicates that the system has a frequency regulation demand, and at this time the energy storage system should operate in the normal adaptive frequency regulation mode. According to the size of the frequency deviation and considering the real-time SOC state of the energy storage, determine the charging and discharging law of the energy storage at this time. When the system frequency deviation is in the frequency regulation dead zone, and the energy storage SOC state is not good at this time, the energy storage system should operate in the abnormal energy storage self-recovery working mode. The energy storage takes the restoration of SOC as the first output target, and makes full use of the remaining frequency regulation standby capacity of conventional units, taking into account the operating state restrictions of the power grid, and restores the energy storage SOC state.

[0121] Furthermore, the deep reinforcement learning load frequency control framework based on the Actor-Critic architecture is as follows Figure 2 As shown in the figure, the designed control framework adopts an artificial intelligence controller obtained by deep reinforcement learning, in which the control module adopts a load frequency controller designed by the DDPG algorithm integrating random network distillation technology. The controller collects the grid frequency deviation signal and the energy storage battery SOC signal in real time, and guides the controller action network to output action signals a1 and a2 acting on the conventional unit and the energy storage battery respectively according to the reward function, so as to realize the load frequency control of the grid.

[0122] Furthermore, the dual-output load frequency controller includes three parts: agent input state, agent output action and multi-objective dynamic adaptive reward function.

[0123] State S: In the power system, the state observed by the intelligent agent includes the real-time SOC of the energy storage system and the frequency deviation Δf.

[0124] Action A: For the agent, its task is to control the power output of the energy storage system and the thermal power unit. The action set is defined as:

[0125] A=[a1,a2]

[0126] Among them, a1 acts on the thermal power unit and a2 acts on the energy storage system.

[0127] Multi-objective dynamic adaptive reward function: The primary goal of the load frequency control system is to quickly suppress system frequency oscillations and ensure the safe and stable operation of the system. Secondly, make full use of the rapid response characteristics of energy storage to participate in system frequency regulation, while ensuring that the SOC of the energy storage system is in an appropriate range. Finally, on the basis of rapid frequency regulation, the intelligent agent should fully explore the unknown state of the power system environment and improve the environmental adaptability of the controller. Therefore, the following multi-objective dynamic adaptive reward function is designed:

[0128]

[0129] in, It represents the external reward value designed according to the real-time status feedback of the power grid, which is a true reflection of the system frequency deviation. represents the internal reward value, that is, the evaluation value of the random network distillation algorithm on the novelty of the current state. λ1 and λ2 represent the two objectives respectively. and The weight coefficient is a positive number. Defined as:

[0130]

[0131] Among them, when |Δf| is in the frequency regulation dead zone [0, α1], the system frequency meets the minimum error requirement for normal operation, so the maximum reward given to the agent is 0; when |Δf| is in the interval [α1, α2], [α2, α3] and the interval (α3, +∞), it indicates that the agent has not found the optimal strategy and there is a power shortage in the power grid system. At this time, the agent will receive a corresponding negative reward, that is, a penalty term; among them, β1>0, β2>0 and β3>0 are the weights corresponding to the reward functions of each control area; C is the end of the round mark, and γ>0 is the weight coefficient.

[0132] In order to ensure that the energy storage state of charge is always maintained at a good level and to ensure the effectiveness of the training round, the round end penalty γC is introduced into the reward function. It means that during the round training process, when the system frequency deviation is too large or the energy storage system SOC is low, a large negative reward value is given to the agent, ending the round training and entering the next round. The round end flag is defined as:

[0133] C=1,|Δf|>α4 or SOC<α5

[0134] Among them, α4 and α5 are the values ​​of the constraint conditions.

[0135] Furthermore, the internal reward of the multi-objective dynamic adaptive reward function It is obtained by the random network distillation algorithm, which includes the following steps:

[0136] S51, Environmental state feature extraction based on random network distillation:

[0137] Assume that the state of the power system at time t is s t , the state at time t+1 is s t+1 When state s t+1 Input to the random network, the prediction network gives the state prediction features The target network gives the state target feature f(s t+1 ). The random network consists of a prediction network and a target network. Each network consists of two fully connected layers and two RELU layers, and their parameters are and θ f .

[0138] S52, internal reward solution based on state characteristics:

[0139] Status prediction features For state s t+1 The novelty prediction value of the state target feature f(s t+1 ) is state s t+1 The prediction network is trained to predict the output of the target network. The mean squared error between the predicted and target values ​​represents the internal reward term:

[0140]

[0141] Furthermore, the weight coefficients λ1 and λ2 of the two objective functions are expressed as follows:

[0142] System frequency deviation average value Δf aveIt is an intuitive reflection of the frequency modulation effect of the control strategy. If there is a large frequency error, then it is particularly important to choose appropriate control measures to reduce it. On the other hand, if there is a small deviation, then the correction of this indicator is not so important and other goals can be given priority. External reward goals The weight coefficient is defined as:

[0143]

[0144] Among them, gain1 represents the dynamic gain of the weight, represents a positive constant weight, which represents the relative priority of the corresponding target. gain1 is determined by the following formula:

[0145]

[0146] Where Δf max Denotes the maximum acceptable frequency deviation of the power system in steady state, Δf ave represents the system average frequency deviation at the end of the DDPG agent training round, and h1 represents the gain change rate. The curve of gain1 is as follows Figure 3 shown.

[0147] The error between the predicted value and the target value of the random network distillation model is an objective evaluation of the network prediction effect. Control measures should be taken to reduce it when the error is large, and other goals should be given priority when the error is small. Internal reward target The weight coefficient is defined as:

[0148]

[0149] Among them, gain2 represents the dynamic gain of the weight, represents a positive constant weight, which represents the relative priority of the corresponding target. gain2 is determined by the following formula:

[0150]

[0151] in, represents the maximum acceptable error between the predicted value and the target value of the random network distillation model, represents the average error between the predicted value and the target value of the random network distillation model at the end of the DDPG agent round training, and h2 represents the gain change rate. The gain2 curve is as follows Figure 4 shown.

[0152] Furthermore, based on the external rewards and internal rewards in the multi-objective dynamic adaptive reward function, a multi-objective deep reinforcement learning solution algorithm based on dynamic weight adjustment is designed, including:

[0153] Initialize constant weights for multi-objective dynamic adaptive reward functions Deep reinforcement learning network parameters, random distillation network parameters θ f , select an appropriate maximum acceptable steady-state frequency deviation Δf max and the maximum acceptable error of the random network distillation model

[0154] The agent interacts with the environment, the action network is used to output control actions a1 and a2 to act on the power system, and the evaluation network evaluates the reward function and The action value q e and q i , and the system gives the reward value R of the current action e and R i ;

[0155] Based on the evaluation action value, the multi-objective weighted loss function is calculated, the evaluation network parameters are updated by minimizing the weighted loss function, the action network parameters are updated by the gradient descent method, and the random distillation network is updated by minimizing the mean square error between the predicted value and the target value; the multi-objective weighted loss function is expressed as follows:

[0156]

[0157] in It represents the target evaluation value of the evaluation network for the action and state of the next time step, and N represents the number of random sampling samples.

[0158] The agent repeatedly interacts with the environment until the round ends, and the average frequency deviation Δf in the round is calculated ave , the average error between the predicted value and the target value of the random network distillation model And determine Δf ave and If the error is large, adjust the increase rate of dynamic gains h1 and h2, calculate and update the values ​​of weight coefficients λ1 and λ2. If the error is small, fine-tune the weight coefficients to obtain the best frequency modulation performance.

[0159] The agent continues to interact with the power system environment in the next round until the multi-objective dynamic adaptive reward function R t The cumulative reward converges to 0, which means that the agent has solved the optimal control strategy.

[0160] In order to verify the beneficial effects of the deep reinforcement learning load frequency control method for energy storage-assisted thermal power units proposed in the present invention, the following simulation experiment was conducted. The reinforcement learning training environment of the present invention is built using Matlab / Simulink. The rated power of the system thermal power unit is 1000MW, the frequency regulation reserve capacity is -50~50MW, the climbing rate is 30MW / min, and the rated power and capacity of the energy storage are 12.5MW / 2.5. The system simulation parameters are shown in Table 1 based on the rated frequency of 50Hz and the maximum rated capacity of the unit.

[0161] Table 1 Simulation parameters

[0162] parameter value parameter value parameter value parameter value <![CDATA[T g / s]]> 0.08 <![CDATA[K W1 ]]> 1 <![CDATA[SOC l ]]> 0.20 <![CDATA[α5]]> 0.2 <![CDATA[T RH / s]]> 10 <![CDATA[K W2 ]]> 1.4 <![CDATA[SOC0]]> 0.45 <![CDATA[β1]]> 1 <![CDATA[T CH / s]]> 0.3 <![CDATA[T W1 / s]]> 1.25 <![CDATA[SOC1]]> 0.55 <![CDATA[β2]]> 5 <![CDATA[F HP / s]]> 0.5 <![CDATA[T W2 / s]]> 0.041 <![CDATA[SOC h ]]> 0.75 <![CDATA[β3]]> 20 <![CDATA[T e / s]]> 0.04 <![CDATA[K PV1 ]]> -18 <![CDATA[α1]]> 0.003 γ -40 <![CDATA[D / (pu·Hz -1 )]]> 1 <![CDATA[K PV2 ]]> 900 <![CDATA[α2]]> 0.005 n 10 <![CDATA[M / (pu·Hz -1 )]]> 10 <![CDATA[T PV1 / s]]> 100 <![CDATA[α3]]> 0.01 <![CDATA[n1]]> 20 <![CDATA[P r / (MW·h)]]> 2.5 <![CDATA[T PV2 / s]]> 50 <![CDATA[α4]]> 0.2 <![CDATA[n2]]> 10

[0163] according to Figure 3 The load disturbance is used to simulate the load change of the real power system, and the simulation results are as follows Figures 5 to 9 As shown in the figure, it can be seen that the energy storage-assisted thermal power unit deep reinforcement learning load frequency control method proposed in the present invention can dynamically adjust the active output of the energy storage system and the thermal power unit compared with the traditional fixed-ratio PI control frequency modulation, and only requires less energy storage output to achieve safe and stable operation of the power system. In terms of energy storage SOC maintenance, the present invention has a better SOC maintenance effect, better reserves capacity for subsequent frequency modulation tasks, and extends the frequency modulation time of the energy storage system.

[0164] In summary, it can be seen from the simulation diagram that the designed energy storage-assisted thermal power unit deep reinforcement learning load frequency control method can dynamically adjust the output of the energy storage system and the thermal power unit, quickly suppress the frequency oscillation of the power system, and ensure the safety and smooth operation of the system. In addition, the optimal coordination between multiple objectives is achieved through the dynamic weight adjustment algorithm. The effectiveness of the energy storage-assisted thermal power unit deep reinforcement learning load frequency control method proposed in the present invention is verified.

[0165] The present invention also provides a computer device, including a processor and a storage medium; the storage medium is used to store instructions; the processor is used to operate according to the instructions to execute the steps of the method described in Example 1.

[0166] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method described in the first embodiment are implemented.

[0167] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A deep reinforcement learning load frequency control method for energy storage-assisted thermal power units, characterized in that: The following steps are involved: S1. Establish a deep reinforcement learning environment for power system load frequency control problems to simulate the characteristics and control requirements of each component in the actual power system; S2. Design an energy storage system output control strategy that takes into account the frequency deviation of the power system and the state of charge (SOC) of the energy storage system for the lithium-ion battery energy storage system in the power system components; S3. Model the load frequency control problem of the power system as a Markov decision model, and combine the Markov decision model with the output control law of the energy storage system to build a deep reinforcement learning load frequency control framework based on the Actor-Critic architecture; S4. Combining deep deterministic policy gradient (DDPG) and random network distillation technology, a dual-output load frequency controller (agent) is designed, including agent input state, agent output action and multi-objective dynamic adaptive reward function. S5. Based on the external rewards and internal rewards in the multi-objective dynamic adaptive reward function, design an agent (controller) solution algorithm based on dynamic weight adjustment. S6. A computer device, characterized in that it comprises a memory and a processor; the memory is used to store computer programs; the processor is used to operate according to the instructions to execute the steps of any one of the methods S1 to S5. S7. A computer-readable storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of any one of the methods described in S1 to S5 are implemented.

2. According to claim 1, a method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units is characterized in that: In step S1, a general deep reinforcement learning environment for load frequency control model is constructed, which can accurately simulate the characteristics of the real power frequency regulation system. The model components of the load frequency control model include thermal power units, lithium-ion battery energy storage systems, photovoltaic power generation, wind power generation and loads.

3. According to claim 1, a method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units is characterized in that: In step S2, the output control strategy of the energy storage system includes two modes: normal adaptive frequency regulation and abnormal energy storage self-recovery. 1) Normal adaptive frequency modulation mode: When the system frequency deviation exceeds the frequency dead zone, it indicates that the system has a frequency modulation demand at this time. The charging and discharging rules of the energy storage at this time should be determined according to the energy storage operation characteristics and the real-time SOC state of the energy storage. c and P d . Among them, P r is the rated charge and discharge power of energy storage; SOC l , SOC h , SOC0 and SOC1 represent different partition boundaries of energy storage charge state, which are obtained through parameter identification according to the operating characteristics of the energy storage system; n is an adjustable factor determined according to the operating characteristics of the energy storage battery. 2) Abnormal energy storage self-recovery mode: When the system frequency deviation is in the frequency regulation dead zone, and the energy storage SOC state is not good at this time, the energy storage takes the recovery of SOC as the first output target. The controller uses the remaining frequency regulation reserve capacity of the conventional unit and takes into account the grid operation state restrictions to restore the energy storage SOC state. The following SOC self-recovery charging and discharging rules P are designed c1 and P d1 : Among them, n1 and n2 are adjustable factors, which can be determined according to the operating characteristics of the energy storage battery.

4. According to claim 1, a method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units is characterized in that: In step S3, the steps of modeling the power system load frequency control problem as a Markov decision model are as follows: The Markov decision process consists of a four-tuple<S,A,P,R> It can be described as follows, where S′ represents the state set; A represents the action set; P represents the state transition function; and R represents the reward function. In each interaction between the agent and the environment, the power system is regarded as the environment that the agent interacts with. First, the agent observes the system state s t , based on the current strategy, give action a t ; Then, the system performs action a t And update the system's operating status to s t+1 , and returns the agent’s reward R for the action performed t ; Finally, the agent collects all interaction information and completes the iteration and update of its own strategy until the strategy converges to the optimal one.

5. According to claim 1, a method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units is characterized in that: In step S3, a load frequency control framework is constructed. The framework uses the power system model as the environment, and the system frequency deviation signal and the energy storage system SOC signal given by the environment as the controller input. According to the input information, the intelligent agent outputs two frequency modulation actions a1 and a2, which act on the thermal power unit and the energy storage system respectively. The a1 signal is superimposed with the adaptive factor and acts on the thermal power unit model. The a2 signal acts on the energy storage system output strategy module to generate energy storage active output to balance the load changes and realize the load frequency control of the power system. The energy storage output strategy module can independently control the active output and SOC self-recovery of the energy storage system. It avoids the use of energy storage output as a constraint condition for controller design in existing methods, which greatly simplifies the design of the controller. In addition, the output limit considered in the existing method is relatively simple, which only includes the upper and lower bounds of SOC, and does not consider the output characteristics of different energy storage SOC intervals.

6. The method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units according to claim 1 is characterized in that: In the step S4, a dual-output load frequency controller is designed, including an agent input state, an agent output action, and a multi-objective dynamic adaptive reward function. State S: In the power system, the state observed by the intelligent agent includes the real-time SOC of the energy storage system and the frequency deviation Δf. Action A: For the agent, its task is to control the power output of the energy storage system and the thermal power unit. The action set is defined as: A=[a1,a2] Among them, a1 acts on the thermal power unit and a2 acts on the energy storage system. Multi-objective dynamic adaptive reward function: The primary goal of the load frequency control system is to quickly suppress system frequency oscillations and ensure the safe and stable operation of the system. Secondly, make full use of the rapid response characteristics of energy storage to participate in system frequency regulation, while ensuring that the SOC of the energy storage system is in an appropriate range. Finally, on the basis of rapid frequency regulation, the intelligent agent should fully explore the unknown state of the power system environment and improve the environmental adaptability of the controller. Therefore, the following multi-objective dynamic adaptive reward function is designed: in, It represents the external reward value designed according to the real-time state feedback of the power grid, which is a true reflection of the system frequency deviation. represents the internal reward value, that is, the evaluation value of the random network distillation algorithm on the novelty of the current state. λ1 and λ2 represent the two objectives respectively. and The weight coefficient is a positive number. Defined as: Among them, α1, α2 and α3 are the partition values ​​of frequency deviation; β1>0, β2>0 and β3>0 are the weights corresponding to the reward functions of each control area; C is the end of the round mark, and γ>0 is the weight coefficient. In order to ensure that the energy storage state of charge is always maintained at a good level and to ensure the effectiveness of the training round, the round end penalty γC is introduced into the reward function. It means that during the round training process, when the system frequency deviation is too large or the energy storage system SOC is low, a large negative reward value is given to the agent, ending the round training and entering the next round. The round end flag is defined as: C=1,|Δf|>α4 or SOC<α5 Among them, α4 and α5 are the values ​​of the constraint conditions.

7. A method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units according to claim 1, characterized in that: In step S4, the internal reward of the multi-objective dynamic adaptive reward function It is obtained by the random network distillation algorithm, which includes the following steps: S51, Environmental state feature extraction based on random network distillation: Assume that the state of the power system at time t is s t , the state at time t+1 is s t+1 When state s t+1 Input to the random network, the prediction network gives the state prediction features The target network gives the state target feature f(s t+1 ). The random network consists of a prediction network and a target network. Each network consists of two fully connected layers and two RELU layers, and their parameters are and θ f . S52, internal reward solution based on environmental state characteristics: Status prediction features For state s t+1 The novelty prediction value of the state target feature f(s t+1 ) is state s t+1 The prediction network is trained to predict the output of the target network. The mean squared error between the predicted and target values ​​represents the internal reward term:

8. The method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units according to claim 1 is characterized in that: In step S5, the weight coefficients λ1 and λ2 of the two objective functions are expressed as follows: System frequency deviation average value Δf ave It is an intuitive reflection of the frequency modulation effect of the control strategy. If there is a large frequency error, it is particularly important to select appropriate control measures to reduce it. On the other hand, if there is a small deviation, indicating that the system frequency error is already within the steady-state range, then the correction of this indicator is not so important and other goals can be given priority. External reward goals The weight coefficient is determined by the following formula: Among them, gain1 represents the dynamic gain of the weight, represents a positive constant weight, which represents the relative priority of the corresponding target. gain1 is determined by the following formula: Where Δf max Denotes the maximum acceptable frequency deviation of the power system in steady state, Δf ave represents the system average frequency deviation at the end of the DDPG agent round training, and h1 represents the gain change rate. The error between the predicted value and the target value of the random network distillation model is an objective evaluation of the network prediction effect. When the error is large, control measures should be taken to improve the network prediction effect. When the error is small, other goals should be given priority. Internal reward goal The weight coefficient is defined as: Among them, gain2 represents the dynamic gain of the weight, represents a positive constant weight, which represents the relative priority of the corresponding target. gain2 is determined by the following formula: in, represents the maximum acceptable error between the predicted value and the target value of the random network distillation model, represents the average error between the random network distillation model prediction value and the target value at the end of the DDPG agent round training, and h2 represents the gain change rate.

9. The method for deep reinforcement learning load frequency control of energy storage-assisted thermal power units according to claim 1 is characterized in that: In step S5, based on the external rewards and internal rewards in the multi-objective dynamic adaptive reward function, a multi-objective deep reinforcement learning solution algorithm based on dynamic weight adjustment is designed. It is characterized by: Initialize constant weights for multi-objective dynamic adaptive reward functions Deep reinforcement learning network parameters, random distillation network parameters θ f , select an appropriate maximum acceptable steady-state frequency deviation Δf max and the maximum acceptable error of the random network distillation model The agent interacts with the environment, the action network is used to output control actions a1 and a2 to act on the power system, and the evaluation network evaluates the reward function and The action value q e and q i , and the system gives the reward value R of the current action e and R i ; Based on the evaluation action value, the multi-objective weighted loss function is calculated, the evaluation network parameters are updated by minimizing the weighted loss function, the action network parameters are updated by the gradient descent method, and the random distillation network is updated by minimizing the mean square error between the predicted value and the target value; the multi-objective weighted loss function is expressed as follows: in It represents the target evaluation value of the evaluation network for the action and state of the next time step, and N represents the number of random sampling samples. The agent repeatedly interacts with the environment until the round ends, and the average frequency deviation Δf in the round is calculated ave , the average error between the predicted value and the target value of the random network distillation model And determine Δf ave and If the error is large, adjust the increase rate of dynamic gains h1 and h2, calculate and update the values ​​of weight coefficients λ1 and λ2. If the error is small, fine-tune the weight coefficients to obtain the best frequency modulation performance. The agent continues to interact with the power system environment in the next round until the multi-objective dynamic adaptive reward function R t The cumulative reward converges to 0, which means that the agent has solved the optimal control strategy.

10. A computer device and a computer readable storage medium, characterized in that: include: Data acquisition module: collects real-time operating environment information of the power system; Calculation module: takes environmental information as input, calculates the action value function under different states, and further obtains the loss function for neural network update; Output module: outputs control actions based on environmental input and action value function estimation; Storage module: used to store computer programs and trained neural network information.

Citation Information

Patent Citations

  • Power distribution network optimization method based on multi-agent depth determination strategy gradient algorithm

    CN113972645A

  • Deep learning multi-agent micro-grid cooperative control method based on double neural networks

    CN115333143A

  • New energy station frequency control method based on deep reinforcement learning

    CN116436029A

Cited By

  • Resource adjustment method and system based on power grid frequency deviation

    CN120497971A

  • Resource regulation method and system based on power grid frequency deviation

    CN120497971B

  • Hydrogen energy storage auxiliary power system control method and system based on dynamic reward DDPG

    CN120601455A

  • Energy management intelligent control method and system based on reinforcement learning algorithm

    CN120767858A

  • AI-driven power system dynamic frequency stability optimization scheduling method, device and equipment and storage medium

    CN121097845A