Shared energy storage frequency adjusting method and system based on deep reinforcement learning

Abstract

The invention discloses a shared energy storage frequency adjusting method and system based on deep reinforcement learning. The frequency regulation method comprises the following steps that an LFC model of multiple microgrids is established, frequency response and power balance under physical constraints and frequency modulation constraints of the multiple microgrids are simulated according to the LFC model, and the multiple microgrids comprise a shared energy storage system; establishing an objective function satisfying the power balance in consideration of the frequency response and the cost; and establishing a frequency modulation framework based on intelligent power generation control of the DRL, generating a control instruction meeting the target function according to the state information of the LFC model based on a Markov decision, and executing the control instruction by the frequency modulation framework to perform frequency modulation. The method is specially used for the micro-grid with the shared energy storage system, aims to improve the performance and economical efficiency of frequency regulation through an intelligent optimization algorithm, and has the characteristics of cost reduction and high efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems and smart grids, and in particular to a shared energy storage frequency regulation method and system based on deep reinforcement learning. Background Art

[0002] In order to promote the consumption of large-scale renewable energy, various types of microgrids have been widely established. Microgrids effectively improve energy utilization efficiency and system flexibility by integrating renewable energy into local power systems. However, due to the randomness and intermittency of renewable energy, the frequency stability problem of microgrids needs to be solved urgently. Existing frequency regulation methods mainly rely on traditional automatic generation control AGC and model-based control strategies. However, these methods rely on precise mathematical models, require accurate parameters and complex calculations, and are difficult to cope with uncertainties and dynamic changes in actual operation.

[0003] In addition, traditional control strategies have a slow response speed when dealing with rapid frequency changes, making it difficult to meet the needs of high-frequency fluctuations, and most of them ignore the economic efficiency of system operation, resulting in high overall operating costs. At the same time, with the rise of the sharing economy, a variety of sharing models have begun to emerge in various industries, and the energy field is no exception. In the power system, the shared energy storage system (SES), as an important part of frequency regulation, has broad practical application potential. The shared energy storage system achieves optimal allocation and efficient utilization of resources by integrating decentralized energy storage resources. Through the sharing model, not only can the cost of individual users be reduced, but also the utilization and economy of the energy storage system can be improved, and the flexibility and stability of the power grid can be enhanced. However, there is currently no specific method for frequency regulation of multiple microgrids equipped with shared energy storage systems. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a cost-effective and efficient shared energy storage frequency regulation method and system based on deep reinforcement learning, which is specifically designed for microgrids with shared energy storage systems, and aims to improve the performance and economy of frequency regulation through intelligent optimization algorithms. A load frequency control LFC model and an integrated benefit IB objective function are constructed, and the shared energy storage system SES and deep reinforcement learning DRL technology are used in a multi-microgrid system to improve the performance and economy of frequency regulation. At the same time, the TD3 algorithm is introduced in the solution process to achieve the optimal allocation of intelligent power generation control SGC commands.

[0005] Technical solution: The shared energy storage frequency regulation method based on deep reinforcement learning described in the present invention comprises the following steps:

[0006] Establishing an LFC model of multiple microgrids, and simulating frequency response and power balance under physical constraints and frequency regulation constraints of the multiple microgrids according to the LFC model, wherein the multiple microgrids include a shared energy storage system;

[0007] Establishing an objective function satisfying the power balance by considering the frequency response and cost;

[0008] A frequency modulation framework is established based on the intelligent power generation control of DRL, and a control instruction satisfying the objective function is generated according to the state information of the LFC model based on Markov decision making, and the frequency modulation is performed by executing the control instruction through the frequency modulation framework.

[0009] Furthermore, the physical constraint is established based on the actual output power of the shared energy storage system, wherein the actual output power formula is as follows:

[0010]

[0011] The formula for the physical constraint is as follows:

[0012]

[0013] In the formula, ΔP sES (s) is the charging power change of the shared energy storage system sES; represents the proportional gain of the i-th energy storage unit; F i (s) is the frequency adjustment transfer function, F1(s) = G b (s), F2(s)=G iac (s); α is a Boolean variable, α=1 means charging, α=0 means discharging; ΔP ord-sES1 It is an SGC command executed by the physical energy storage device PES; It is an SGC command executed by the virtual energy storage device VES; Maximum power for charging sES; is the maximum power discharged by sES; is the maximum down-regulated power of the inverter; The maximum up-regulated power of the inverter; is the maximum charging power of the battery; is the maximum discharge power of the battery; The battery storage energy at time t; is the maximum energy storage capacity of the battery; is the charging power at time t-1; is the discharge power at time t-1; η b is the charge and discharge efficiency of the battery; is the time interval; T A is the current temperature; is the maximum temperature set; is the lowest temperature set.

[0014] Furthermore, the formula of the frequency modulation constraint is as follows:

[0015]

[0016] In the formula, is the minimum power variation of distributed generators; Maximum power change of distributed generators; ΔP d (t) is the power change of the distributed generator at time t; is the maximum change rate of distributed generators; f is the frequency deviation coefficient; f rate is the system rated frequency; is the maximum power transmission capacity of the interconnection line.

[0017] Furthermore, the frequency response is used to describe the power output, including the frequency response on the power generation side and the frequency response on the shared energy storage system side.

[0018] Furthermore, the power balance is used to describe frequency stability. The multi-microgrid includes multiple regions. The specific formula of the power balance of region m is as follows:

[0019] ΔP L,m =ΔP d,m +ΔP sES,m +ΔP PV,m -ΔP tie

[0020] Where ΔP L,m is the load power change of microgrid area m; ΔP d,m is the power change of distributed generators in microgrid area m; ΔP sES,m is the power change of the energy storage system in microgrid area m, including the power output change of the physical energy storage device PES and the virtual energy storage device VES; ΔP PV,m is the power change of the photovoltaic power generation system in microgrid area m; ΔP tie is the power variation transmitted through the tie lines between microgrid areas.

[0021] Furthermore, the multi-microgrid includes a shared energy storage system, loads and power generation equipment. Correspondingly, in the frequency regulation framework, the regulator adjusts the operating parameters of the loads and power generation equipment according to the control instructions, and the shared energy storage aggregator adjusts the charging and discharging power of the shared energy storage system according to the control instructions.

[0022] Furthermore, the formula of the objective function is as follows:

[0023]

[0024] Where X1 represents the sum of variables related to power plant operation cost, fuel cost and other related costs and load power change; X2 represents the sum of the difference between the total orderly dispatch power in the system and the output power of multiple power generation units; X3 represents the comprehensive index of frequency deviation and tie line power deviation, including the penalty items of frequency change and tie line power change; M I M is the first-level mileage of the frequency modulation device; I (t)=|ΔP ord-I (t)-ΔP ord-I (t-1)|;A I =1, I = 1 for DG, I = 2 for SES; G is the total number of frequency modulation devices, G = 2; N = 75 is a settlement cycle; c fuel represents the fuel cost; k ce The coefficient representing the carbon emission cost; c ce represents the carbon emission cost; ΔP d (t) represents the change in load power at time t; ΔP I (t) represents the power change of the power generation unit I at time t; ΔP ord-all (t) represents the total orderly dispatch power change of the system at time t, ΔP ord-all =ΔP ord-d +ΔP ord-sES1 +ΔP ord-sES2 ; μ represents the weight factor; ΔP tie (t) represents the change in the tie line power at time t; k pr represents the proportional coefficient; Δf(t) represents the frequency deviation at time t.

[0025] Further, the control instruction satisfying the objective function is generated according to the state information based on the Markov decision, and the specific process is as follows:

[0026] Based on the LFC model, a state set and an action set are defined, and an actor network is initialized at the same time; a reward function is established according to the objective function; wherein the state set is a set of state information, and the action set is a set of control instructions;

[0027] State information is obtained to determine the current state of the state set. The actor network generates the current control instruction of the action set according to the current state update, and the current control instruction is executed by the frequency modulation framework. After the execution is completed, the reward function evaluates the execution status, re-collects the state information, and repeats the above steps until the objective function is met.

[0028] Furthermore, the TD3 algorithm is introduced when the reward function evaluates the execution status, which specifically includes the following steps:

[0029] Establishing a value function based on the reward function;

[0030] Based on the TD3 algorithm, random noise is added to the actor network to find the optimal strategy for generating control instructions to maximize the value function while satisfying the objective function.

[0031] The shared energy storage frequency regulation system based on deep reinforcement learning described in the present invention includes:

[0032] A multi-microgrid simulation module, used to establish an LFC model of the multi-microgrid, and simulate the frequency response and power balance under the physical constraints and frequency regulation constraints of the multi-microgrid according to the LFC model, wherein the multi-microgrid includes a shared energy storage system;

[0033] An objective function establishment module, used to establish an objective function that satisfies the power balance by considering the frequency response and the cost;

[0034] The frequency modulation module is used to establish a frequency modulation framework based on the intelligent power generation control of the DRL, and to generate a control instruction that satisfies the objective function according to the state information of the LFC model based on Markov decision, and the frequency modulation framework executes the control instruction to perform frequency modulation.

[0035] Beneficial effects: The present invention has the following significant effects: 1. Reduce costs: First, in order to reduce the cost of frequency regulation, the present invention introduces a shared energy storage device SES in multiple microgrids, and specifically establishes and solves a model for multiple microgrids containing shared energy storage devices, constructs a load frequency control LFC model containing a shared energy storage system, describes the frequency response of batteries and inverter air conditioners, simulates the frequency response and power balance of multiple microgrids, improves the utilization rate and economy of energy storage resources, reduces user costs, and realizes efficient energy management; secondly, the present invention considers economic factors and frequency response performance to set an integrated benefit IB objective function, and the generated control instructions are constrained by the objective function, while ensuring frequency stability, optimizing system operating costs and improving overall economic benefits; 2. High efficiency; The present invention is based on deep reinforcement learning DRL. A frequency regulation method is established, and the DRL technology is applied to the frequency regulation of multiple microgrids. By constructing a DRL-based intelligent power generation control SGC command allocation framework, dynamic frequency regulation can be achieved in continuous state and action space, thereby improving the accuracy and response speed of frequency regulation. The DRL method adapts to the complex environment of source and load uncertainty by learning historical data of state sets and action sets, avoiding dependence on precise models and enhancing the practicality and robustness of the method. In addition, the TD3 algorithm is introduced in SGC, and the entire process of frequency regulation is optimized by adopting a double-delayed deep deterministic policy gradient TD3 algorithm. The robustness and accuracy are also enhanced by introducing noise and updating parameters. In general, the efficiency is improved, and the frequency stability of microgrids under the condition of a high proportion of renewable energy access is improved, ensuring the safe and reliable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a schematic diagram of a multi-microgrid architecture including sES;

[0037] Figure 2 It is a schematic diagram of the LFC model and frequency regulation framework of multiple microgrids including SES;

[0038] Figure 3 Schematic diagram of the overall frequency regulation process of multiple microgrids including SES;

[0039] Figure 4 An embodiment of a multi-microgrid including SES established for verification;

[0040] Figure 5 is the average reward curve of offline learning during the validation process. DETAILED DESCRIPTION

[0041] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0042] See also Figures 1 to 3 As shown, the shared energy storage frequency regulation method based on deep reinforcement learning described in the present invention includes the following steps:

[0043] Establishing an LFC model of multiple microgrids, and simulating frequency response and power balance under physical constraints and frequency regulation constraints of the multiple microgrids according to the LFC model, wherein the multiple microgrids include a shared energy storage system;

[0044] Establishing an objective function satisfying the power balance by considering the frequency response and cost;

[0045] A frequency modulation framework is established based on the intelligent power generation control of DRL, and a control instruction satisfying the objective function is generated according to the state information of the LFC model based on Markov decision making, and the frequency modulation is performed by executing the control instruction through the frequency modulation framework.

[0046] The shared energy storage frequency regulation system based on deep reinforcement learning described in the present invention includes:

[0047] A multi-microgrid simulation module, used to establish an LFC model of the multi-microgrid, and simulate the frequency response and power balance under the physical constraints and frequency regulation constraints of the multi-microgrid according to the LFC model, wherein the multi-microgrid includes a shared energy storage system;

[0048] An objective function establishment module, used to establish an objective function that satisfies the power balance by considering the frequency response and the cost;

[0049] The frequency modulation module is used to establish a frequency modulation framework based on the intelligent power generation control of the DRL, and to generate a control instruction that satisfies the objective function according to the state information of the LFC model based on Markov decision, and the frequency modulation framework executes the control instruction to perform frequency modulation.

[0050] A multi-microgrid consists of multiple areas connected in series by a bus. Each area is an independent microgrid system, which includes a shared energy storage system, loads and power generation equipment. The shared energy storage system consists of physical energy storage equipment and virtual energy storage equipment, which are connected to each microgrid area through wireless communication. The shared energy storage system can adjust and optimize the power demand of multiple microgrids and improve the frequency stability and economy of the entire system. The power generation equipment of the multi-microgrid itself can be distributed generators, renewable energy generators, and combined thermal power systems. The load includes adjustable loads and non-adjustable loads. Please refer to Figure 1 , is an embodiment of a multi-microgrid architecture including a shared energy storage system, wherein the microgrids are connected via a bus and share a centrally managed shared energy storage system.

[0051] The multi-microgrid includes a shared energy storage system, loads and power generation equipment. Correspondingly, in the frequency regulation framework, the regulator adjusts the operating parameters of the loads and power generation equipment according to the control instructions, and the shared energy storage aggregator adjusts the charging and discharging power of the shared energy storage system according to the control instructions. Figure 2 , which is a load frequency control LFC model and frequency regulation framework in multiple microgrids, including the interaction between microgrid areas and shared energy storage systems. The microgrid area combines renewable energy generation and load change △P through regulators, distributed generators and load responses. L Achieve power balance and frequency regulation. The shared energy storage system includes physical energy storage devices and virtual energy storage devices, which receive power distribution instructions. By adjusting the charge and discharge power ΔP sES Participate in frequency regulation. Tie line power ΔP tie Connect different microgrid areas and coordinate power exchange between areas. The overall system realizes dynamic adjustment and optimization of frequency stability through frequency deviation △f feedback control.

[0052] The frequency response is used to describe the power output, including the frequency response on the power generation side and the frequency response on the shared energy storage system side. In this embodiment, the frequency response on the power generation side is mainly based on the frequency response of the inverter AC power, and the frequency response formula is as follows:

[0053]

[0054] In the formula, G iac (s) is the gain function, C(s) is the temperature controller, D(s) is the inverter air conditioning controller involved in frequency modulation, T a (s), T c (s) and Φ iac (s) related to room temperature and thermal deviation;

[0055] The frequency response of the shared energy storage system is mainly based on the frequency response of the battery. The frequency response of the battery is described by the first-order inertial element and the battery gain, as shown in the following formula:

[0056]

[0057] In the formula, ΔP b (s) is the change of power output of the shared energy storage system; G b is the gain function of the shared energy storage system; Δf(s) is the frequency deviation of the shared energy storage system; K b is the gain of the shared energy storage system; T bis the battery time constant; s is the Laplace transform variable.

[0058] The physical constraint is established based on the actual output power of the shared energy storage system, wherein the actual output power formula is as follows:

[0059]

[0060] The formula for the physical constraint is as follows:

[0061]

[0062] In the formula, ΔP sES (s) is the charging power change of the shared energy storage system sES; represents the proportional gain of the i-th energy storage unit; F i (s) is the frequency adjustment transfer function, F1(s) = G b (s), F2(s)=G iac (s); α is a Boolean variable, α=1 means charging, α=0 means discharging; ΔP ord-sES1 It is an SGC command executed by the physical energy storage device PES; It is an SGC command executed by the virtual energy storage device VES; Maximum power for charging sES; is the maximum power discharged by sES; is the maximum down-regulated power of the inverter; The maximum up-regulated power of the inverter; is the maximum charging power of the battery; is the maximum discharge power of the battery; The battery storage energy at time t; is the maximum energy storage capacity of the battery; is the charging power at time t-1; is the discharge power at time t-1; η b is the charge and discharge efficiency of the battery; is the time interval; T A is the current temperature; is the maximum temperature set; is the lowest temperature set.

[0063] The formula for the frequency modulation constraint is as follows:

[0064]

[0065] In the formula, is the minimum power variation of distributed generators; Maximum power change of distributed generators; ΔP d(t) is the power change of the distributed generator at time t; is the maximum change rate of distributed generators; f is the frequency deviation coefficient; f rate is the system rated frequency; is the maximum power transmission capacity of the interconnection line.

[0066] The power balance is used to describe frequency stability. The multi-microgrid includes multiple regions. The specific formula of the power balance of region m is as follows:

[0067] ΔP L,m =ΔP d,m +ΔP sES,m +ΔP PV,m -ΔP tie (6)

[0068] Where ΔP L,m is the load power change of microgrid area m; ΔP d,m is the power change of distributed generators in microgrid area m; ΔP sES,m is the power change of the energy storage system in microgrid area m, including the power output change of the physical energy storage device PES and the virtual energy storage device VES; ΔP PV,m is the power change of the photovoltaic power generation system in microgrid area m; ΔP tie is the power change transmitted through the tie line between microgrid area m and other areas.

[0069] Among them, the tie line power between area m and area l can be described as follows:

[0070]

[0071] Where ΔP tie,m-l is the power change transmitted through the tie line between microgrid area m and area l; T tie,m-l is the time constant of the tie line; Δf m ,Δf is the frequency deviation between microgrid areas m and l.

[0072] The change of the area control error (ACE) in the m region is as follows:

[0073] ΔACE m =ΔP tie +k m Δf m (8)

[0074] Where ΔACE m is the regional control error of microgrid area m; k m is the regulation coefficient of microgrid area m.

[0075] In order to achieve precise optimization, the present invention designs an objective function taking into account the frequency response and cost and satisfying the power balance. The design of the objective function includes the integrated benefit IB of frequency response performance and economy, and comprehensively considers the response speed, frequency deviation, interconnection line power deviation, and the cost of the shared energy storage system. The objective function consists of three parts. The first part X1 uses the carbon emission cost to further compensate for the services provided by sES. The second part X2 uses the error of the device responding to the SGC command to ensure the frequency regulation performance of the device. In the third part X3, ACE is used as a collection of frequency and interconnection line power errors to reflect the ability of multiple microgrids to suppress source load fluctuations. Specifically, the formula of the objective function is as follows:

[0076] f(t)=min(X1+X2+X3)(9)

[0077]

[0078] Where X1 represents the sum of variables related to power plant operating costs, fuel costs and other related costs, as well as load power changes; X2 represents the sum of the differences between the total orderly dispatch power in the system and the output power of multiple power generation units; X3 represents the comprehensive index of frequency deviation and tie line power deviation, including the penalty terms of frequency change and tie line power change; M I M is the first-level mileage of the frequency modulation device; I (t)=|ΔP ord-I (t)-ΔP ord-I (t-1)|;A I =1, I = 1 for DG, I = 2 for SES; G is the total number of frequency modulation devices, G = 2; N = 75 is a settlement cycle; c fuel represents the fuel cost; k ce The coefficient representing the carbon emission cost; c ce represents the carbon emission cost; ΔP d (t) represents the change in load power at time t; ΔP I (t) represents the power change of the power generation unit I at time t; ΔP ord-all (t) represents the total orderly dispatch power change of the system at time t, ΔP ord-all =ΔP ord-d +ΔP ord-sES1 +ΔP ord-sES2 ; μ represents the weight factor; ΔP tie (t) represents the change in the tie line power at time t; k pr represents the proportional coefficient; Δf(t) represents the frequency deviation at time t; the SGC command interval is 4 seconds.

[0079] The objective function established considering IB works better in terms of frequency deviation amplitude and recovery speed. In addition, the tie line power of the IB-based method is 0, so the optimal frequency regulation can be achieved in its microgrid area, avoiding unnecessary tie line power flow, thereby improving the frequency regulation performance. And the main frequency regulation needs are borne by VES and PES, which helps to reduce the fuel and carbon emission costs of DG. In addition, VES is prioritized as a flexibility regulation resource, avoiding the over-construction of PES and reducing maintenance costs. The SGC command allocation method based on IB enhances the economy of frequency regulation. In contrast, other methods all use DG as the main frequency regulation device, and ES is not effectively utilized. Compared with other methods, the maximum improvements of the IB-based method in IBI and OCI are 75% and 81.6%, respectively.

[0080] Solve the control instructions that meet the above objective function for frequency regulation, model the frequency regulation problem as a Markov decision process MDP, and use the TD3 algorithm to optimize it. Through offline learning and online execution, the dynamic allocation of smart power generation control commands SGC is realized.

[0081] The solution process is explained in detail below. Markov decision-making establishes a state set S, an action set A and a reward function r. The state set determines the current state based on the real-time state information of the multi-microgrid collected, and the actor network generates the corresponding current action based on the current state and the action set. The reward function is used to evaluate the quality of the current action. During the iterative update process, the actor network is also constantly iterating parameters to become the corresponding target network and then used to solve the problem, obtaining a better current action as a control instruction. The present invention uses the TD3 algorithm for optimization in the Markov decision process. The TD3 algorithm introduces a dual-target network, namely two critic networks, which are used to estimate the value of the action. Later, two delayed update target networks are introduced to smooth the update value to reduce the deviation of the Q value estimation. During the iterative update process, the optimal parameters are continuously found to optimize the action update iteration process. In addition, the concept of experience replay is introduced through the TD3 algorithm to form an experience replay memory D, which is used to store training experience (s t ,a t ,r t ,s t+1 ), and then the robustness of iterative updates is enhanced by randomly extracting samples for testing.

[0082] The main process is as follows:

[0083] 1. Initialization. First, initialize the actor network π based on the LFC model θ , two critic networks Q φ1 and Q φ2They are respectively equipped with parameters θ and φ1, φ2, and the corresponding target network is initialized at the same time. The parameters of the target network correspond to the main network one by one, namely θ' and φ1', φ'2. Finally, the experience replay storage size D is initialized. A total of K trainings are performed, each training is an episode, and each training contains T time steps.

[0084] Adaptive definition: define the state set S and action set A based on the LFC model; set the reward function R based on the objective function. The state set is a collection of state information. In the Markov decision process, the state set S contains the key parameters of the multi-microgrid, which reflect the specific operating status of the multi-microgrid. The state set includes the following variables: frequency deviation Δf(s), which indicates the difference between the system frequency and the rated frequency. The larger the frequency deviation, the more unstable the system. The goal of the control system is to minimize the frequency deviation; the power change ΔP of the energy storage system sES (s) represents the change in the power output or input of the SES shared energy storage system. This is the regulation power provided by the shared energy storage system in frequency regulation. The system will dynamically adjust the energy storage power according to the frequency deviation; the power change of the distributed generator Indicates the change in the power output of the distributed generator, that is, the power adjustment made by the distributed generator according to the load demand and frequency deviation; the power change of the tie line ΔP tie , which indicates the power change transmitted through the interconnection line between microgrid areas. This part of power is used to coordinate the power balance between different microgrids; the charging and discharging status of the battery Indicates the current storage capacity of the energy storage battery, reflecting the charging or discharging capacity of the energy storage system. In frequency regulation, the state of battery charging and discharging determines whether the energy storage system can continue to provide support. Together, these variables constitute a complete description of the state of the LFC system and provide a basis for the decision-making of the intelligent agent.

[0085] Similarly, the action set is a collection of actions, that is, control instructions. These actions control the operation of key devices in the microgrid to achieve the frequency regulation target. The action set includes the following aspects: the charging and discharging power adjustment ΔP of the energy storage system sES (s), indicating that the intelligent agent can adjust the charging and discharging power of the energy storage system and control the power change of the energy storage system according to physical constraints, so that the energy storage system can provide additional support or absorb excess power when the frequency fluctuates; the output power adjustment of the distributed generator ΔP d (t), the agent can also adjust the power output of distributed generators, referring to the power change constraints of distributed generators to ensure that the system can cope with load changes and maintain frequency stability; the tie line power adjustment ΔP tie, which means that the intelligent agent achieves energy balance between different microgrid areas by adjusting the power exchange between microgrid areas, reducing the impact of frequency deviation on system stability. The variables in the above action sets determine the specific operations of the system in the frequency regulation process. The intelligent agent selects the optimal action based on the current state and historical experience to ensure the frequency stability and economy of the system.

[0086] Design of reward function R. The reward function is the feedback obtained by the agent after each time step based on the actions performed and the state changes of the system. The reward function in the present invention comprehensively considers various aspects of frequency regulation: frequency deviation reward, the larger the frequency deviation, the lower the reward, the goal of the agent is to reduce the frequency deviation by controlling the energy storage system and distributed generators; energy storage system economic reward, taking into account the operating cost of the shared energy storage system, especially the costs or losses that may be incurred during the charging and discharging process, the agent needs to balance frequency regulation and economy, and perform effective frequency regulation under the premise of reducing costs as much as possible; interconnection line power error reward, taking into account the interconnection line power deviation between different microgrid areas. If the power exchange between areas is unbalanced, it will lead to overall instability of the system. The agent needs to control this deviation to ensure the power balance of the system. The agent selects the optimal regulation strategy by maximizing this reward function. Among them, the reward function r based on the objective function t (s t ,a t )The specific formula is as follows:

[0087] r t (s t ,a t )=-k1ξ r-1 X r-1 (t)-k2ξ r-2 X r-2 (t)-k3ξ r-3 X r-3 (t)+ζ p (t)(11)

[0088]

[0089] In the formula, k1+k2+k3=1, the ratio of k2 and k3 is fixed to 1 / 3. In order to reflect the effect of economy on frequency modulation, k1 should not be less than 0.5, while k2 has no significant effect on the reward function, and the device response delay is not considered. Therefore, the default configurations of k1, k2, and k3 in this embodiment are 0.6, 0.1, and 0.3;ζ p ={ξ,stL is false||0,stL is true} is the constraint penalty, L is the set of physical constraints and frequency modulation constraints; X r-1 (t), X r-2(t), X r-3 (t) operating costs, performance incentives for equipment response and frequency regulation; c mill ,c fuel ,c ce Represent the prices of mileage compensation, fuel and carbon emissions respectively; k ce is the carbon emission offset factor; μ, k pr are the scaling factors and constants in ACE.

[0090] 2. Iterative update of actions and networks. Obtain state information to determine the current state of the state set, and the actor network generates the current control instructions of the action set based on the current state update, which is executed by the multi-microgrid; after the execution is completed, the reward function evaluates the execution status, re-collects the state information, and repeats the above steps until the objective function is met. In the above process, the TD3 algorithm introduces the concept of experience replay: the state, action, reward and next state of each time step are stored in the experience pool D. Then randomly select small batches of samples from the experience pool D for training, which can avoid the correlation problem between consecutive time steps and make the model more generalizable. Specifically, in each time step t, an action a~π with exploration noise is selected. θ (s)+∈, where ∈ follows a normal distribution with mean 0 and standard deviation σ. Execute the action and receive the reward r and the new state s t+1 . The current state, action, reward and new state (s t ,a t ,r t ,s t+1 ) is stored in the experience replay storage D.

[0091] Among them, the current state is determined by the state information collected by the LFC model, and the formula of the current state is as follows:

[0092] s t ={Δf(t),ΔP tie (t),ΔP I (t),ΔP ord-all (t)} (13)

[0093] Where ΔP ord-all (t) is the distributed generation power command change at time t.

[0094] The current control command is determined by the following formula:

[0095] a t = {ΔP ord-d (t),ΔP ord-sES1 (t),ΔP ord-sES2 (t)}(14)

[0096] In the formula, ΔP ord-d (t),ΔP ord-sES1 (t),ΔP ord-sES2 (t) is the power command change to the inverter, the power command change to the energy storage system 1, and the power command change to the energy storage system 2 at time t.

[0097] Afterwards, random noise is added to the actor network based on the TD3 algorithm to find the optimal strategy for generating control instructions to maximize the value function while satisfying the objective function. Noise is added to update the strategy to reduce the negative impact of overfitting, generate a smooth target strategy and improve robustness. When updating the strategy, the agent will add a small amount of noise to make the strategy update smoother, avoid instability caused by overfitting, and thus improve the robustness of the agent in frequency regulation. Taking strategy π as an example, the process of generating actions by strategy π is as follows The agent's goal is to find the optimal policy π by * To maximize the value function That is π * = argmaxQ π (s t ,a t ). Specifically, randomly extract a small batch M of transfer samples from the experience replay storage D. For each sample extracted, calculate the target action a' and its noisy version a'~π θ′ (s′)+∈, where the noise is clipped to the range [-c,c]. Calculate the target Q value The target Q value is determined by the value function, which is established based on the reward function. t When determining, use the value function Q π (s,a) describes the agent action a t The reward value obtained.

[0098] Among them, the formula for adding random noise when updating the strategy is as follows:

[0099]

[0100] The formula of the value function is as follows:

[0101]

[0102] In the formula, E π () represents the expectation of strategy π.

[0103] Network iterative update: The TD3 algorithm uses the policy gradient method to optimize the parameters of the network. By maximizing the cumulative reward, the agent continuously adjusts its strategy so that it can choose the optimal action in a more efficient way in future frequency adjustments. The comment network parameters φ1, φ2 are updated using the mean square loss function. Delayed update of the actor, critic network, and target network. The goal of the dual-objective network update is to update the network parameters through the Bellman equation so that the target value converges to a more stable level and avoids overestimation. TD3 calculates the loss function by substituting the selected smaller value into the Bellman equation to avoid the possibility of overestimation, as follows:

[0104]

[0105] In the formula, y t represents the target Q value obtained through the target network and the instantaneous reward; Represents the instantaneous reward after taking action at the current moment; represents the current Q value obtained by the critic network; J is the loss function.

[0106] The actor network is updated less frequently than the critic network to ensure that the estimation error is reduced before updating the actor. Every d time steps, the parameters of the actor network are updated using policy gradient ascent. The parameters of the actor network and the critic network are updated as follows:

[0107]

[0108] In the formula, represents the gradient policy associated with the actor network parameters θ, which determines the update direction by maximizing the Q value; is the gradient of the policy π, defining the update rule for θ in the actor network; and Used to update the parameters of the actor network and the key network.

[0109] The target network parameters are updated via smoothing. The parameters of the target critic network and the target actor network are slowly updated as follows:

[0110]

[0111] Where τ is the smoothing factor.

[0112] 3. Online execution and dynamic adjustment. After training is completed, the TD3 algorithm adjusts the frequency of the microgrid system in real time during the online execution phase. At each time step, the agent dynamically selects actions based on the current state (such as frequency deviation, energy storage system state, and distributed generator output). These actions, or control instructions, are used to adjust the output of the energy storage system and power generation equipment to stabilize the system frequency. Figure 3, which is a specific example of the working process of the intelligent generation control SGC command distribution based on deep reinforcement learning DRL in a multi-microgrid system, that is, the frequency regulation process. First, the microgrid operator and the shared energy storage system aggregator monitor the system status, including load, power generation and energy storage status. The intelligent agent Agent receives this status information, generates control instructions based on the internal DRL model, and sends them to the microgrid operator and sES aggregator. The operator adjusts the operating parameters of distributed generators, loads and renewable energy generation, while the sES aggregator adjusts the charging and discharging power of physical and virtual energy storage devices. The system executes these instructions, adjusts power output and frequency stability, and feeds back the response to the intelligent agent. Based on the feedback information, the intelligent agent evaluates the effect of the control strategy through the reward function, and continuously learns and optimizes to improve the accuracy, response speed and economy of frequency regulation, thereby achieving efficient management of the multi-microgrid system.

[0113] The overall system optimizes power supply and improves the system's frequency stability and operating efficiency by dynamically adjusting the output of energy storage equipment and distributed power generation equipment.

[0114] In addition, the present invention is verified. In order to verify the effect of the overall method, a multi-microgrid simulation model with SES is established in Matlab R2021aSimulink software. Figure 4 As shown in the figure, two regional microgrids and SES are interconnected through a bus. Microgrid area 1 consists of distributed power sources, rooftop photovoltaic systems and local loads, and microgrid area 2 consists of distributed power sources, rooftop photovoltaic systems, batteries and local loads. SES consists of VES and PES. The adjustable loads of microgrid areas 1 and 2 are 400kW and 200kW respectively. The PES of microgrid area 2 and SES are both 250kW / 500kWh. The reference power is 4MW. SES only provides frequency support for microgrid area 1 to verify the impact of SES on frequency regulation. The frequency response model is established during the simulation process, and the frequency change is limited to [49.8,50.2]Hz. The detailed parameters of each part of the multi-microgrid are shown in Table 1.

[0115] Table 1 Multi-microgrid system parameters

[0116]

[0117] In this verification, offline training and testing are run on a workstation with an INTEL i9-10900×3.7GHz CPU and 64GB RAM. In offline training, randomly generated renewable energy generation and load data are used as input for agent training. In each SGC command update cycle, the agent updates the current state of the microgrid according to the current state of the microgrid. t , based on DQN, DDPG or TD3 algorithm, action a is givent , Reward t and the next state s t+1 The parameter configurations of the three DRL algorithms are shown in Table 2.

[0118] Table 2 Evaluation indicators and key factors

[0119]

[0120] For the DQN algorithm used, the input is a 6-dimensional state vector and the output is the Q value of the state-action pair. The actions are discretized into 8 values, 4 values, and 5 values ​​respectively. Through the interaction between the agent and the environment, training samples are generated and stored in the experience pool. Then, through a large amount of offline training, the agent obtains the SGC command allocation strategy π(s t |θ π ). The average reward curve of offline learning of multi-microgrid frequency regulation based on DQN, DDPG and TD3 algorithms under the above settings is shown as follows Figure 5 The advantages of TD3 algorithm are Figure 5 The rewards of the TD3 and DDPG algorithms stabilized at around -0.7 after 100 and 1200 episodes, respectively, while the average reward of the DQN algorithm remained unstable after 2000 episodes, with a value less than -4.5. Figure 5 This shows that DQN cannot solve the problem of continuous actions, and the degree of action dispersion determines the training effect of the agent. In addition, it can be seen that DDPG and TD3 performed poorly in the early stage, but as the training progressed, the average reward gradually increased and stabilized. The final performance and stability of TD3 were better than DDPG, while the reward of DQN fluctuated greatly and there was no obvious convergence trend, indicating that it was not as stable and effective as DDPG and TD3 in this task. Therefore, the TD3 algorithm has better robustness and stability in the face of randomly changing data, while the overestimation disadvantage of the DDPG algorithm leads to an unstable training process.

[0121] The frequency modulation method proposed in this invention aggregates the adjustable resources of energy users and reduces the cost of self-built PES; through the power interaction between multiple microgrids, the equipment utilization rate and the overall microgrid economy are improved. Figure 4 In the multi-microgrid shown in Figure 1, three SES configurations are used to test their advantages in frequency regulation. Configuration 1, microgrid areas 1 and 2 have only PES. Configuration 2, microgrid area 1 has SES, and microgrid area 2 has only PES. Configuration 3, microgrid areas 1 and 2 have SES. Microgrid areas without SES mean that when the adjustable load reaches its maximum frequency regulation capability, frequency regulation can only be achieved by building a larger capacity PES or increasing DG output. In a multi-microgrid with SES, such as Figure 5 As shown, when k CW=(0.6,1) IBI has an advantage. CW =(0.6,0.8) when OCI has an advantage. CW =(0.6,0.8), it is the best choice to obtain larger OCI and IBI with smaller performance sacrifice. The effect of kCR on the evaluation indicators is simpler. Better results are obtained when the user responsiveness reaches 60%. As kCR increases, all indicators are improved, and the best results are obtained when VES is fully responsive.

Claims

1. A shared energy storage frequency regulation method based on deep reinforcement learning, characterized in that: The frequency adjustment method comprises the following steps: Establishing an LFC model of multiple microgrids, and simulating frequency response and power balance under physical constraints and frequency regulation constraints of the multiple microgrids according to the LFC model, wherein the multiple microgrids include a shared energy storage system; Establishing an objective function satisfying the power balance by considering the frequency response and cost; A frequency modulation framework is established based on the intelligent power generation control of DRL, and a control instruction satisfying the objective function is generated according to the state information of the LFC model based on Markov decision making, and the frequency modulation is performed by executing the control instruction through the frequency modulation framework.

2. The shared energy storage frequency regulation method based on deep reinforcement learning according to claim 1 is characterized in that: The physical constraint is established based on the actual output power of the shared energy storage system, wherein the actual output power formula is as follows: The formula for the physical constraint is as follows: In the formula, ΔP sES (s) is the charging power change of the shared energy storage system sES; represents the proportional gain of the i-th energy storage unit; F i (s) is the frequency regulation transfer function, F1(s) = Gb(s), F2(s) = Giac(s); α is a Boolean variable, α = 1 means charging, α = 0 means discharging; ΔP ord-sES1 It is an SGC command executed by the physical energy storage device PES; It is an SGC command executed by the virtual energy storage device VES; Maximum power for charging sES; is the maximum power discharged by sES; is the maximum down-regulated power of the inverter; The maximum up-regulated power of the inverter; is the maximum charging power of the battery; is the maximum discharge power of the battery; The battery storage energy at time t; is the maximum energy storage capacity of the battery; is the charging power at time t-1; is the discharge power at time t-1; η b is the charge and discharge efficiency of the battery; is the time interval; T A is the current temperature; is the maximum temperature set; is the lowest temperature set.

3. The shared energy storage frequency regulation method based on deep reinforcement learning according to claim 1 is characterized in that: The formula for the frequency modulation constraint is as follows: In the formula, is the minimum power variation of distributed generators; Maximum power change of distributed generators; ΔP d (t) is the power change of the distributed generator at time t; is the maximum change rate of distributed generators; f is the frequency deviation coefficient; f rate is the system rated frequency; is the maximum power transmission capacity of the interconnection line.

4. The shared energy storage frequency adjustment method based on deep reinforcement learning according to claim 1 is characterized in that: The frequency response is used to describe the power output, including the frequency response on the power generation side and the frequency response on the shared energy storage system side.

5. The shared energy storage frequency regulation method based on deep reinforcement learning according to claim 1 is characterized in that: The power balance is used to describe frequency stability. The multi-microgrid includes multiple regions. The specific formula of the power balance of region m is as follows: ΔP L,m =ΔP d,m +ΔP sES,m +ΔP PV,m -ΔP tie In the formula, ΔP L,m is the load power change of microgrid area m; ΔP d,m is the power change of distributed generators in microgrid area m; ΔP sES,m is the power change of the energy storage system in microgrid area m, including the power output change of the physical energy storage device PES and the virtual energy storage device VES; ΔP PV,m is the power change of the photovoltaic power generation system in microgrid area m; ΔP tie is the power variation transmitted through the tie lines between microgrid areas.

6. The shared energy storage frequency regulation method based on deep reinforcement learning according to claim 1 is characterized in that: The multi-microgrid includes a shared energy storage system, loads and power generation equipment. Correspondingly, in the frequency regulation framework, the regulator adjusts the operating parameters of the loads and power generation equipment according to the control instructions, and the shared energy storage aggregator adjusts the charging and discharging power of the shared energy storage system according to the control instructions.

7. The shared energy storage frequency regulation method based on deep reinforcement learning according to claim 1 is characterized in that: The objective function formula is as follows: Where X1 represents the sum of variables related to power plant operation cost, fuel cost and other related costs and load power change; X2 represents the sum of the difference between the total orderly dispatch power in the system and the output power of multiple power generation units; X3 represents the comprehensive index of frequency deviation and tie line power deviation, including the penalty items of frequency change and tie line power change; M I M is the first-level mileage of the frequency modulation device; I (t)=|ΔP ord-I (t)-ΔP ord-I (t-1)|;A I =1, I = 1 for DG, I = 2 for SES; G is the total number of frequency modulation devices, G = 2; N = 75 is a settlement cycle; c fuel represents the fuel cost; k ce The coefficient representing the carbon emission cost; c ce represents the carbon emission cost; ΔP d (t) represents the change in load power at time t; ΔP I (t) represents the power change of the power generation unit I at time t; ΔP ord-all (t) represents the total orderly dispatch power change of the system at time t, ΔP ord-all =ΔP ord-d +ΔP ord-sES1 +ΔP ord-sES2 ; μ represents the weight factor; ΔP tie (t) represents the change in the tie line power at time t; k pr represents the proportional coefficient; Δf(t) represents the frequency deviation at time t.

8. The shared energy storage frequency regulation method based on deep reinforcement learning according to claim 1 is characterized in that: The control instruction satisfying the objective function is generated according to the state information based on the Markov decision, and the specific process is as follows: Based on the LFC model, a state set and an action set are defined, and an actor network is initialized at the same time; a reward function is established according to the objective function; wherein the state set is a set of state information, and the action set is a set of control instructions; State information is obtained to determine the current state of the state set. The actor network generates the current control instruction of the action set according to the current state update, and the current control instruction is executed by the frequency modulation framework. After the execution is completed, the reward function evaluates the execution status, re-collects the state information, and repeats the above steps until the objective function is met.

9. The shared energy storage frequency adjustment method based on deep reinforcement learning according to claim 8 is characterized in that: The TD3 algorithm is introduced when the reward function evaluates the execution status, which specifically includes the following steps: Establishing a value function based on the reward function; Based on the TD3 algorithm, random noise is added to the actor network to find the optimal strategy for generating control instructions to maximize the value function while satisfying the objective function.

10. A shared energy storage frequency regulation system based on deep reinforcement learning, characterized in that: include: A multi-microgrid simulation module, used to establish an LFC model of the multi-microgrid, and simulate the frequency response and power balance under the physical constraints and frequency regulation constraints of the multi-microgrid according to the LFC model, wherein the multi-microgrid includes a shared energy storage system; An objective function establishment module, used to establish an objective function that satisfies the power balance by considering the frequency response and the cost; The frequency modulation module is used to establish a frequency modulation framework based on the intelligent power generation control of the DRL, and to generate a control instruction that satisfies the objective function according to the state information of the LFC model based on Markov decision, and the frequency modulation framework executes the control instruction to perform frequency modulation.

Images