Multi-energy-storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning and related device

Abstract

The invention provides a multi-energy-storage cooperative frequency modulation method based on improved maximum entropy deep reinforcement learning and a related device, and the method comprises the steps: building a frequency control model of a micro-grid comprising a micro gas turbine, battery energy storage, wind power, photovoltaic power and a load, and enabling the micro-grid to be equipped with a plurality of energy storage devices, performing frequency modulation control on the micro-grid by adopting virtual inertia control and virtual droop control; controlling the contribution degree of multi-energy-storage virtual droop control and the contribution degree of virtual inertia control by adopting an SAC algorithm, and meanwhile, improving a droop coefficient of virtual droop control; a deep neural network is adopted to establish an SAC agent, a success experience pool and a failure experience pool are set to improve an SAC algorithm, the training speed of the SAC agent is increased, and a multi-energy-storage coordinated frequency modulation method is generated through interactive training with a micro-grid environment. The multi-energy-storage collaborative frequency modulation method is generated through interactive training with the micro-grid environment, and the frequency regulation capability of multiple energy storage is improved.

Description

Technical Field

[0001] The present invention relates to the research field of multi-energy storage frequency control, and specifically to a multi-energy storage coordinated frequency modulation method and related devices based on improved maximum entropy deep reinforcement learning. Background Art

[0002] The large-scale burning of traditional fossil energy has caused global energy crises and environmental problems. In order to achieve the goal of sustainable economic and social development, new energy sources such as wind power and photovoltaics are connected to the power grid on a large scale to reduce the proportion of traditional thermal power generation and promote green power generation. Combining distributed power sources such as wind power, photovoltaics, and energy storage to form a microgrid can not only increase the penetration rate of wind and solar new energy in the power grid, but also reduce the impact of randomness and uncertainty of wind and solar output on the power grid. The control technology of microgrids is particularly critical.

[0003] Energy storage plays a key supporting role in microgrids. In order to smooth out the power fluctuations of wind power and loads, multiple energy storages are equipped in microgrids to improve the regulation capacity of the entire microgrid. Due to its fast response speed, energy storage is often used to participate in the frequency regulation of microgrids. The control methods are mainly virtual inertia control and virtual droop control. When energy storage adopts these two types of control methods, the traditional model often linearly superimposes the two methods, which may cause sudden changes in the output of energy storage and cause greater frequency fluctuations. The fixed droop coefficient in virtual droop control also leads to a contradiction between the response speed and stability of energy storage, reducing the frequency regulation ability of energy storage.

[0004] Therefore, it is urgently needed to reasonably adjust the contribution of the two control methods, virtual inertia control and virtual droop control of energy storage, improve the traditional fixed droop coefficient, and at the same time consider the coordinated operation between multiple energy storages to improve the frequency stability of the entire microgrid. Summary of the invention

[0005] In order to improve the frequency regulation capability of multi-energy storage in microgrids, the present invention provides a multi-energy storage collaborative frequency regulation method and related devices based on improved maximum entropy deep reinforcement learning. The deep reinforcement learning agent that introduces information entropy has strong robustness and is an effective way to solve the above problems.

[0006] A multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning includes the following steps:

[0007] Step 1: Establish a frequency control model of a microgrid including micro gas turbines, battery energy storage, wind power, photovoltaics, and loads. The loads, wind power, and photovoltaics are used as disturbance sources. Micro gas turbines and battery energy storage are responsible for the frequency regulation of the microgrid. Multiple battery energy storages support the microgrid. The control methods are virtual inertia control and virtual droop control.

[0008] Step 2: Introduce the maximum entropy deep reinforcement learning Soft Actor Critic (SAC) algorithm to control and adjust the contribution of virtual inertial control and virtual droop control by establishing action space, state space and reward function, and improve the virtual droop coefficient to solve the contradiction between the response speed and stability of energy storage;

[0009] Step 3: Based on the frequency control model of the microgrid established in step 1 and the action space, state space and reward function established in step 2, the successful experience pool and the failed experience pool are set to improve the SAC algorithm, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

[0010] Furthermore, the frequency control model has two battery energy storages, and the frequency control model of the microgrid is:

[0011]

[0012]

[0013] ΔP EI,i =-λ a,i ·J E,i s·Δf

[0014] ΔP ED,i =-λ b,i ·K E,i Δf

[0015]

[0016] Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, and ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment of the wind power system output, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PVis the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contribution of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔP EI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

[0017] Furthermore, in step 2, the SAC agent obtains frequency information from the microgrid as the state space, and the frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation; the action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients:

[0018]

[0019] a=(λ a,1 ,λ b,1 ,K E,1 ,λ a,2 ,λ b,2 ,K E,2 )

[0020] Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, respectively, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, respectively. E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints:

[0021] λ a,1 +λ b,1 ≤1

[0022] λ a,2 +λ b,2 ≤1

[0023] According to the frequency deviation of the microgrid, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established:

[0024]

[0025] Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively.

[0026] Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

[0027]

[0028] Furthermore, step 3 specifically includes: training the SAC agent based on the microgrid model established in step 1 and the action space, state space and reward function established in step 2. The SAC algorithm is a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, including two types of network structures: action network and evaluation network. The action network outputs variables in the action space and acts on the environment, and the evaluation network evaluates the action.

[0029] The minimization optimization objective J(φ) of the action network parameter φ is

[0030]

[0031] Among them, π φ (a t |s t ) is s t Output a t probability, α is the temperature coefficient, Q θ (s t ,a t ) is the output of the evaluation network;

[0032] The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is:

[0033] y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))]

[0034]

[0035] Among them, γ is the discount factor, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value;

[0036] The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is:

[0037]

[0038] Among them, H0 is the entropy limit;

[0039] The target evaluation network parameter θ' is updated in a soft update mode:

[0040] θ′=τθ+(1-τ)θ′

[0041] Where τ is the update coefficient;

[0042] In order to improve the training speed of the SAC agent, the successful experience pool and the failed experience pool are set to improve the SAC algorithm to store samples of agent interaction. ξ pieces of sample data are extracted from the successful experience pool and the failed experience pool, respectively. ξ is obtained by the adaptive batch function:

[0043]

[0044] ξ=[ρ·N]

[0045] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After the pre-learning process, the action network, evaluation network and target evaluation network are updated, and the training generates a multi-energy storage coordinated frequency modulation method based on maximum entropy deep reinforcement learning.

[0046] A multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning, comprising:

[0047] The first module is used to establish a frequency control model of a microgrid including a micro gas turbine, battery energy storage, wind power, photovoltaic power, and load. The load, wind power, and photovoltaic power are used as disturbance sources. The micro gas turbine and battery energy storage are responsible for the frequency regulation of the microgrid. Multiple battery energy storages support the microgrid. The control mode is virtual inertia control and virtual droop control.

[0048] The second module is used to introduce the SAC algorithm. By establishing the action space, state space and reward function, the contribution of virtual inertia control and virtual droop control is controlled and adjusted, and the virtual droop coefficient is improved to solve the contradiction between the response speed and stability of energy storage.

[0049] The third module is used to improve the SAC algorithm by setting up the successful experience pool and the failed experience pool based on the established frequency control model of the microgrid and the established action space, state space and reward function, thereby increasing the training speed of the SAC agent and generating a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

[0050] Furthermore, the frequency control model has two battery energy storages, and the frequency control model of the microgrid is:

[0051]

[0052] ΔP EI,i =-λ a,i ·J E,i s·Δf

[0053] ΔP ED,i =-λ b,i ·K E,i Δf

[0054]

[0055] Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, and D

[0056] is the damping constant, ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment of the wind power system output, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PV is the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contribution of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔPEI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

[0057] Furthermore, the SAC agent obtains frequency information from the microgrid as the state space, and the frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation; the action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients:

[0058]

[0059] a=(λ a,1 ,λ b,1 ,K E,1 ,λ a,2 ,λ b,2 ,K E,2 )

[0060] Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, respectively, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, respectively. E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints:

[0061] λ a,1 +λ b,1 ≤1

[0062] λ a,2 +λ b,2 ≤1

[0063] According to the frequency deviation of the microgrid, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established:

[0064]

[0065] Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively.

[0066] Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

[0067]

[0068] Furthermore, the third module is specifically used for:

[0069] The SAC agent is trained based on the established microgrid model and the established action space, state space and reward function. The SAC algorithm is a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, including two types of network structures: action network and evaluation network. The action network outputs variables in the action space and acts on the environment, and the evaluation network estimates the action.

[0070] The minimization optimization objective J(φ) of the action network parameter φ is

[0071]

[0072] Among them, π φ (a t |s t ) is s t Output a t probability, α is the temperature coefficient, Q θ (s t ,a t ) is the output of the evaluation network;

[0073] The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is:

[0074] y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))]

[0075]

[0076] Among them, γ is the discount factor, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value;

[0077] The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is:

[0078]

[0079] Among them, H0 is the entropy limit;

[0080] The target evaluation network parameter θ' is updated in a soft update mode:

[0081] θ′=τθ+(1-τ)θ′

[0082] Where τ is the update coefficient;

[0083] In order to improve the training speed of the SAC agent, the successful experience pool and the failed experience pool are set to improve the SAC algorithm to store samples of agent interaction. ξ pieces of sample data are extracted from the successful experience pool and the failed experience pool, respectively. ξ is obtained by the adaptive batch function:

[0084]

[0085] ξ=[ρ·N]

[0086] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After the pre-learning process, the action network, evaluation network and target evaluation network are updated, and the training generates a multi-energy storage coordinated frequency modulation method based on maximum entropy deep reinforcement learning.

[0087] A multi-energy storage coordinated frequency modulation system based on improved maximum entropy deep reinforcement learning, comprising: a computer-readable storage medium and a processor;

[0088] The computer-readable storage medium is used to store executable instructions;

[0089] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning.

[0090] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning.

[0091] Compared with the existing methods, the present invention has the following advantages:

[0092] The method proposed in the present invention is based on the maximum entropy deep reinforcement learning algorithm. The SAC agent has extremely strong exploration ability and robustness, and the successful experience pool and the failed experience pool are designed to shorten the training time of the SAC agent. By improving the SAC agent, the contribution of the virtual inertia control and virtual droop control of multiple energy storages in the microgrid is adjusted to ensure the stability of the energy storage output, avoid the sudden change in output caused by the simple superposition of the two traditional control methods, and adaptively adjust the droop coefficient of multiple energy storages to increase the active output range while ensuring the response speed of the energy storage. Through the control instructions of the SAC agent to multiple energy storages in the microgrid, the coordinated frequency modulation between the multiple energy storages is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 It is a flow chart of a multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning in an embodiment of the present invention.

[0094] Figure 2 is a diagram of the energy storage control structure of an embodiment of the present invention;

[0095] Figure 3 It is a multi-energy storage frequency regulation control model based on an improved SAC algorithm in an embodiment of the present invention.

[0096] Figure 4 It is a flow chart of improving SAC algorithm training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0097] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0098] See also Figure 1-4 The embodiment of the present invention provides a multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning, comprising:

[0099] Step 1: First, establish a frequency control model of a microgrid including a micro gas turbine, battery energy storage, wind power, photovoltaics, and loads. The operation modes of a microgrid include grid-connected operation and island operation. During island operation, the microgrid lacks the support of the external power grid and needs to maintain internal power balance and system frequency stability on its own. Wind power and photovoltaics, as clean energy sources, are connected to the grid in a constant power control mode. They serve as disturbance sources in the microgrid and do not participate in the frequency regulation of the microgrid. Considering the high leakage rate of distributed power sources, multiple battery energy storages are equipped to support the microgrid. The control modes are virtual inertia control and virtual droop control. The frequency control model of the microgrid is:

[0100]

[0101] ΔP EI,i =-λ a,i ·J E,i s·Δf

[0102] ΔP ED,i =-λ b,i ·K E,i Δf

[0103]

[0104] Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, and ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment of the wind power system output, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PV is the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contribution of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔP EI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

[0105] The present invention takes two battery energy storage as an example, and the energy storage adjustment instruction is obtained by virtual inertia control and virtual droop control. Figure 3 In the microgrid, load, wind power, and photovoltaic power are used as disturbance quantities, and micro gas turbines and battery energy storage are responsible for the frequency regulation of the microgrid. The frequency control model of the system is as follows: Figure 3 shown.

[0106] Step 2: The frequency control model of the microgrid established in step 1 has multiple battery energy storages. Multiple battery energy storages use virtual droop control and virtual inertia control to generate control instructions. The simple superposition of the two control methods may lead to a sudden change in the output power of the energy storage, reducing the frequency recovery capability of the microgrid. By introducing the SAC algorithm to establish the action space, state space and reward function, the contribution of the two controls is controlled and adjusted. At the same time, in order to solve the contradiction between the response speed and stability of the energy storage, the virtual droop coefficient is improved to increase the power output range of the energy storage.

[0107] In order to realize the coordinated operation of multiple energy storages in the microgrid, the SAC agent needs to obtain frequency information from the microgrid as the state space, specifically the frequency deviation of the microgrid and the rate of change of the frequency deviation. The action space is selected as the contribution of the virtual inertia control and virtual droop control of the two energy storages and their respective droop coefficients:

[0108]

[0109] a=(λ a,1 ,λ b,1 ,K E,1 ,λ a,2 ,λ b,2 ,K E,2 )

[0110] Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, respectively, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, respectively. E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control satisfies the following constraints

[0111] λ a,1 +λ b,1 ≤1

[0112] λ a,2 +λ b,2 ≤1

[0113] According to the frequency deviation of the microgrid, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established:

[0114]

[0115] Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively.

[0116] Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

[0117]

[0118] Among them, T termination Indicates that the agent enters the terminal state, T non-termination Indicates that the agent has not entered the terminal state.

[0119] Through the termination function, it is defined that in each episode of agent training, if the SAC agent does not enter the termination state, the agent's training in this episode is considered "successful" and the data samples in this episode are "successful experience"; if the SAC agent enters the termination state, the agent's training in this episode is considered "failed" and the sample data in this episode are "failed experience".

[0120] Step 3: Based on the frequency control model of the microgrid established in step 1 and the action space, state space and reward function established in step 2, the successful experience pool and the failed experience pool are set to improve the SAC algorithm and increase the training speed of the SAC agent. A multi-energy storage coordinated frequency modulation method is generated through interactive training with the microgrid environment.

[0121] The SAC algorithm is the first deep reinforcement learning algorithm based on maximum entropy. By introducing information entropy, it improves the exploration ability of deep reinforcement learning agents. Different from traditional deep reinforcement learning, the SAC algorithm pursues the maximization of long-term rewards and entropy value:

[0122]

[0123] Among them, π * is the optimal control strategy of the intelligent agent; α is the temperature coefficient; H(π(·|s t )) is the information entropy, and the specific function is:

[0124] H(π(·|s t ))=-Εlnπ(a t ∣s t )

[0125] Among them, π(a t |s t ) means in s t Next take a t After the introduction of information entropy, the state value of the SAC agent is modified to:

[0126] V(s t )=E[Q(s t ,a t )-αlnπ(a t ∣s t )]

[0127] The SAC algorithm is a maximum entropy reinforcement learning algorithm under the actor-cirtic framework. The present invention improves the SAC algorithm by setting a successful experience pool and a failed experience pool, thereby increasing the training speed of the SAC agent, and utilizing interactive training with the environment to generate a multi-energy storage coordinated frequency regulation method, thereby improving the frequency regulation capability of multi-energy storage in the microgrid.

[0128] The SAC agent consists of two types of network structures: action network and evaluation network. The action network outputs variables in the action space and acts on the environment, while the evaluation network evaluates the action.

[0129] The optimization update of the action network parameters φ is performed by minimizing the relative entropy.

[0130]

[0131] in is the partition function, which has no effect on the update of network parameters. By simplifying the relative entropy, the minimization optimization target J(φ) of the action network parameter φ is obtained as follows:

[0132]

[0133] Where Q θ (s t ,a t ) is the output of the evaluation network.

[0134] The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is:

[0135] y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))]

[0136]

[0137] Among them, γ is the discount factor, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value.

[0138] Different from traditional deep reinforcement learning algorithms, the goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. In order to reduce the sensitivity of the SAC algorithm to parameters, the temperature coefficient α is updated during the training process, and the minimization optimization goal is:

[0139]

[0140] Here, H0 is the entropy limit.

[0141] Based on the action network parameter φ, the optimization target of the evaluation network parameter θ and the temperature coefficient α is updated using the gradient descent method:

[0142]

[0143] Among them, β φ , β θ , β α are the learning rate of the action network parameter φ, the evaluation network parameter θ and the temperature coefficient α respectively.

[0144] The target evaluation network parameter θ' is updated in a soft update mode:

[0145] θ′=τθ+(1-τ)θ′

[0146] Where τ is the update coefficient.

[0147] In order to improve the training speed of the SAC agent, the SAC algorithm is improved by setting the successful experience pool and the failed experience pool to store samples of agent interaction. According to the set termination function, the training successful sample data is put into the successful experience pool, and the training failed sample data is put into the failed experience pool. ξ sample data are extracted from the successful experience pool and the failed experience pool, respectively. ξ is obtained by the adaptive batch function:

[0148]

[0149] ξ=[ρ·N]

[0150] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] means rounding down.

[0151] By setting up a successful experience pool and a failed experience pool, it is possible to avoid the SAC agent from learning under the same type of experience data during training, thereby improving the generalization ability of the SAC agent and the efficiency of sample data utilization. The improved SAC agent continuously interacts with the established microgrid model to generate sample data, and extracts data from the two experience pools to train the action network, evaluation network, and target evaluation network. As the number of training sets increases, the long-term rewards obtained by the agent will gradually converge, and finally a multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning is obtained.

[0152] The training process is as follows: Figure 4 :

[0153] Initialize the evaluation network parameters θ, the evaluation main network parameters θ, the target evaluation network parameters θ', clear the successful experience pool and the failed experience pool, and set the number of training sets:

[0154] (1) Initialize the output of each distributed power source and obtain the frequency deviation and frequency deviation change rate as the state s t ;

[0155] (2) Calculate action a t Act on the environment to get the next state s t+1 , environmental feedback reward r t ;

[0156] (3) Let s t =s t+1 , the agent continues to interact with the environment until the last state;

[0157] (4) Determine whether the agent has entered the termination state. If so, store (s t ,a t ,r t ,s t+1 ) to the failure experience pool, otherwise the sample is stored in the success experience pool;

[0158] (5) Samples are collected from the experience pool, the evaluation network parameters, target evaluation network parameters, action network parameters are updated, and the temperature coefficient is adjusted:

[0159] (6) Whether the microgrid environmental constraints are met, if yes, go to step (2), otherwise go to step (5);

[0160] (7) Determine whether the reward obtained by the agent in each episode meets the requirements or whether the number of training episodes is completed. If so, end the training; otherwise, go to step (1);

[0161] After the training, based on the improved SAC agent, the contribution of the virtual inertia control and virtual droop control of the two energy storages is improved according to the frequency deviation and frequency deviation change rate of the microgrid, so as to avoid sudden changes in the output of the energy storage due to inappropriate control methods. At the same time, the virtual droop coefficients of the two energy storages are improved. While achieving a fast response speed, the range of active power output of the energy storage is increased, the coordination between different energy storages is achieved, and the anti-interference performance of the entire microgrid is improved.

[0162] On the other hand, the present invention provides a multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning, comprising:

[0163] The first module is used to establish a frequency control model of a microgrid including a micro gas turbine, battery energy storage, wind power, photovoltaic power, and load. The load, wind power, and photovoltaic power are used as disturbance sources. The micro gas turbine and battery energy storage are responsible for the frequency regulation of the microgrid. Multiple battery energy storages support the microgrid. The control mode is virtual inertia control and virtual droop control.

[0164] The second module is used to introduce the SAC algorithm. By establishing the action space, state space and reward function, the contribution of virtual inertia control and virtual droop control is controlled and adjusted, and the virtual droop coefficient is improved to solve the contradiction between the response speed and stability of energy storage.

[0165] The third module is used to improve the SAC algorithm by setting up the successful experience pool and the failed experience pool based on the established frequency control model of the microgrid and the established action space, state space and reward function, thereby increasing the training speed of the SAC agent and generating a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

[0166] Another aspect of the present invention provides a multi-energy storage coordinated frequency modulation system based on improved maximum entropy deep reinforcement learning, comprising: a computer-readable storage medium and a processor;

[0167] The computer-readable storage medium is used to store executable instructions;

[0168] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning described in the first aspect.

[0169] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning as described in the first aspect.

[0170] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning, characterized in that: The steps include: Step 1: Establish a frequency control model of a microgrid including micro gas turbines, battery energy storage, wind power, photovoltaics, and loads. The loads, wind power, and photovoltaics are used as disturbance sources. Micro gas turbines and battery energy storage are responsible for the frequency regulation of the microgrid. Multiple battery energy storages support the microgrid. The control methods are virtual inertia control and virtual droop control. Step 2: Introduce the SAC algorithm, and control and adjust the contribution of virtual inertia control and virtual droop control by establishing action space, state space and reward function, and improve the virtual droop coefficient to solve the contradiction between the response speed and stability of energy storage; Step 3: Based on the frequency control model of the microgrid established in step 1 and the action space, state space and reward function established in step 2, the successful experience pool and the failed experience pool are set to improve the SAC algorithm, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

2. The multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning as claimed in claim 1, characterized in that: There are two battery energy storages in the frequency control model, and the frequency control model of the microgrid is: ΔP EI,i =-λ a,i ·J E,i s·Δf ΔP ED,i =-λ b,i ·K E,i ·Δf Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, and ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment of the wind power system output, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PV is the photovoltaic time constant, ΔP wind， is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contribution of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔP EI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

3. The multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning as claimed in claim 2, characterized in that: In step 2, the SAC agent obtains frequency information from the microgrid as the state space, and the frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation; the action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients: a=(λ a,1 ,l b,1 ,K E,1 ,l a,2 ,l b,2 ,K E,2 ) Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, respectively, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, respectively. E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints: l a,1 +λ b,1 ≤1 l a,2 +λ b,2 ≤1 According to the frequency deviation of the microgrid, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established: Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively. Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

4. The multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning as claimed in claim 3, characterized in that: Step 3 specifically includes: training the SAC agent based on the microgrid model established in step 1 and the action space, state space and reward function established in step 2. The SAC algorithm is a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, including two types of network structures: action network and evaluation network. The action network outputs variables in the action space and acts on the environment, and the evaluation network evaluates the action. The minimization optimization objective J(φ) of the action network parameter φ is Among them, π φ (a t |s t ) is s t Output a t probability, α is the temperature coefficient, Q θ (s t ,a t ) is the output of the evaluation network; The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is: y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))] Among them, γ is the discount factor, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value; The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is: Among them, H0 is the entropy limit; The target evaluation network parameter θ' is updated in a soft update mode: θ′=τθ+(1-τ)θ′ Where τ is the update coefficient; In order to improve the training speed of the SAC agent, the successful experience pool and the failed experience pool are set to improve the SAC algorithm to store samples of agent interaction. ξ pieces of sample data are extracted from the successful experience pool and the failed experience pool, respectively. ξ is obtained by the adaptive batch function: ξ=[ρ·N] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After the pre-learning process, the action network, evaluation network and target evaluation network are updated, and the training generates a multi-energy storage coordinated frequency modulation method based on maximum entropy deep reinforcement learning.

5. A multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning, characterized in that: include: The first module is used to establish a frequency control model of a microgrid including a micro gas turbine, battery energy storage, wind power, photovoltaic power, and load. The load, wind power, and photovoltaic power are used as disturbance sources. The micro gas turbine and battery energy storage are responsible for the frequency regulation of the microgrid. Multiple battery energy storages support the microgrid. The control mode is virtual inertia control and virtual droop control. The second module is used to introduce the SAC algorithm. By establishing the action space, state space and reward function, the contribution of virtual inertia control and virtual droop control is controlled and adjusted, and the virtual droop coefficient is improved to solve the contradiction between the response speed and stability of energy storage. The third module is used to improve the SAC algorithm by setting up the successful experience pool and the failed experience pool based on the established frequency control model of the microgrid and the established action space, state space and reward function, thereby increasing the training speed of the SAC agent and generating a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

6. The multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning as claimed in claim 5, characterized in that: There are two battery energy storages in the frequency control model, and the frequency control model of the microgrid is: ΔP EI,i =-λ a,i ·J E,i s·Δf ΔP ED,i =-λ b,i ·K E,i ·Δf Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, and ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment of the wind power system output, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PV is the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contribution of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔP EI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

7. The multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning as claimed in claim 6, characterized in that: The SAC agent obtains frequency information from the microgrid as the state space, which includes the frequency deviation of the microgrid and the rate of change of the frequency deviation; the action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients: a=(λ a,1 ,l b,1 ,K E,1 ,l a,2 ,l b,2 ,K E,2 ) Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, respectively, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, respectively. E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints: l a,1 +λ b,1 ≤1 l a,2 +λ b,2 ≤1 According to the frequency deviation of the microgrid, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established: Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively. Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

8. The multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning as claimed in claim 7, characterized in that: The third module is specifically used for: The SAC agent is trained based on the established microgrid model and the established action space, state space and reward function. The SAC algorithm is a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, including two types of network structures: action network and evaluation network. The action network outputs variables in the action space and acts on the environment, and the evaluation network estimates the action. The minimization optimization objective J(φ) of the action network parameter φ is Among them, π φ (a t |s t ) is s t Output a t probability, α is the temperature coefficient, Q θ (s t ,a t ) is the output of the evaluation network; The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is: y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))] Among them, γ is the discount factor, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value; The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is: Among them, H0 is the entropy limit; The target evaluation network parameter θ' is updated in a soft update mode: θ′=τθ+(1-τ)θ′ Where τ is the update coefficient; In order to improve the training speed of the SAC agent, the successful experience pool and the failed experience pool are set to improve the SAC algorithm to store samples of agent interaction. ξ pieces of sample data are extracted from the successful experience pool and the failed experience pool, respectively. ξ is obtained by the adaptive batch function: ξ=[ρ·N] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After the pre-learning process, the action network, evaluation network and target evaluation network are updated, and the training generates a multi-energy storage coordinated frequency modulation method based on maximum entropy deep reinforcement learning.

9. A multi-energy storage coordinated frequency modulation system based on improved maximum entropy deep reinforcement learning, comprising: A computer readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning as described in any one of claims 1-4.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning as described in any one of claims 1 to 4.

Images