Multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning and related device

Abstract

The application provides a kind of based on the improved maximum entropy deep reinforcement learning multi-energy collaborative frequency modulation method and related device, the method comprises: the frequency control model of microgrid including micro gas turbine, battery energy storage, wind power, photovoltaic, load is established, microgrid is equipped with multiple energy storage, and virtual inertia control and virtual droop control are used to carry out frequency modulation control to microgrid;The contribution of virtual droop control and the contribution of virtual inertia control of multiple energy storages are controlled using SAC algorithm, and the droop coefficient of virtual droop control is improved;Deep neural network is used to establish SAC agent, and success experience pool and failure experience pool are set to improve SAC algorithm, improve the training speed of SAC agent, and generate multi-energy coordinated frequency modulation method by interacting with microgrid environment training.The application generates multi-energy collaborative frequency modulation method by interacting with microgrid environment training, and improves the frequency regulation capability of multiple energy storages.

Description

Technical Field

[0001] The present invention relates to the research field of multi-energy storage frequency control, and specifically to a multi-energy storage collaborative frequency modulation method and related devices based on improved maximum entropy deep reinforcement learning. Background Art

[0002] The widespread combustion of traditional fossil fuels has caused global energy crises and environmental problems. To achieve sustainable economic and social development, renewable energy sources such as wind power and photovoltaics are being integrated into the power grid on a large scale, reducing the share of traditional thermal power generation and promoting green power generation. Combining distributed power sources such as wind power, photovoltaics, and energy storage to form microgrids not only increases the penetration of wind and solar power in the grid but also reduces the impact of the randomness and uncertainty of wind and solar output on the grid. Control technology for microgrids is crucial.

[0003] Energy storage plays a key supporting role in microgrids. To smooth power fluctuations from wind power and loads, microgrids are equipped with multiple energy storage devices to increase the regulation capacity of the entire microgrid. Due to its fast response speed, energy storage is often used to regulate the frequency of microgrids. The main control methods are virtual inertia control and virtual droop control. When energy storage uses these two control methods, traditional models often linearly superimpose the two methods, which may cause sudden changes in the energy storage output and cause greater frequency fluctuations. The fixed droop coefficient in virtual droop control also leads to a conflict between the response speed and stability of energy storage, reducing the frequency regulation capability of energy storage.

[0004] Therefore, there is an urgent need to reasonably adjust the contribution of the two control methods of energy storage virtual inertia control and virtual droop control, improve the traditional fixed droop coefficient, and at the same time consider the coordinated operation between multiple energy storages to improve the frequency stability of the entire microgrid. Summary of the Invention

[0005] In order to improve the frequency regulation capability of multi-energy storage in microgrids, the present invention provides a multi-energy storage collaborative frequency regulation method and related devices based on improved maximum entropy deep reinforcement learning. The deep reinforcement learning agent that introduces information entropy has strong robustness and is an effective way to solve the above problems.

[0006] A multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning includes the following steps:

[0007] Step 1: Establish a frequency control model for a microgrid that includes a micro gas turbine, battery energy storage, wind power, photovoltaic power, and loads. The loads, wind power, and photovoltaic power serve as disturbance sources, while the micro gas turbine and battery energy storage are responsible for frequency regulation. Multiple battery energy storage systems support the microgrid, and the control methods are virtual inertia control and virtual droop control.

[0008] Step 2: Introduce the maximum entropy deep reinforcement learning Soft Actor Critic (SAC) algorithm. By establishing the action space, state space, and reward function, the contribution of virtual inertial control and virtual droop control is controlled and adjusted. At the same time, the virtual droop coefficient is improved to resolve the contradiction between the response speed and stability of energy storage.

[0009] Step 3: Based on the frequency control model of the microgrid established in step 1 and the action space, state space, and reward function established in step 2, the successful experience pool and the failed experience pool are set to improve the SAC algorithm, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

[0010] Furthermore, the frequency control model has two battery energy storages, and the frequency control model of the microgrid is:

[0011]

[0012]

[0013] ΔP EI,i =-λ a,i ·J E,i s·Δf

[0014] ΔP ED,i =-λ b,i ·K E,i Δf

[0015]

[0016] Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, and ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment output by the wind power system, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PVis the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contributions of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔP EI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

[0017] Furthermore, in step 2, the SAC agent obtains frequency information from the microgrid as the state space. The frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation. The action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients:

[0018]

[0019] a=(λ a,1 ,λ b,1 ,K E,1 ,λ a,2 ,λ b,2 ,K E,2 )

[0020] Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, in [K E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints:

[0021] λ a,1 +λ b,1 ≤1

[0022] λ a,2 +λ b,2 ≤1

[0023] According to the microgrid frequency deviation, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established:

[0024]

[0025] Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively.

[0026] Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

[0027]

[0028] Furthermore, step 3 specifically includes: training the SAC agent based on the microgrid model established in step 1 and the action space, state space, and reward function established in step 2. The SAC algorithm, as a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, includes two network structures: an action network and an evaluation network. The action network outputs variables in the action space and acts on the environment, and the evaluation network evaluates the action.

[0029] The minimization optimization objective J(φ) of the action network parameter φ is

[0030]

[0031] Among them, π φ (a t |s t ) is s t Output a t The probability of α is the temperature coefficient, Q θ (s t ,a t ) is the output of the evaluation network;

[0032] The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is:

[0033] y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))]

[0034]

[0035] Where γ is the discount coefficient, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value;

[0036] The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is:

[0037]

[0038] Where H0 is the entropy limit;

[0039] The target evaluation network parameter θ' is updated in a soft update mode:

[0040] θ′=τθ+(1-τ)θ′

[0041] Where τ is the update coefficient;

[0042] In order to improve the training speed of the SAC agent, the SAC algorithm is improved by setting a successful experience pool and a failed experience pool to store samples of agent interactions. ξ sample data are extracted from the successful experience pool and the failed experience pool respectively. ξ is obtained by the adaptive batch function:

[0043]

[0044] ξ=[ρ·N]

[0045] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After a pre-learning process, the action network, evaluation network, and target evaluation network are updated, and a multi-energy storage collaborative frequency modulation method based on maximum entropy deep reinforcement learning is trained and generated.

[0046] A multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning, comprising:

[0047] The first module is used to establish a frequency control model for a microgrid that includes a micro gas turbine, battery energy storage, wind power, photovoltaic power, and loads. The loads, wind power, and photovoltaic power serve as disturbance sources, while the micro gas turbine and battery energy storage are responsible for frequency regulation. Multiple battery energy storage systems support the microgrid, and the control methods are virtual inertia control and virtual droop control.

[0048] The second module introduces the SAC algorithm. By establishing the action space, state space, and reward function, it regulates the contribution of virtual inertia control and virtual droop control, while also improving the virtual droop coefficient to resolve the conflict between the response speed and stability of energy storage.

[0049] The third module is used to improve the SAC algorithm based on the established microgrid frequency control model and the established action space, state space and reward function, set up successful experience pools and failed experience pools, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency regulation method through interactive training with the microgrid environment.

[0050] Furthermore, the frequency control model has two battery energy storages, and the frequency control model of the microgrid is:

[0051]

[0052] ΔP EI,i =-λ a,i ·J E,i s·Δf

[0053] ΔP ED,i =-λ b,i ·K E,i Δf

[0054]

[0055] Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, and D

[0056] is the damping constant, ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment output by the wind power system, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PV is the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contributions of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔPEI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

[0057] Furthermore, the SAC agent obtains frequency information from the microgrid as the state space. The frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation. The action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients:

[0058]

[0059] a=(λ a,1 ,λ b,1 ,K E,1 ,λ a,2 ,λ b,2 ,K E,2 )

[0060] Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, in [K E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints:

[0061] λ a,1 +λ b,1 ≤1

[0062] λ a,2 +λ b,2 ≤1

[0063] According to the microgrid frequency deviation, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established:

[0064]

[0065] Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively.

[0066] Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

[0067]

[0068] Furthermore, the third module is specifically used to:

[0069] The SAC agent is trained based on the established microgrid model and the established action space, state space, and reward function. The SAC algorithm, as a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, includes two network structures: an action network and an evaluation network. The action network outputs variables in the action space and acts on the environment, while the evaluation network evaluates the actions.

[0070] The minimization optimization objective J(φ) of the action network parameter φ is

[0071]

[0072] Among them, π φ (a t |s t ) is s t Output a t The probability of α is the temperature coefficient, Q θ (s t ,a t ) is the output of the evaluation network;

[0073] The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is:

[0074] y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))]

[0075]

[0076] Where γ is the discount coefficient, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value;

[0077] The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is:

[0078]

[0079] Where H0 is the entropy limit;

[0080] The target evaluation network parameter θ' is updated in a soft update mode:

[0081] θ′=τθ+(1-τ)θ′

[0082] Where τ is the update coefficient;

[0083] In order to improve the training speed of the SAC agent, the SAC algorithm is improved by setting a successful experience pool and a failed experience pool to store samples of agent interactions. ξ sample data are extracted from the successful experience pool and the failed experience pool respectively. ξ is obtained by the adaptive batch function:

[0084]

[0085] ξ=[ρ·N]

[0086] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After a pre-learning process, the action network, evaluation network, and target evaluation network are updated, and a multi-energy storage collaborative frequency modulation method based on maximum entropy deep reinforcement learning is trained and generated.

[0087] A multi-energy storage coordinated frequency modulation system based on improved maximum entropy deep reinforcement learning, comprising: a computer-readable storage medium and a processor;

[0088] The computer-readable storage medium is used to store executable instructions;

[0089] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning.

[0090] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning.

[0091] Compared with the existing methods, the present invention has the following advantages:

[0092] The method proposed in the present invention is based on the maximum entropy deep reinforcement learning algorithm. The SAC intelligent agent has extremely strong exploration ability and robustness, and the successful experience pool and the failed experience pool are designed to shorten the training time of the SAC intelligent agent. By improving the SAC intelligent agent, the contribution of the virtual inertia control and virtual droop control of multiple energy storages in the microgrid is adjusted to ensure the stability of the energy storage output, and avoid the sudden change in output caused by the simple superposition of the two traditional control methods. At the same time, the droop coefficient of multiple energy storages is adaptively adjusted to increase the active output range while ensuring the response speed of the energy storage. Through the control instructions of the SAC intelligent agent to multiple energy storages in the microgrid, coordinated frequency modulation between multiple energy storages is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 This is a flowchart of a multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning in an embodiment of the present invention.

[0094] Figure 2 This is a diagram of the energy storage control structure of an embodiment of the present invention;

[0095] Figure 3 It is a multi-energy storage frequency regulation control model based on the improved SAC algorithm in an embodiment of the present invention.

[0096] Figure 4 This is a flowchart of improving the SAC algorithm training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0097] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0098] See also Figure 1-4 The embodiment of the present invention provides a multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning, including:

[0099] Step 1: First, establish a frequency control model for a microgrid that includes a micro gas turbine, battery energy storage, wind power, photovoltaics, and loads. The operating modes of a microgrid include grid-connected operation and island operation. During island operation, the microgrid lacks the support of the external power grid and needs to maintain internal power balance and system frequency stability on its own. Wind power and photovoltaics, as clean energy sources, are connected to the grid in a constant power control mode. They serve as disturbance sources in the microgrid and do not participate in the frequency regulation of the microgrid. Considering the high leakage rate of distributed power sources, multiple battery energy storages are equipped to support the microgrid. The control mode is virtual inertia control and virtual droop control. The frequency control model of the microgrid is:

[0100]

[0101] ΔP EI,i =-λ a,i ·J E,i s·Δf

[0102] ΔP ED,i =-λ b,i ·K E,i Δf

[0103]

[0104] Where Δf is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, and ΔP M is the output power increment of the micro gas turbine, ΔP PV is the output power increment of the photovoltaic system, ΔP W is the power increment output by the wind power system, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, T t is the time constant of the gas turbine, T g is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; R is the adjustment coefficient; K P ,K S is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, T PV is the photovoltaic time constant, ΔP wind , is the initial wind power change, T W is the time constant of the wind turbine, i=1,2, λ a,i ,λ b,i are the power contributions of energy storage virtual inertia control and virtual droop control, J E,i ,K E,i are the virtual inertia power coefficient and virtual droop power coefficient, ΔP EI,i ,ΔP ED,i are the energy storage adjustment instructions for virtual inertia control and virtual droop control, T E,i is the energy storage time constant.

[0105] The present invention takes two battery energy storage as an example, and the energy storage adjustment instruction is obtained by virtual inertia control and virtual droop control. Figure 3 As shown in the figure, load, wind power and photovoltaic power are used as disturbance variables in the microgrid, micro gas turbine and battery energy storage are responsible for the frequency regulation of the microgrid, and the frequency control model of the system is shown in the figure. Figure 3 shown.

[0106] Step 2: The frequency control model of the microgrid established in step 1 has multiple battery energy storages. Multiple battery energy storages use virtual droop control and virtual inertia control to generate control instructions. The simple superposition of the two control methods may lead to a sudden change in the energy storage output power, reducing the frequency recovery capability of the microgrid. By introducing the SAC algorithm to establish the action space, state space and reward function, the contribution of the two controls is controlled and adjusted. At the same time, in order to resolve the contradiction between the response speed and stability of energy storage, the virtual droop coefficient is improved to increase the power output range of energy storage.

[0107] To achieve coordinated operation between multiple energy storage systems in a microgrid, the SAC agent needs to obtain frequency information from the microgrid as its state space. Specifically, it contains the frequency deviation and the rate of change of the frequency deviation. The action space is selected as the contribution of the virtual inertia control and virtual droop control of the two energy storage systems, as well as their respective droop coefficients:

[0108]

[0109] a=(λ a,1 ,λ b,1 ,K E,1 ,λ a,2 ,λ b,2 ,K E,2 )

[0110] Among them, s is the state space, a is the action space, Δf is the frequency deviation, λ a,1 ,λ b,1 ,λ a,2 ,λ b,2 are the contribution of virtual inertia control and virtual droop control of the two battery energy storages, K E,1 ,K E,2 are the virtual droop power coefficients of the two battery energy storages, in [K E,imin ,K E,imax ], the contribution of energy storage virtual inertia control and virtual droop control meets the following constraints

[0111] λ a,1 +λ b,1 ≤1

[0112] λ a,2 +λ b,2 ≤1

[0113] According to the microgrid frequency deviation, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established:

[0114]

[0115] Among them, μ1, μ2, μ3, and μ4 are weights of different intervals respectively.

[0116] Based on the reward function, a termination function is established to reduce the training time of the SAC agent:

[0117]

[0118] Among them, T termination Indicates that the agent enters the terminal state, T non-termination Indicates that the agent has not entered the terminal state.

[0119] Through the termination function, it is defined that in each episode of agent training, if the SAC agent does not enter the termination state, the agent's training in this episode is considered "successful" and the data samples in this episode are "successful experience"; if the SAC agent enters the termination state, the agent's training in this episode is considered "failed" and the sample data in this episode are "failed experience".

[0120] Step 3: Based on the microgrid frequency control model established in Step 1 and the action space, state space, and reward function established in Step 2, we set up success and failure experience pools to improve the SAC algorithm and increase the training speed of the SAC agent. This interactive training with the microgrid environment generates a multi-energy storage coordinated frequency regulation method.

[0121] The SAC algorithm is the first deep reinforcement learning algorithm based on maximum entropy. By introducing information entropy, it improves the exploration ability of deep reinforcement learning agents. Unlike traditional deep reinforcement learning, the SAC algorithm pursues the maximization of long-term rewards and entropy:

[0122]

[0123] Among them, π * is the optimal control strategy of the intelligent agent; α is the temperature coefficient; H(π(·|s t )) is the information entropy, and the specific function is:

[0124] H(π(·|s t ))=-Εlnπ(a t ∣s t )

[0125] Among them, π(a t |s t ) means in s t Next take a t After the introduction of information entropy, the state value of the SAC agent is modified as follows:

[0126] V(s t )=E[Q(s t ,a t )-αlnπ(a t ∣s t )]

[0127] The SAC algorithm is a maximum entropy reinforcement learning algorithm under the actor-cirtic framework. The present invention improves the SAC algorithm by setting a success experience pool and a failure experience pool, thereby increasing the training speed of the SAC agent. It uses interactive training with the environment to generate a multi-energy storage coordinated frequency modulation method, thereby improving the frequency regulation capability of the microgrid with multi-energy storage.

[0128] The SAC agent includes two types of network structures: action network and evaluation network. The action network outputs variables in the action space and acts on the environment, while the evaluation network evaluates the action.

[0129] The optimization update of the action network parameters φ is performed by minimizing the relative entropy.

[0130]

[0131] in is the partition function, which has no effect on the update of network parameters. By simplifying the relative entropy, the minimization optimization target J(φ) of the action network parameter φ is obtained as follows:

[0132]

[0133] where Q θ (s t ,a t ) is the output of the evaluation network.

[0134] The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is:

[0135] y SAC =r t +γE[(Q θ' (s t+1 ,a t+1 )-αln(π φ (s t+1 ∣a t+1 )))]

[0136]

[0137] Where γ is the discount coefficient, Q θ' (s t+1 ,a t+1 ) is the target evaluation network output value.

[0138] Unlike traditional deep reinforcement learning algorithms, the goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. In order to reduce the sensitivity of the SAC algorithm to parameters, the temperature coefficient α is updated during training, and the minimization optimization goal is:

[0139]

[0140] Here, H0 is the entropy limit.

[0141] Based on the action network parameter φ, the optimization target of the evaluation network parameter θ and the temperature coefficient α, the parameters are updated using the gradient descent method:

[0142]

[0143] Among them, β φ , β θ , β α are the learning rate of the action network parameter φ, the evaluation network parameter θ and the temperature coefficient α, respectively.

[0144] The target evaluation network parameter θ' is updated in a soft update mode:

[0145] θ′=τθ+(1-τ)θ′

[0146] Where τ is the update coefficient.

[0147] To speed up the training of the SAC agent, the SAC algorithm is improved by setting up a successful experience pool and a failed experience pool to store samples of agent interactions. According to the set termination function, the training successful sample data is placed in the successful experience pool, and the training failed sample data is placed in the failed experience pool. ξ sample data are extracted from each of the successful experience pool and the failed experience pool. ξ is obtained by the adaptive batch function:

[0148]

[0149] ξ=[ρ·N]

[0150] Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [·] means rounding down.

[0151] By setting up a successful experience pool and a failed experience pool, the SAC agent can be prevented from learning from the same type of experience data during training, improving its generalization ability and sample data utilization efficiency. The improved SAC agent continuously interacts with the established microgrid model to generate sample data. Data is then extracted from the two experience pools to train the action network, evaluation network, and target evaluation network. As the number of training sets increases, the long-term rewards obtained by the agent gradually converge, ultimately resulting in a multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning.

[0152] The training process is as follows: Figure 4 :

[0153] Initialize the evaluation network parameters θ, the evaluation main network parameters θ, the target evaluation network parameters θ', clear the successful experience pool and the failed experience pool, and set the number of training sets:

[0154] (1) Initialize the output of each distributed power source and obtain the frequency deviation and frequency deviation change rate as the state s t ;

[0155] (2) Calculate action a t Act on the environment to get the next state s t+1 , environmental feedback reward r t ;

[0156] (3) Let s t =s t+1 , the agent continues to interact with the environment until the last state;

[0157] (4) Determine whether the agent has entered the termination state. If it is stored (s t ,a t ,r t ,s t+1 ) to the failure experience pool, otherwise the sample is stored in the success experience pool;

[0158] (5) Sample from the experience pool, update the evaluation network parameters, target evaluation network parameters, action network parameters, and adjust the temperature coefficient:

[0159] (6) Whether the microgrid environmental constraints are met, if yes, go to step (2), otherwise go to step (5);

[0160] (7) Determine whether the reward obtained by the agent in each episode meets the requirements or whether the number of training episodes is completed. If so, end the training; otherwise, go to step (1);

[0161] After the training, based on the improved SAC agent, the contribution of the virtual inertia control and virtual droop control of the two energy storages is improved according to the frequency deviation and frequency deviation change rate of the microgrid, so as to avoid sudden changes in the output of the energy storage due to inappropriate control methods. At the same time, the virtual droop coefficients of the two energy storages are improved. While achieving a fast response speed, the range of the active power output of the energy storage is increased, the coordination between different energy storages is achieved, and the anti-interference performance of the entire microgrid is improved.

[0162] Another aspect of the present invention provides a multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning, comprising:

[0163] The first module is used to establish a frequency control model for a microgrid that includes a micro gas turbine, battery energy storage, wind power, photovoltaic power, and loads. The loads, wind power, and photovoltaic power serve as disturbance sources, while the micro gas turbine and battery energy storage are responsible for frequency regulation. Multiple battery energy storage systems support the microgrid, and the control methods are virtual inertia control and virtual droop control.

[0164] The second module introduces the SAC algorithm. By establishing the action space, state space, and reward function, it regulates the contribution of virtual inertia control and virtual droop control, while also improving the virtual droop coefficient to resolve the conflict between the response speed and stability of energy storage.

[0165] The third module is used to improve the SAC algorithm based on the established microgrid frequency control model and the established action space, state space and reward function, set up successful experience pools and failed experience pools, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency regulation method through interactive training with the microgrid environment.

[0166] Another aspect of the present invention provides a multi-energy storage coordinated frequency modulation system based on improved maximum entropy deep reinforcement learning, comprising: a computer-readable storage medium and a processor;

[0167] The computer-readable storage medium is used to store executable instructions;

[0168] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning described in the first aspect.

[0169] On the other hand, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning described in the first aspect is implemented.

[0170] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0171] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0172] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning, characterized in that: The steps include: Step 1: Establish a frequency control model for a microgrid that includes a micro gas turbine, battery energy storage, wind power, photovoltaic power, and loads. The loads, wind power, and photovoltaic power serve as disturbance sources, while the micro gas turbine and battery energy storage are responsible for frequency regulation. Multiple battery energy storage systems support the microgrid, and the control methods are virtual inertia control and virtual droop control. Step 2: Introduce the SAC algorithm. By establishing the action space, state space, and reward function, the contribution of virtual inertia control and virtual droop control is controlled and adjusted. At the same time, the virtual droop coefficient is improved to resolve the contradiction between the response speed and stability of energy storage. Step 3: Based on the frequency control model of the microgrid established in step 1 and the action space, state space, and reward function established in step 2, the successful experience pool and the failed experience pool are set to improve the SAC algorithm, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency modulation method through interactive training with the microgrid environment.

2. The multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning according to claim 1, characterized in that: The frequency control model has two battery energy storages, and the frequency control model of the microgrid is: ； ； ； ； ； ； ； ； ； in, is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, is the output power increment of the micro gas turbine, is the output power increment of the photovoltaic system, ΔP W is the power increment output by the wind power system, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, is the time constant of the gas turbine, is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; is the adjustment coefficient; is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, is the photovoltaic time constant, ΔP wind， is the initial wind power variation, is the time constant of the wind turbine, i=1,2, are the power contributions of energy storage virtual inertia control and virtual droop control, are the virtual inertia power coefficient and virtual droop power coefficient, They are the energy storage adjustment instructions for virtual inertia control and virtual droop control, is the energy storage time constant.

3. The multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning according to claim 2, characterized in that: In step 2, the SAC agent obtains frequency information from the microgrid as the state space. The frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation. The action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients: ； ； Among them, s is the state space, a is the action space, is the frequency deviation, are the contributions of the virtual inertia control and virtual droop control of the two battery energy storages, are the virtual droop power coefficients of the two battery energy storages, The contribution of energy storage virtual inertia control and virtual droop control satisfies the following constraints: ； According to the microgrid frequency deviation, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established: ； in, are the weights of different intervals respectively; Based on the reward function, a termination function is established to reduce the training time of the SAC agent: 。 4. The multi-energy storage coordinated frequency modulation method based on improved maximum entropy deep reinforcement learning according to claim 3 is characterized in that: Step 3 specifically includes: training the SAC agent based on the microgrid model established in step 1 and the action space, state space, and reward function established in step 2. The SAC algorithm, as a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, includes two network structures: an action network and an evaluation network. The action network outputs variables in the action space and acts on the environment, while the evaluation network evaluates the actions. The optimization objective of minimizing the action network parameter ϕ is for ； in, For s t Output a t The probability of α is the temperature coefficient, To evaluate the output of the network; The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is: ； ； Where γ is the discount coefficient, Evaluate the network output value for the target; The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is: ； Where H0 is the entropy limit; The target evaluation network parameter θ' is updated in a soft update mode: ； Where τ is the update coefficient; In order to improve the training speed of the SAC agent, the SAC algorithm is improved by setting a successful experience pool and a failed experience pool to store samples of agent interactions. ξ sample data are extracted from the successful experience pool and the failed experience pool respectively. ξ is obtained by the adaptive batch function: ； ； Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [•] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After a pre-learning process, the action network, evaluation network, and target evaluation network are updated, and the training generates a multi-energy storage collaborative frequency modulation method based on maximum entropy deep reinforcement learning.

5. A multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning, characterized in that: include: The first module is used to establish a frequency control model for a microgrid that includes a micro gas turbine, battery energy storage, wind power, photovoltaic power, and loads. The loads, wind power, and photovoltaic power serve as disturbance sources, while the micro gas turbine and battery energy storage are responsible for frequency regulation. Multiple battery energy storage systems support the microgrid, and the control methods are virtual inertia control and virtual droop control. The second module introduces the SAC algorithm. By establishing the action space, state space, and reward function, it regulates the contribution of virtual inertia control and virtual droop control, while also improving the virtual droop coefficient to resolve the conflict between the response speed and stability of energy storage. The third module is used to improve the SAC algorithm based on the established microgrid frequency control model and the established action space, state space and reward function, set up successful experience pools and failed experience pools, increase the training speed of the SAC agent, and generate a multi-energy storage coordinated frequency regulation method through interactive training with the microgrid environment.

6. The multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning according to claim 5, characterized in that: The frequency control model has two battery energy storages, and the frequency control model of the microgrid is: ； ； ； ； ； ； ； ； ； in, is the frequency deviation of the microgrid, H is the inertia time constant of the microgrid, D is the damping constant, is the output power increment of the micro gas turbine, is the output power increment of the photovoltaic system, ΔP W is the power increment output by the wind power system, P E1 ,P E1 is the power output increment of energy storage 1 and energy storage 2, ΔP L is the load change of the microgrid, ΔP G Changes in fuel system valve position, is the time constant of the gas turbine, is the time constant of the fuel system, ΔP C is the control signal of the micro gas turbine; is the adjustment coefficient; is the proportional coefficient and integral coefficient, ΔP solar is the initial optical power change, is the photovoltaic time constant, ΔP wind， is the initial wind power variation, is the time constant of the wind turbine, i=1,2, are the power contributions of energy storage virtual inertia control and virtual droop control, are the virtual inertia power coefficient and virtual droop power coefficient, They are the energy storage adjustment instructions for virtual inertia control and virtual droop control, is the energy storage time constant.

7. The multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning according to claim 6, characterized in that: The SAC agent obtains frequency information from the microgrid as the state space. The frequency information includes the frequency deviation of the microgrid and the rate of change of the frequency deviation. The action space is selected as the contribution of the two energy storage virtual inertia control and virtual droop control and their respective droop coefficients: ； ； Among them, s is the state space, a is the action space, is the frequency deviation, are the contributions of the virtual inertia control and virtual droop control of the two battery energy storages, are the virtual droop power coefficients of the two battery energy storages, The contribution of energy storage virtual inertia control and virtual droop control satisfies the following constraints: ； According to the microgrid frequency deviation, a function is established to transform the frequency minimization problem into a reward maximization problem, and a reward function is established: ； in, are the weights of different intervals respectively; Based on the reward function, a termination function is established to reduce the training time of the SAC agent: 。 8. The multi-energy storage coordinated frequency modulation device based on improved maximum entropy deep reinforcement learning according to claim 7, characterized in that: The third module is specifically used to: The SAC agent is trained based on the established microgrid model and the established action space, state space, and reward function. The SAC algorithm, as a maximum entropy reinforcement learning algorithm under the actor-cirtic framework, includes two network structures: an action network and an evaluation network. The action network outputs variables in the action space and acts on the environment, while the evaluation network evaluates the actions. The optimization objective of minimizing the action network parameter ϕ is for ； in, For s t Output a t The probability of α is the temperature coefficient, To evaluate the output of the network; The evaluation network includes two networks: the main network and the target network. The mean square loss function for evaluating the parameters θ of the main network is: ； ； Where γ is the discount coefficient, Evaluate the network output value for the target; The goal of the maximum entropy SAC algorithm is to maximize long-term rewards and expected entropy. The entropy changes accordingly at different stages of training. During the training process, the optimization goal of the temperature coefficient α is: ； Where H0 is the entropy limit; The target evaluation network parameter θ' is updated in a soft update mode: ； Where τ is the update coefficient; In order to improve the training speed of the SAC agent, the SAC algorithm is improved by setting a successful experience pool and a failed experience pool to store samples of agent interactions. ξ sample data are extracted from the successful experience pool and the failed experience pool respectively. ξ is obtained by the adaptive batch function: ； ； Among them, M is the number of training sets, M max is the set number of training sets, N is the originally set number of samples, and [•] represents rounding down. The SAC agent obtains frequency information from the microgrid and outputs actions to act on the environment. After a pre-learning process, the action network, evaluation network, and target evaluation network are updated, and the training generates a multi-energy storage collaborative frequency modulation method based on maximum entropy deep reinforcement learning.

9. A multi-energy storage coordinated frequency modulation system based on improved maximum entropy deep reinforcement learning, comprising: Computer-readable storage medium and processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning described in any one of claims 1-4.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-energy storage collaborative frequency modulation method based on improved maximum entropy deep reinforcement learning according to any one of claims 1 to 4.

Images