A multi-microgrid frequency management method based on deep reinforcement learning

Abstract

The present invention relates to a multi-microgrid frequency management method based on deep reinforcement learning, the method comprising the following steps: 1) constructing a shared energy storage aggregation platform of multi-microgrids with physical energy storage and virtual energy storage, and constructing an isolated multi-microgrid containing the shared energy storage aggregation platform and its load frequency control model based on the shared energy storage aggregation platform; 2) proposing an AGC instruction allocation method considering the optimal comprehensive benefit of the microgrid, the comprehensive benefit taking into account the two key elements of microgrid operation cost and frequency regulation performance; 3) proposing an AGC instruction dynamic allocation optimization method based on the DDPG algorithm, and constructing the state, action and reward function of the AGC instruction dynamic allocation method. Compared with the prior art, the present invention realizes the centralized management of shared energy storage by aggregating physical energy storage and virtual energy storage, and provides frequency auxiliary services for multi-microgrids, taps the adjustable capacity of the demand side of the microgrid, gives full play to the superior frequency regulation performance of different frequency regulation equipment, and can improve the economy of microgrid operation, meet the adaptive and real-time requirements of multi-microgrid frequency management, and also enhances the ability to cope with the uncertainty of microgrid source and load.

Description

Technical Field

[0001] The present invention relates to a multi-microgrid frequency management method based on deep reinforcement learning, and belongs to the field of microgrid frequency control. Background Art

[0002] Due to the randomness of renewable energy and the large-scale access to microgrids, the frequency stability operation of microgrids is greatly challenged. Frequency and tie-line power control in multi-microgrids is achieved through automatic generation control (AGC), which is one of the important measures to maintain frequency stability and economic operation of microgrids. With the reduction of energy storage (ES) costs and the increase of user-side electrification rate, more types of adjustable resources are used to respond to AGC instructions of multi-microgrids. On the power supply side, scholars such as Hu Zechun and Li Yanjun proposed active power control strategies including variable pitch angle control and rotor speed control, and incorporated wind turbines into automatic generation control to improve frequency regulation performance. In addition, scholars such as Kashem M. Muttaqi also proposed auxiliary control strategies to enable hybrid ES to cooperate with wind turbines to assist in frequency management. In terms of ES, the value of ES in providing fast frequency response has been accepted by more and more people. In the frequency coordination control strategy proposed by He Xingtang, the fast response performance of the hybrid ES composed of supercapacitors and batteries is used to improve frequency stability. In some power systems that can operate independently, such as the PJM power grid in the United States and the China Southern Power Grid, ES are encouraged to participate in frequency regulation and earn benefits based on actual frequency regulation performance and frequency regulation mileage. On the user side, flexible loads, especially variable frequency air conditioners, water heaters, dryers, refrigerators, electric vehicles, etc., are a promising solution to promote stable frequency operation of power systems. Hui Hongxun and other scholars studied an electrical model of variable frequency air conditioners to enable them to participate in frequency regulation services. Xu Yan and other scholars proposed a hierarchical control framework to enable variable frequency air conditioners to participate in primary frequency regulation through load aggregation. Mu Chaoxu and other scholars proposed an adaptive control strategy using adaptive dynamic programming algorithm and sliding mode control algorithm for smart grids with electric vehicle integration to verify the effectiveness of electric vehicles in improving frequency regulation. With the development of technology and the reduction of costs, more adjustable resources are used to achieve better frequency management. However, one of the key issues that need to be solved is to rationally optimize and allocate regulation resources to improve frequency regulation performance while reducing frequency operation costs. In addition, real-time and adaptability must be met in the AGC command allocation problem, and Deep Reinforcement Learning (DRL) is widely used due to its strong adaptive learning ability and optimal decision-making ability for non-convex nonlinear problems. Xi Lei and other scholars designed a DRL framework for AGC control for large-scale power systems with multi-region interconnection and improved the Q-Learning algorithm.Xu Yan and other scholars proposed a multi-agent DRL based on the Deep Deterministic Policy Gradient (DDPG) algorithm to improve the frequency and interconnection line power performance by coordinating load frequency control in multi-regional power systems. Based on the above analysis, DRL has obvious advantages in frequency regulation. However, most of the research on frequency regulation in multi-microgrids focuses on the optimization of frequency and interconnection line power, and further research is needed on economic optimization methods that consider frequency regulation mileage and regulation performance. Based on the above analysis, the present invention establishes a multi-microgrid frequency management method based on deep reinforcement learning. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-microgrid frequency management method based on deep reinforcement learning.

[0004] A multi-microgrid frequency management method based on deep reinforcement learning, the method comprising the following steps:

[0005] S1: Construct a shared energy storage aggregation platform for multiple microgrids with physical energy storage and virtual energy storage, and build an isolated multi-microgrid architecture containing a shared energy storage aggregation platform and its load frequency control model based on the shared energy storage aggregation platform;

[0006] 101) A multi-microgrid is composed of single-domain microgrids through interconnection lines. Each microgrid contains renewable energy generation systems, fossil energy generation systems (such as combined cooling, heating and power (CCHP), gas turbines, diesel generators) and loads. The sES aggregator connects the self-built physical energy storage to the multi-microgrid bus through a power converter. At the same time, the sES aggregator also manages the virtual energy storage (such as variable frequency air conditioners (AC), electric heat pumps, robotic arms, etc.) composed of adjustable loads on the user side of the microgrid through information collection and control terminals and wireless communications. The sES aggregator needs to sign a contract with the microgrid user for the right to use adjustable equipment in order to manage the user-side virtual energy storage. Users and sES aggregators will obtain income by participating in energy transactions and ancillary service transactions in multi-microgrids. As a platform for managing physical energy storage and virtual energy storage, the sES AP of the sES aggregator will interact and trade with the microgrid in the form of leasing and selling the right to use energy storage power and capacity. The present invention only considers the frequency auxiliary services of sES aggregators. Microgrid operators realize power balance and frequency management in the region. SES aggregators realize the management of physical energy storage and virtual energy storage and provide frequency support for microgrids. Figure 1 shown.

[0007] 102) Taking batteries as an example of physical energy storage, the frequency response of batteries is described by the first-order inertia link and battery gain, as shown in (1).

[0008]

[0009] Where, T b , K b , ΔP b are the time constant, gain and output power of the battery respectively, and Δf is the frequency of the microgrid.

[0010] Taking the variable-frequency air conditioner as an example of virtual energy storage, the frequency response of the variable-frequency air conditioner is obtained by deriving and simplifying the thermoelectric model of the room, as shown in (2).

[0011]

[0012] In the formula, K iac and ΔP iac are the gain and output power of the frequency response of the variable frequency air conditioner, C(s) is the temperature control function of the frequency response of the variable frequency air conditioner, D(s) is the control function of the variable frequency air conditioner participating in frequency management, T c is the time constant of the variable frequency air conditioner compressor, T a (s) and Φ iac (s) is a function of the changing temperature and heat of the room.

[0013] The shared energy storage aggregation platform accepts the AGC commands assigned by the microgrid operator and distributes them to physical energy storage and virtual energy storage devices for execution. The actual output power of the shared energy storage can be described as shown in (3).

[0014]

[0015] In the formula, F i (s) is the transfer function of shared energy storage frequency management, g = 2, F 1 (s) = G b (s), F 2 (s) = G iac (s). It is the AGC instruction that the shared energy storage needs to participate in the secondary frequency regulation. pi is the droop coefficient of primary frequency regulation.

[0016] The power change of shared energy storage frequency management needs to meet the following constraints:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023] Where ΔP sES,c , ΔP sES,d , and They are the shared energy storage charging and discharging power and the maximum charging and discharging power. α is a 0-1 variable, when α=1 it means charging, when α=0 it means discharging. and They are the maximum powers of the variable frequency air conditioner and battery participating in frequency management. and They are the maximum capacity of the battery involved in frequency management and the capacity at time t. b,c and η b,d is the charge and discharge efficiency of the battery. A , and They are the actual temperature of the room and the maximum and minimum values ​​of the set temperature.

[0024] The power balance equation of multi-microgrid is shown in (10).

[0025] ΔP L,m =ΔP d,m +ΔP sES,m +ΔP PV,m -ΔP tie (10)

[0026] Where ΔP L,m , ΔP d,m , ΔP sES,m , ΔP PV,m and ΔP tie They are the uncontrollable load power of the microgrid in the mth area, the output power of the diesel generator, the output power of the shared energy storage, the photovoltaic power generation power and the interconnection line power.

[0027] The tie line power is shown in formula (11).

[0028]

[0029] Where, T tie,m-l is the synchronization coefficient of the interconnection line between the microgrids in the mth and lth regions.

[0030] The change of the regional control error (ACE) of the m-th regional microgrid is shown in (12).

[0031] ΔACE m =ΔP tie +k m Δf m (12)

[0032] In the formula, k m is the frequency deviation coefficient.

[0033] The state vector of the m-th area microgrid is x = [Δf, ΔP d ,ΔP sES ,ΔP PV ,ΔP L ,ΔP tie ,ΔACE]. The output power of photovoltaic and uncontrollable load are regarded as uncertain disturbances. The frequency regulation power of multi-microgrid needs to meet the following constraints.

[0034]

[0035]

[0036] |Δf|≤λ f f rate (15)

[0037]

[0038] In the formula, and It is the maximum and minimum output power of the diesel generator. is the rated climbing power of the diesel generator set. rate is the rated frequency of the microgrid, f rate =50Hz. is the maximum power of the tie line. f and λ t It is the allowable deviation coefficient of frequency and interconnection line power.

[0039] S2: A method for allocating AGC commands considering the optimal comprehensive benefit of the microgrid is proposed. The comprehensive benefit takes into account two key factors: the microgrid operation cost and the frequency regulation performance.

[0040] 201) In the constructed isolated multi-microgrid with a shared energy storage aggregation platform, the frequency regulation instruction, namely the AGC instruction, is responded to by the diesel generator owned by the microgrid and the energy storage equipment (batteries and variable-frequency air conditioners) managed by the sES AP. The frequency management process is as follows: First, the microgrid operator collects the user's energy consumption information and photovoltaic power generation information from the terminal, and the sES AP reports the status information of the energy storage to the microgrid operator. Then, the microgrid operator calculates the regional control error in real time, and the AGC dynamic allocation controller based on deep reinforcement learning generates AGC instructions. Finally, the AGC instructions are executed by the diesel generator of the microgrid, the battery of the shared energy storage aggregator, and the variable-frequency air conditioner respectively. A microgrid frequency management framework containing a shared energy storage aggregation platform is shown in the attached figure. Figure 2 shown.

[0041] 202) An AGC command allocation method considering the optimal comprehensive benefits of the microgrid is proposed. The dynamic allocation target of the AGC command of a single-area microgrid is decomposed into three parts. The first part uses the minimum shared energy storage mileage compensation and the minimum diesel generator cost to ensure the economy of the microgrid operation. The second part uses the minimum AGC command response error to ensure the frequency regulation performance of different devices. The third part uses the minimum regional control error to reduce the impact of source-load uncertainty on the stable operation of the microgrid. The dynamic allocation objective function of the AGC command is shown in (17).

[0042]

[0043] In the formula, c milI , M I and A I are the mileage compensation unit price, mileage and performance index A of frequency management equipment I respectively I =1, I = 1 indicates a diesel generator, I = 2 indicates shared energy storage. G is the type of frequency management equipment, G = 2. c fuel is the unit fuel consumption price of diesel generator. ce is the unit price of carbon emissions. k ce is the carbon emission penalty factor. ΔP I-2 is the actual output power of the frequency management device I participating in the secondary frequency regulation. N = 75 is the control cycle. Each control cycle of the AGC instruction is 4 seconds. μ is the weight coefficient, k pr is the proportionality coefficient. ΔP ord-all is the AGC general instruction, as shown in (18).

[0044] ΔP ord-all =ΔP ord-d +ΔP ord-sES1 +ΔP ord-sES2 (18)

[0045] Where ΔP ord-d, ΔP ord-sES1 , ΔP ord-sES2 These are the AGC instructions executed by the diesel generator, physical energy storage (battery), and virtual energy storage (inverter air conditioner) respectively.

[0046] The mileage of frequency management equipment I is shown in (19).

[0047] M I (t)=|ΔP ord-I (t)-ΔP ord-I (t-1)| (19)

[0048] S3: A DDPG algorithm-based AGC instruction dynamic allocation optimization method is proposed, and the state, action and reward functions of the AGC instruction dynamic allocation method are constructed.

[0049] 301) proposed a method for optimizing the dynamic allocation of AGC instructions based on the DDPG algorithm. This method provides an adaptive learning strategy through continuous interaction between the agent and the environment, which makes it have obvious advantages in solving the uncertain fluctuations of microgrid sources and loads. The DDPG algorithm is a kind of reinforcement learning. The basic components of reinforcement learning include state set S, action set A and reward r. The AGC instruction controller is regarded as an agent, the LFC model of the microgrid is regarded as the environment, and the state s of the environment is t Provided to the agent, the agent follows the strategy and status t Generate actions. Before the online application of the DDPG-based reinforcement learning framework, the agent training needs to be carried out through offline learning. During the offline training of the agent, the random fluctuations of renewable energy generators and loads are injected into the multi-microgrid as input. The update period of the AGC command is set to 4 seconds, and the frequency management period of the microgrid is 5 minutes, so there are 75 AGC command update cycles in one frequency management period. In each AGC command update period, the agent gives actions, rewards and the state of the next moment based on the DDPG algorithm according to the current state of the microgrid. The training samples are generated and stored in the experience pool through the interaction between the agent and the environment. After a lot of offline training, the agent obtains the AGC command allocation strategy with the best overall benefit. The state of the microgrid is determined by the attached Figure 2 The terminal information collection device in the system collects and calculates the status information, and then the status information is transmitted to the AGC instruction allocation controller of the microgrid operator. The controller is an intelligent agent that generates actions based on the optimal strategy. It generates the optimal AGC instruction allocation strategy through the status and the optimal strategy. Finally, various types of frequency management devices coordinate and execute according to the optimal AGC instruction allocation strategy to assist the microgrid in achieving frequency management.

[0050] 302) The environment state of an AGC instruction dynamic allocation optimization method based on the DDPG algorithm is shown in (20), and the action of the intelligent agent is shown in (21). The goal of the intelligent agent is to maximize the comprehensive benefits of the microgrid operation, that is, the optimal AGC instruction allocation. The intelligent agent uses the maximum cumulative reward as the final reward, and the reward function of the intelligent agent is r t (s t ,a t ) as shown in (22).

[0051] s t ={Δf(t),ΔP tie (t),ΔP I (t),ΔP ord-all (t)} (20)

[0052] a t = {ΔP ord-d (t),ΔP ord-sES1 (t),ΔP ord-sES2 (t)} (21)

[0053]

[0054] In the formula, X r-1 is the microgrid operating cost, X r-2 is the performance reward of the frequency management equipment in response to AGC instructions, X r-3 It is a performance reward for microgrid frequency management. r-1 ,ξ r-2 ,ξ r-3 is a reduction factor to ensure that each term is of the same magnitude. k 1 , k 2 , k 3 is the weight coefficient, k 1 +k 2 +k 3 =1. p ={ε,stL is false||0,stL is true} is the penalty term of the constraint condition to ensure that the microgrid frequency management is within a reasonable range. ε is the penalty value, and L is the constraint condition (3)-(9), (13)-(16).

[0055] When a state s t After being determined, the agent action a t The reward value obtained can be expressed as the action-value function Q π (s,a) as shown in (23).

[0056]

[0057] In the formula, E π(·) is the expectation of policy π. γ∈[0,1] is the decay coefficient.

[0058] The goal of AGC instruction allocation is to find the optimal strategy π * To implement the action-value function The optimal strategy is shown in (24).

[0059] π * = argmaxQ π (s t ,a t ) (twenty four)

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] (1) The present invention constructs a multi-microgrid frequency management framework including a shared energy storage aggregation platform, realizes centralized management of shared energy storage by aggregating physical energy storage and virtual energy storage, and provides frequency auxiliary services for multiple microgrids;

[0062] (2) The present invention adopts an AGC instruction allocation method that considers the optimal comprehensive benefits of the microgrid, takes into account the microgrid frequency management performance and the microgrid operation cost, taps the adjustable capacity of the microgrid demand side, gives full play to the advantageous frequency regulation performance of different frequency regulation equipment, and can improve the economic efficiency of microgrid operation;

[0063] (3) The present invention adopts the DDPG algorithm of deep reinforcement learning to optimize the AGC instruction allocation, which meets the adaptive and real-time requirements of multi-microgrid frequency management, and also enhances the ability to cope with the uncertainty of microgrid source and load. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of an isolated multi-microgrid with a shared energy storage aggregation platform;

[0065] Figure 2 A frequency management framework for microgrids with a shared energy storage aggregation platform;

[0066] Figure 3 This is a typical interconnected island multi-microgrid architecture diagram;

[0067] Figure 4 is the random net load fluctuation curve;

[0068] Figure 5 Frequency changes for different frequency management methods;

[0069] Figure 6 The tie line power variation for different frequency management methods;

[0070] Figure 7 (a) is the output power of the device based on the PID frequency management method;

[0071] Figure 7 (b) Output power of the device based on the FD frequency management method;

[0072] Figure 7 (c) Output power of the device based on the PA frequency management method;

[0073] Figure 7 (d) Output power of equipment based on SM frequency management method;

[0074] Figure 7 (e) is the output power of the equipment based on the IB frequency management method. DETAILED DESCRIPTION

[0075] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Example

[0077] A typical isolated multi-microgrid is composed of two interconnected microgrids and an SES aggregator. Figure 3 As shown. Microgrid 1 has a 600kW diesel generator, a 2MW photovoltaic system, and a 1MW load, including a 400kW adjustable load. The sES aggregator has a 250kW / 500kWh battery pack and has signed an equipment use rights contract with the adjustable load of Microgrid 1. Microgrid 2 has an 800kW diesel generator, a 250kW / 500kWh battery pack, a 1.6MW photovoltaic system, and a 1MW load, including a 200kW adjustable load. The adjustable load of Microgrid 2 has not signed an equipment use rights contract with the sES aggregator, and all equipment in Microgrid 2 is managed by the operator of Microgrid 2. The reference power of the multi-microgrid is 4MW. The system parameters of the multi-microgrid are shown in Table 1.

[0078] Table 1 Parameters of interconnected island multi-microgrid system

[0079]

[0080] According to a multi-microgrid frequency management method based on deep reinforcement learning proposed by the present invention, this example compares and analyzes 5 different frequency management methods. The 5 frequency management methods are: the traditional frequency management method based on PID, represented by PID-based, the frequency management method based on frequency division (FD) applied to the PJM power market in the United States, represented by FD-based, the frequency management method based on proportional allocation (PA), represented by PA-based, the frequency management method based on sliding mode control (SM), represented by SM-based, and the frequency management method based on integrated benefits (IB) proposed by the present invention, represented by IB-based. In the PID-based frequency management method, all frequency regulation devices respond to the same AGC instruction. In the FD-based frequency management method, the AGC instruction is decomposed into high-frequency components and low-frequency components, and is responded by devices with different performance. In the PA-based frequency management method, the AGC command allocates the frequency regulation power of the device according to the proportion of the total frequency regulation power. SM-Based Frequency Management Method In the FD-based frequency management method, a SM compensation controller is designed to improve the robust regulation capability of the microgrid.

[0081] The results of the embodiment are analyzed with average reward (AR), microgrid operation cost (MC), average frequency deviation (AFD) and average tie line deviation (ATD) as the main evaluation indicators. The expressions of AR, MC, AFD and ATD are as follows:

[0082]

[0083]

[0084]

[0085]

[0086] This example uses the random net load fluctuation curve as input, as shown in the attached Figure 4 As shown in Figure 1, the net load fluctuation curve is obtained by subtracting the photovoltaic power generation curve from the load curve. Figure 5 is a frequency change diagram of different frequency management methods, attached Figure 6 This is the power variation diagram of the tie line under different frequency management methods. Figure 7 It is the equipment output power diagram of different frequency management methods. Table 2 is the evaluation index results of different frequency management methods.

[0087] Table 2 Evaluation index results of different frequency management methods

[0088]

[0089] According to the above Table 2, Appendix Figure 5 , Attachment Figure 6 and attached Figure 7 It can be analyzed that the multi-microgrid frequency management method based on deep reinforcement learning proposed in the present invention is second only to the frequency management method based on PID in the AR evaluation index. The frequency management method based on PID is the best method in all evaluation indicators among all methods. This is because the structure of the PID controller is simple, but the parameter setting of the PID controller of different models is relatively cumbersome, and continuous debugging is required to obtain the best performance of the model. The frequency management method based on IB is closest to the frequency management method based on PID. For the MC evaluation index, the gap between the two methods is less than 9, while the operating costs of the frequency management methods based on FD, PA and SM are much greater than 102, especially the operating costs of the frequency management methods based on FD and SM are more than 100 higher than those of the frequency management method based on IB. For the AFD and ATD evaluation indicators, the frequency modulation method based on IB reduces the operating cost of the microgrid by sacrificing less frequency performance, achieving the best comprehensive benefit of the microgrid operation. The frequency management method based on IB can dynamically give the optimal AGC instruction allocation strategy according to different source-load fluctuations, achieving the best comprehensive benefit of the microgrid operating cost and frequency performance. In addition, the output power of the equipment through the IB-based frequency management method can also reflect the dynamic allocation of AGC instructions according to the best comprehensive benefits, and the user-side adjustable virtual energy storage is given priority to responding to AGC instructions, which will reduce the use of diesel generators and physical energy storage, thereby reducing the operating cost of the microgrid and the frequency regulation power / capacity requirements of physical energy storage.

[0090] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed by the present invention, and these modifications or substitutions should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multi-microgrid frequency management method based on deep reinforcement learning, characterized in that: The method comprises the following steps: 1) A shared energy storage aggregation platform for multiple microgrids is constructed with physical energy storage and virtual energy storage, and an isolated multi-microgrid architecture containing a shared energy storage aggregation platform and its load frequency control model are constructed based on the shared energy storage aggregation platform; an isolated multi-microgrid load frequency control model containing a shared energy storage aggregation platform is constructed, which consists of a physical energy storage frequency response function, a virtual energy storage frequency response function, a multi-microgrid power balance function, a tie line power function, a regional control error function and constraints; 2) Consider the optimal AGC command allocation for the comprehensive benefit of the microgrid, which takes into account both the operating cost and frequency regulation performance of the microgrid. The specific steps include: The microgrid operator collects the user's energy consumption information and photovoltaic power generation information from the terminal, and the sES AP reports the energy storage status information to the microgrid operator; The microgrid operator calculates the regional control error in real time, and the AGC command is generated by the AGC dynamic allocation controller based on deep reinforcement learning; The AGC commands are executed by the diesel generators of the microgrid, the batteries of the shared energy storage aggregator, and the variable frequency air conditioner respectively; The dynamic allocation target of the AGC command of a single-area microgrid is decomposed into three parts; the first part uses the minimum shared energy storage mileage compensation and the minimum diesel generator cost to ensure the economy of the microgrid operation; the second part uses the minimum AGC command response error to ensure the frequency regulation performance of different devices; the third part uses the minimum regional control error to reduce the impact of source-load uncertainty on the stable operation of the microgrid; the dynamic allocation target function of the AGC command is as follows: f(t)=X1+X2+X3 In the formula, c milI , M I and A I are the mileage compensation unit price, mileage and performance index A of frequency management equipment I respectively I =1, I = 1 indicates a diesel generator, I = 2 indicates shared energy storage; G is the type of frequency management equipment, G = 2; c fuel is the unit fuel consumption price of diesel generator; c ce is the unit price of carbon emissions; k ce is the carbon emission penalty factor; ΔP I-2 is the actual output power of the frequency management device I participating in the secondary frequency regulation; N = 75 is the control period; each control period of the AGC command is 4 seconds; μ is the weight coefficient, k pr is the proportionality coefficient; 3) Based on the DDPG algorithm, the dynamic allocation optimization of AGC instructions is carried out to construct the state, action and reward functions of the dynamic allocation of AGC instructions; the AGC instruction controller is regarded as an intelligent agent, the LFC model of the microgrid is regarded as the environment, and the state of the environment s t Provided to the agent, the agent follows the strategy and status t Produce action.

2. According to the multi-microgrid frequency management method based on deep reinforcement learning according to claim 1, it is characterized in that: In step 1), an isolated multi-microgrid architecture with a shared energy storage aggregation platform is constructed. An isolated multi-microgrid is composed of single-domain microgrids through interconnection lines. Each microgrid contains renewable energy power generation systems, fossil energy power generation systems and loads. The sES aggregator connects the self-built physical energy storage to the isolated multi-microgrid bus through a power converter. At the same time, the sES aggregator also manages the virtual energy storage composed of adjustable loads on the user side of the microgrid through information collection control terminals and wireless communications.

3. The multi-microgrid frequency management method based on deep reinforcement learning according to claim 1, characterized in that: ΔP ord-all is the AGC general instruction, as shown below: ΔP ord-all =ΔP ord-d +ΔP ord-sES1 +ΔP ord-sES2 In the formula, ΔP ord-d , ΔP ord-sES1 , ΔP ord-sES2 These are the AGC instructions executed by the diesel generator, physical energy storage, and virtual energy storage respectively.

4. The multi-microgrid frequency management method based on deep reinforcement learning according to claim 1, characterized in that: The mileage of frequency management equipment I is as follows: M I (t)=|ΔP ord-I (t)-ΔP ord-I (t-1)|。 5. The multi-microgrid frequency management method based on deep reinforcement learning according to claim 1, characterized in that: In step 3), the environment state of an AGC instruction dynamic allocation optimization method based on the DDPG algorithm is as follows: s t ={Δf(t),ΔP tie (t),ΔP I (t),ΔP ord-all (t)} The actions of the agent are as follows: a t ={ΔP ord-d (t),ΔP ord-sES1 (t),ΔP ord-sES2 (t)} The goal of the intelligent agent is to maximize the comprehensive benefits of microgrid operation, that is, to optimize the AGC instruction allocation; the intelligent agent uses the maximum cumulative reward as the final reward, and the reward function of the intelligent agent is r t (s t ,a t ) is as follows: r t (s t ,a t )=-k1ξ r-1 X r-1 (t)-k2ξ r-2 X r-2 (t)-k3ξ r-3 X r-3 (t)+ζ p (t) Where, X r-1 is the microgrid operating cost, X r-2 is the performance reward of the frequency management equipment in response to AGC instructions, X r-3 It is the performance reward for microgrid frequency management; r-1 ,ξ r-2 ,ξ r-3 is the reduction factor to ensure that each item is of the same magnitude; k1, k2, k3 are weight coefficients, k1+k2+k3=1; ζ p ={ε,stL is false||0,stL is true} is the penalty term of the constraint condition to ensure that the microgrid frequency management is within a reasonable range; ε is the penalty value and L is the constraint condition.

Images