Micro-grid fault recovery method based on DASAC

Abstract

The DASAC-based micro-grid fault recovery method comprises the following steps: establishing a micro-grid fault recovery model considering DG output and load demand time variation; the method comprises the following steps: converting a fault recovery problem of a micro-grid into a constrained nonlinear programming problem, constructing a double-agent reinforcement learning model by taking the minimum fault recovery cost of the micro-grid as a target, and distributing discrete and continuous double agents for different action types in an action space for control; a DASAC method is provided, efficient self-adaptive training learning is carried out on the double-agent reinforcement learning model, and the fault recovery efficiency is improved to meet the operation requirements of the micro-grid. According to the DASAC-based microgrid fault recovery method provided by the invention, discrete and continuous actions are processed respectively, so that the fault recovery efficiency is improved, the training process is simplified, and the robustness and adaptability of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of microgrid fault recovery, and in particular to a microgrid fault recovery method based on DASAC. Background Art

[0002] With the rapid development of the economy and society, people's demand for electricity is increasing. At the same time, distributed power generation technologies represented by wind turbines and photovoltaics are becoming increasingly mature, and the penetration rate of new energy in distribution networks is getting higher and higher. As a platform for integrating and managing various distributed power sources (DG), the stability of microgrid operation is directly related to the normal production and life of users. However, affected by the time-varying power demand of DG and loads, the complexity of microgrid structure and power flow has increased dramatically, which has put forward higher requirements for microgrid fault management. How to deal with the impact of time-varying DG output and load demand on the recovery of microgrids after faults and ensure that microgrids quickly resume normal operation is an issue worthy of in-depth study.

[0003] In the prior art, the literature [1]: "Research on dynamic repair strategy for multiple faults in active distribution network based on multi-agent system" (Yang Lijun, Lv Xuejiao, Li Dan, et al. Research on dynamic repair strategy for multiple faults in active distribution network based on multi-agent system [J]. Proceedings of the CSEE, 2017, 37(23): 6855-6865+7076) proposes a multi-fault rapid power supply recovery strategy for active distribution network containing DG in different regions and scenarios. The optimal fault recovery strategy is solved by using a bacterial algorithm based on the DG regulation capacity in each sub-region after the fault and the minimum comprehensive economic loss as the objective function. However, this literature only considers the output uncertainty of the power supply side, and does not construct a time-varying load demand model that considers the load power consumption preference at different times, so the practical applicability is poor.

[0004] Reference [2]: "Two-stage optimization of distribution network fault recovery considering uncertainty" (Jiang Tianyuan, Hao Lili, Wang Hui, et al. Two-stage optimization of distribution network fault recovery considering uncertainty [J]. Power Capacitors and Reactive Compensation, 2020, 41(05): 150-157.) Based on the time-varying nature of DG output and load demand, a two-stage distribution network fault recovery control model is proposed to minimize fault losses. However, this research method has large computational complexity and low efficiency, and is less adaptable to distribution systems with higher complexity.

[0005] Reference [3]: "Active distribution network fault recovery strategy based on mutation particle swarm algorithm" (Xu Yan, Zhang Hui, Sun Yizhou. Active distribution network fault recovery strategy based on mutation particle swarm algorithm [J]. Electric Power Automation Equipment, 2021, 41(12): 45-53.) The particle swarm algorithm is discretized and mutated and introduced into the fault recovery strategy solution to obtain the global optimal recovery strategy.

[0006] Reference [4]: ​​"Distribution network fault recovery strategy based on particle swarm hybrid gray wolf algorithm" (Liu Qing, Dong Haoran, Hu Ziqi, et al. Distribution network fault recovery strategy based on particle swarm hybrid gray wolf algorithm [J / OL]. Journal of North China Electric Power University (Natural Science Edition), 1-10 [2023-12-22].) Combining the strong convergence performance of the particle swarm algorithm and the good global search ability of the gray wolf optimization algorithm, a particle swarm hybrid gray wolf algorithm is proposed to solve the distribution network fault recovery strategy.

[0007] The above-mentioned references [3] and [4] have improved the convergence speed and accuracy of the algorithm to a certain extent by improving or combining intelligent algorithms, and have achieved certain results in fault recovery solutions. However, there are still some problems: a: When facing microgrids with large-scale DG access, the algorithm calculation speed is slow and cannot meet the real-time control requirements of fault recovery; b: When dealing with complex systems, the algorithm accuracy cannot meet the requirements. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a microgrid fault recovery method based on a double-agent soft actor-critic (DASAC) method. The method comprehensively considers the uncertainty of source-load and realizes the optimal recovery of microgrid faults. The method takes into account the time-varying DG output and load demand, and establishes a microgrid fault recovery model. Aiming at the problem that a high proportion of distributed power sources in microgrids have strong randomness after being affected by natural factors, the fault recovery problem of microgrids is transformed into a constrained nonlinear programming problem, and a double-agent reinforcement learning (DARL) model is constructed with the goal of minimizing the cost of microgrid fault recovery. Discrete and continuous dual agents are assigned to different action types in the action space for control. Aiming at the problem that traditional fault recovery methods have slow calculation speed and cannot realize real-time control of fault recovery, a DASAC method is proposed to efficiently and adaptively train and learn the DARL model to obtain the optimal fault recovery strategy. The microgrid fault recovery model proposed in the present invention can effectively reduce the fault recovery loss, the proposed DASAC method can significantly reduce the decision-making time, reliably learn the optimal fault recovery strategy, and compared with other methods, the recovery performance of the fault recovery method proposed in the present invention is better.

[0009] The technical solution adopted by the present invention is:

[0010] The microgrid fault recovery method based on DASAC includes the following steps:

[0011] Step 1: Establish a microgrid fault recovery model that takes into account the time-varying DG output and load demand;

[0012] Step 2: The fault recovery problem of the microgrid is transformed into a constrained nonlinear programming problem. Taking the minimization of the microgrid fault recovery cost as the goal, a double-agent reinforcement learning (DARL) model is constructed to assign discrete and continuous dual agents to control different action types in the action space.

[0013] Step 3: Propose a DASAC method to efficiently and adaptively train the dual-agent reinforcement learning model to improve the fault recovery efficiency to meet the needs of microgrid operation.

[0014] In step 1, in order to comprehensively consider the uncertainty of source-load and minimize the cost of microgrid fault recovery process, taking into account the time-varying DG output and load demand, a microgrid fault recovery model is established, which specifically includes:

[0015] (1) Constructing a source load time-varying model:

[0016] 1) Wind turbine model: The output of a wind turbine (WT) is mainly affected by wind speed. The relationship between WT output power and wind speed can be expressed as follows:

[0017]

[0018] Where: P WT Indicates WT output power; P r represents WT rated capacity; v represents wind speed; v r Indicates rated wind speed; v ci 、v co Represent the cut-in wind speed and cut-out wind speed respectively.

[0019] Since the wind speed fluctuates greatly at different times, the WT output also has a certain uncertainty. Therefore, the Weibull distribution function is used to fit the WT output in the present invention. The Weibull probability density distribution of wind speed is expressed as:

[0020]

[0021] Where: f 1 (v) represents the probability density function of wind speed; k 1 , c represents the shape parameter of the wind speed probability density function;

[0022] Represents the wind speed v in terms of shape parameters c and k 1 Attenuation characteristics under action.

[0023] 2) Photovoltaic model: Photovoltaic (PV) output is mainly affected by the intensity of solar radiation. The relationship between PV output power and solar radiation intensity can be expressed as follows:

[0024] P PV =ηAg 1

[0025] Where: P PV represents the PV output power; η represents the photoelectric conversion efficiency of the photovoltaic panel; A represents the area of ​​the photovoltaic panel; g 1 Indicates the intensity of solar radiation.

[0026] PV output is affected by changes in solar radiation intensity and has a large degree of randomness. The Beta distribution function is used to approximate the solar radiation situation:

[0027]

[0028] Where: f 2 (g 1 ) represents the probability density function of solar radiation; g 1,max represents the maximum radiation intensity of sunlight; Γ(i) represents the gamma distribution; α and β represent the shape parameters of the beta distribution function.

[0029] 3) Energy storage model: WT and PV are uncontrollable power sources, which are subject to environmental influences and have large fluctuations and time-varying characteristics, and cannot provide high-quality and continuous power to the load. However, if WT or PV is combined with an energy storage device to form a joint power generation system, and the energy storage device is used to charge and discharge in a timely manner, the problem of randomness in WT and PV output can be effectively solved. The specific charging and discharging model of the energy storage device is as follows:

[0030]

[0031] Where: P c,t and P d,t Respectively represent the charging and discharging power of energy storage during time period t; P c max and P d max Respectively represent the maximum charging and discharging power of energy storage; SOC t Indicates the remaining storage capacity during period t; SOC max and SOC min Respectively represent the maximum and minimum power allowed by energy storage; P fc,t , Pgc,t They represent the output of the wind storage system and the solar storage system at time t respectively.

[0032] 4) Load model:

[0033] a) Load time-varying model: According to different uses, the microgrid load can be divided into three categories, namely commercial, residential and industrial loads. For the load, the definite integral calculation is performed in adjacent time periods to obtain the power demand of different types of loads in the period.

[0034]

[0035] Where: L i,t represents the load demand on node i from t to t+1; f i (x) represents the load curve function at node i.

[0036] b) Load demand model: In the process of microgrid fault recovery, the priority power supply of important loads should be considered first. Generally, loads are divided into three categories: primary loads, secondary loads, and tertiary loads. However, this classification method ignores the difference in load time scale. To this end, the present invention uses a priority recovery coefficient W that considers the time-varying load demand. i,t To measure the recovery priority of each node load in different time periods.

[0037] W i,t =D i,t ω

[0038] Where: D i,t It represents the time demand of the load on node i at time t. The demand of the first-level load is set to 10. The demand of other loads at the corresponding time is obtained according to the load demand ratio at different times. ω represents the load level coefficient, which is 100, 10, and 1 according to the load level classification.

[0039] (2) Build a fault recovery model:

[0040] Construct the objective function of the fault recovery model:

[0041] During the microgrid fault recovery process, a large number of power-off loads need to be restored. The main purpose of fault recovery is to minimize the number of switch operations and network losses while restoring more power-off loads. When the load is fixed, more loads restored means that the power-off load should be smaller. Therefore, the objective function F is set as:

[0042] minF=(F 1 +F 2 +F 3 )△t

[0043]

[0044] Where: △t represents the unit time length. The present invention mainly considers the power loss within 1 hour after the microgrid fails, that is, △t = 1 hour; F represents the fault recovery cost; F 1 、F 2 、F 3 They represent the system power failure load loss cost, system network loss cost and system switch operation equivalent loss cost respectively; c 1 、c 2 、c 3 They represent the load loss coefficient, system network loss coefficient and switch operation equivalent loss coefficient respectively; N, M, O and L represent the set of microgrid nodes, the set of branches, the set of section switches and the set of tie lines respectively; x i It is a 0-1 binary value, used to indicate the access status of node i, x i =0 means the node load is not removed, x i =1 means the node load is cut off; P i cut represents the load loss on node i; k s It is a 0-1 binary value, used to indicate the access status of branch s, k s =0 means the branch is not connected to the main network, k s =1 means the branch is connected to the main network; P s and Q s Respectively represent the active and reactive power on branch s; U s represents the voltage at the end node of branch s; R s represents the reactance of branch s; k p and k q It is a 0-1 binary value, which is used to indicate the opening and closing status of the section switch and the tie switch, 0 indicates that the switch is open, and 1 indicates that the switch is closed.

[0045] Constraints for building a fault recovery model:

[0046] a) Island power constraints:

[0047]

[0048] Where: D represents the node set in the island; P i,t P represents the active power of node i in the island during period t; loss,t Represents the island network loss during period t.

[0049] b) Branch power constraints:

[0050] P s,t ≤P s max

[0051] Where: P s,t Represents the active power value of branch s; P s max Indicates the maximum active power allowed to flow through branch s.

[0052] c) Node voltage constraints:

[0053]

[0054] Where: U i,t represents the voltage value of node i at time t; Indicates the maximum and minimum voltages allowed on node i.

[0055] d) Power flow constraints:

[0056]

[0057] Where: P i,t , Q i,t They represent the active power and reactive power of the load at node i at time t respectively; U i,t , U j,t Respectively represent the voltage amplitude of node i and node j at time t; θ ij represents the voltage phase difference between nodes i and j; G ij , B ij Represent the conductance and susceptance between nodes i and j respectively.

[0058] e) Radial operation constraints:

[0059] g 2,t ∈G

[0060] Where: G represents the set of radial topological structures; g 2,t Represents the topology of the microgrid.

[0061] f) Load constraints:

[0062] The loads in a microgrid can be divided into controllable loads and uncontrollable loads. When a microgrid encounters a fault, the controllable loads can be partially removed to ensure its power balance and stable power supply to important loads. There are the following constraints on the amount of load removal during a fault:

[0063]

[0064] Where: represents the load shedding amount at node i during period t; Represents the maximum load shedding factor at node i.

[0065] In step 2, DARL is a process in which two agents learn actions in order to obtain their respective maximum rewards and achieve specific goals during the process of perceiving and interacting with the environment. DARL is generally modeled as a Markov game model based on the actual problem to be solved, which can be (S, O i ,A i ,P i ,R i ,γ i ) represents. Where: S represents all environmental state spaces; O i A represents the local state space that agent i can perceive; i represents the action space of agent i; P i :S×A i ×S→[0,1] represents the state transition probability of agent i; R i :S×A i →R i Represents the reward obtained after the agent takes action; γ i ∈[0,1) represents the discount factor, which is used to adjust the influence of short-term and long-term rewards on the current decision-making of different agents.

[0066] The microgrid recovery process includes two problems: network topology reconstruction and node load shedding. Among them, network topology reconstruction can be represented by switch states, which is a discrete action solution problem; while node load shedding is a continuous action solution problem. However, the action space of the agent to be trained in general DARL can only be one of discrete or continuous. Therefore, in order to cope with the situation where both continuous and discrete actions exist, the microgrid fault recovery problem is modeled as a dual-agent learning task, using two agents to be responsible for the control of discrete actions and continuous actions respectively, and the two agents have their own experience buffer D 1 and D 2 , updated through iterative learning, and the output actions are jointly applied to the microgrid fault recovery scenario.

[0067] The Markov decision process considering the mapping of microgrid fault restoration problem is formulated as follows:

[0068] (1) Discrete Agents:

[0069] 1) Environment state space: the environment state O perceived by the discrete agent 1,t It includes: micro DG output at time t, load power demand, line fault status, time-varying comprehensive weight coefficient of load, power difference within the island, branch power, node voltage and topology structure, which can be expressed as:

[0070] O 1,t = {P fc,t ,P gc,t ,L i,t ,Z t,W i,t ,△P D,t ,P s,t ,U i,t ,g 2,t}

[0071] Where: Z t represents the fault branch at time t; △P D,t It represents the power difference in island D at time t.

[0072] 2) Action space: Discrete agents are mainly used to control the opening and closing of microgrid section switches and tie switches. Therefore, the action space A of discrete agents is 1,t for:

[0073] A 1,t ={k p,t ,k q,t}

[0074] Where: k p,t represents the opening of the pth section switch at time t; k q,t Indicates the opening of the qth tie switch at time t.

[0075] 3) Reward function: Reinforcement learning is adaptive learning through the trial and error of the agent. Every time the agent takes an action, it will receive a corresponding reward to guide the update of the model. In order to solve the problem of minimizing the fault recovery cost of the microgrid, the objective function needs to be reflected in the reward function. At the same time, in order to ensure the consistency of discrete and continuous agent training, the reward of the discrete agent should also include the reward obtained by the continuous agent. Therefore, the reward function R obtained by the discrete agent at time t is 1,t Set to:

[0076] R 1,t =R 2,t -F 2,t -F 3,t

[0077] Where: R 2,t represents the reward obtained by the continuous action agent at time t; F 2,t represents the network loss cost of the discrete agent at time t; F 3,t represents the equivalent loss cost of the switching operation of the discrete agent at time t.

[0078] (2) Continuous Action Agents:

[0079] ①, Environmental state space: the environmental state O that the continuous agent can perceive 2,t It is the same as that of discrete agents and will not be repeated here.

[0080] ② Action space: The continuous action agent is mainly used to control the removal of node loads, so the action space A of the continuous action agent is 2,t for:

[0081]

[0082] ③Reward function: Reward function R of continuous action agent 2,t Including negative continuous agent action costs and action constraint violation penalties. Therefore, the reward function is set as:

[0083] R 2,t =-F 1,t -Φ

[0084] Where: F 1,t represents the power loss load loss cost of the continuous action agent at time t; Φ represents the constraint violation penalty item. When any constraint is not satisfied, the penalty value is -1000, otherwise it is 0.

[0085] In step 3, unlike other DARLs that aim to maximize the cumulative reward of agent actions, the DASAC used in the present invention aims to maximize both the cumulative reward and the policy entropy, thereby improving the exploration performance and robustness of the agent training process, and thus preventing the agent strategy from converging to the local optimal solution too early. The objective functions of training discrete agents and continuous action agents by SAC-D (Soft Actor-Critic for Discrete action spaces) and SAC (Soft Actor-Critic) are:

[0086]

[0087] y∈{1,2}

[0088] Where: represents the optimal strategy of agent y; In the strategy π y Under all states s y,t and action a y,t Distribution The expected value of y,t represents the state space of agent y at time t; a y,t represents the action space of agent y at time t; r y,t (s y,t ,a y,t ) represents the agent y at time t and s y,t In state a, take action a y,t Instant rewards received.

[0089] represents mathematical expectation; y represents the agent number. When y is 1, it represents a discrete action agent, and when y is 2, it represents a continuous action agent; π y represents the strategy of agent y; Represents strategy π y Lower track (s y ,a y ) distribution; H y (·) represents the policy entropy of agent y; ν y Represents strategy π y The entropy coefficient is used to balance the influence of reward size and entropy. It is worth noting that when ν y →0, SAC degenerates into the traditional method of maximizing expected return.

[0090] In traditional reinforcement learning, the method of maximizing expected return usually refers to maximizing cumulative rewards by optimizing the policy without considering the policy entropy. The goal of this method is to find an optimal policy so that the agent can take the best action in a given state to maximize the cumulative reward. For a policy π, in a given state s t In this context, the goal of traditional reinforcement learning is to maximize the expected cumulative reward, i.e., the return, starting from the current state.

[0091]

[0092] Where: π * represents the optimal strategy; represents the expected value under strategy π; represents the cumulative reward (return); γ t is the discount factor at time t.

[0093] In the SAC of the present invention, the policy entropy term ν y H y (π y (i|s y,t )) introduces an incentive that allows the strategy to maintain a certain degree of randomness when selecting actions, which helps to explore the unknown state space. y →0, the influence of the entropy term gradually decreases, and the SAC algorithm degenerates into a method that only focuses on maximizing the reward, that is, the traditional reinforcement learning method.

[0094] Discrete agents and continuous action agents are trained under the framework of SAC-D and SAC respectively. Both contain their own experience pools. The training process includes:

[0095] First, define the neural network structure and initialize the policy network and Q network parameters;

[0096] In the SAC algorithm, the strategy is represented as a neural network, which receives the current state as input and outputs the probability distribution of the action. The Q-value function is also represented as a neural network, which receives the current state and action as input and outputs a Q-value. In order to reduce the risk of overestimation, the SAC algorithm uses two Q-value functions. Therefore, the SAC algorithm has 5 network structures, namely 1 Actor network (strategy network), 2 V Critic networks (state value estimation V and Target V networks), 2 Q Critic networks (action-state value estimation Q 0 and Q 1 network).

[0097] The SAC-D algorithm is consistent with the SAC algorithm in terms of network structure, except that the actions of the strategy output are discrete and continuous respectively. Therefore, taking the SAC algorithm as an example, its network structure is as follows Figure 4 shown.

[0098] Next, the policy network executes the current policy, records the current state, action, reward, next state and other experiences and stores them in the experience pool;

[0099] Then, update the value network and policy network and continue to select actions based on the current state and the learned policy;

[0100] Finally, repeat the above steps until convergence.

[0101] The present invention provides a microgrid fault recovery method based on DASAC, and the technical effects are as follows:

[0102] 1) The present invention proposes a microgrid fault recovery method based on DASAC. In order to comprehensively consider the uncertainty of source-load and realize the optimal recovery of microgrid faults, this method takes into account the time-varying DG output and load demand, and establishes a microgrid fault recovery model. Aiming at the problem that a high proportion of distributed power sources in microgrids have strong randomness after being affected by natural factors, the fault recovery problem of microgrids is transformed into a constrained nonlinear programming problem, and a DARL model is constructed with the goal of minimizing the cost of microgrid fault recovery. Discrete and continuous dual agents are assigned to different action types in the action space for control. The present invention innovatively proposes a microgrid fault recovery strategy based on dual-agent flexible action-evaluation method (DASAC). By processing discrete and continuous actions separately, the fault recovery efficiency is improved, the training process is simplified, and the robustness and adaptability of the system are enhanced.

[0103] 2) In view of the problem that traditional fault recovery methods have slow calculation speed and cannot realize real-time control of fault recovery, the present invention proposes a DASAC method to efficiently and adaptively train and learn the DARL model, thereby improving the fault recovery efficiency to meet the needs of microgrid operation.

[0104] 3) The microgrid fault recovery model proposed in the present invention can effectively reduce the fault recovery loss, the proposed DASAC algorithm can significantly reduce the decision-making time, reliably learn the optimal fault recovery strategy, and compared with other methods, the recovery performance of the fault recovery method proposed in the present invention is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:

[0106] Figure 1 It is a schematic diagram of the learning process of DARL proposed in the present invention.

[0107] Figure 2 It is a comparison chart of the convergence performance of the DASAC method proposed in the present invention and other DRL methods.

[0108] FIG3( a ) is a fault recovery topology diagram of the method proposed in the present invention (period one);

[0109] FIG3( b ) is a fault recovery topology diagram of the method proposed in the present invention (period two).

[0110] Figure 4 This is the network structure diagram of the SAC algorithm. DETAILED DESCRIPTION

[0111] The present invention will be further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0112] Figure 1 This is the learning process of DARL proposed in this invention. As can be seen from the figure, the network topology reconstruction is represented by the switch state, which belongs to the discrete action solution problem, and the discrete action agent is used for learning. The node load shedding problem belongs to the continuous action solution problem, and the continuous agent is used to control the continuous action. The two agents have their own experience buffer D 1 and D 2 , updated through iterative learning, and the output actions are jointly applied to the microgrid fault recovery scenario.

[0113] Figure 2It is a convergence performance comparison chart of the DRL method proposed in the present invention and other DRL methods. It can be seen that the reward value of DQN still cannot converge stably when the set maximum number of iterations is reached. DDPG reaches convergence at about 8200 rounds of training, which is about 3200 rounds and 6000 rounds slower than SAC and the proposed DASAC. This is because SAC and the proposed DASAC use cumulative rewards and policy entropy as the learning objectives of the agent at the same time, which improves the exploration efficiency of the agent in an unknown environment. Further comparison of SAC and the proposed DASAC shows that the proposed method converges about 56% faster than SAC, and the converged reward value is higher and more stable. The main reason is that the proposed DASAC method constructs a dual-agent learning model and assigns discrete and continuous agents to be responsible for the learning of discrete and continuous actions, thereby improving the learning convergence ability of the method. The above comparison shows that compared with other deep reinforcement learning methods, the proposed method has obvious advantages in training convergence speed and stability.

[0114] FIG3(a) and FIG3(b) are fault recovery topology diagrams of the method proposed in the present invention. Assuming that due to natural disasters, permanent circuit breakers 8-9, 20-21, 49-50 and 3-59 of the microgrid system occur in the two periods of 11:00-12:00 and 16:00-17:00. Analysis of FIG3(a) and FIG3(b) shows that due to the large difference in DG output and load characteristics in different periods, the corresponding fault recovery methods are also very different. At 11:00-12:00, the output of the photovoltaic storage system is at a high level, while the output of the wind storage system has not yet reached the maximum. After executing the fault recovery strategy proposed in the present invention, islands are formed around DG2, DG5 and DG6, respectively, and the contact switch S4 is closed, so that DG5 restores power supply to nodes 22-27. Outside the island, only nodes 20, 21, 59, 60 and 69 with lower priority recovery coefficients fail to restore power supply. From 16:00 to 17:00, the output of the photovoltaic storage system dropped to a lower level, while the output of the wind storage system reached the maximum. Analysis of the fault recovery results during the period of 16:00-17:00 shows that due to the large output of wind storage systems DG2 and DG6, a joint island is formed through the interconnection switch S3, which further improves the stability of the island operation. Due to the low output of the photovoltaic storage system DG5, nodes 21 and 22 were not included in the island, but by unloading some controllable loads with low priority recovery coefficients in the island, the power demand of commercial loads with larger priority recovery coefficients such as nodes 23, 25 and 27 during the fault period was guaranteed. Since there was no fault in the upstream branches of DG1, DG3 and DG4, they can still maintain connection with the main grid after the fault, and close the interconnection switch S5 to form the main grid recovery area. Therefore, when the fault location is determined, the same main grid recovery area is formed in the two fault periods. In the main grid recovery area, DG is given priority to supply power, and the main grid power supply is used as a supplement to ensure that all loads in the area do not lose power.

[0115] Table 1 Specific fault recovery results of the method proposed in the present invention at different fault periods

[0116]

[0117] Table 1 shows the specific fault recovery results of the method proposed in the present invention in different fault periods. Analysis of Table 1 shows that by executing the fault recovery strategy, the microgrid segmentation and tie switches were operated 7 times and 5 times in the two fault periods of 11:00-12:00 and 16:00-17:00, respectively, and the network loss of the microgrid during the fault period was reduced by 20.8% and 17.1%, respectively; and after the fault recovery, the voltage amplitude of the lowest node of the system increased by 0.022pu and 0.018pu, both of which were restored to the set amplitude range constraints, and the power supply quality in the microgrid was greatly improved.

[0118] Table 2 Comparison of the recovery effect of the method proposed in this invention and other algorithms

[0119]

[0120] Table 2 is a comparison of the recovery effects of the method proposed in the present invention and other algorithms. The specific settings of the comparison method are as follows:

[0121] Method 1: Fault recovery method based on the Grey Wolf optimization algorithm;

[0122] Method 2: Fault recovery method based on particle swarm hybrid grey wolf algorithm;

[0123] Method 3: SAC-based fault recovery method;

[0124] Method 4: The fault recovery method based on DASAC proposed in the present invention;

[0125] From the analysis of Table 2, we can see that the decision time of methods 1 and 2 is relatively long, reaching 79s and 56s. This is because methods 1 and 2 are heuristic algorithms, and they need to go through a long optimization process before obtaining the optimal solution. And because the algorithm iteration falls into the local optimum, the fault recovery strategy obtained by methods 1 and 2 is not the global optimal, and the average fault recovery cost is higher than that of methods 3 and 4. As methods based on deep reinforcement learning, methods 3 and 4 can generate the optimal recovery strategy according to the specific fault environment of the microgrid in a very short time after being fully trained. In-depth comparison of methods 3 and 4 shows that the average decision time and average fault recovery cost of the proposed method 4 are reduced by 0.16s and 20 yuan respectively compared with scheme 3, that is, method 4 has learned a better fault recovery strategy. This is mainly because method 4 constructs a dual-agent learning model for the Markov decision problem of microgrid fault recovery and allocates discrete and continuous agents to be responsible for the learning of discrete and continuous actions, thereby improving the learning convergence ability of the method. The SAC in method 3 without any improvement belongs to a single-agent deep reinforcement learning method. When dealing with the comprehensive problem of multi-fault recovery in microgrids, it is necessary to discretize the action space, which leads to certain limitations on the learning ability of the agent.

[0126] To sum up, the method proposed in this invention has better performance than other traditional heuristic algorithms and SAC methods. It can quickly obtain the microgrid fault recovery strategy while ensuring the quality of the recovery strategy, which is more in line with the needs of microgrid emergency fault recovery.

Claims

1. A microgrid fault recovery method based on DASAC, characterized in that The following steps are involved: Step 1: Establish a microgrid fault recovery model that takes into account the time-varying DG output and load demand; Step 2: The fault recovery problem of the microgrid is transformed into a nonlinear programming problem with constraints. Taking the minimization of the microgrid fault recovery cost as the goal, a dual-agent reinforcement learning model is constructed to assign discrete and continuous dual-agents to control different action types in the action space. Step 3: Propose a DASAC method to adaptively train the dual-agent reinforcement learning model to improve the fault recovery efficiency to meet the needs of microgrid operation.

2. The microgrid fault recovery method based on DASAC according to claim 1, characterized in that: In step 1, in order to comprehensively consider the uncertainty of source-load and minimize the cost of microgrid fault recovery process, taking into account the time-varying DG output and load demand, a microgrid fault recovery model is established, which specifically includes: (1) Constructing a time-varying source-load model: 1) Wind turbine model: The output of a wind turbine (WT) is affected by wind speed. The relationship between WT output power and wind speed is expressed as follows: Where: P WT Indicates WT output power; P r represents WT rated capacity; v represents wind speed; v r Indicates rated wind speed; v ci 、v co They represent the cut-in wind speed and the cut-out wind speed respectively; Due to the large fluctuation of wind speed at different times, the output of WT is also uncertain. The Weibull distribution function is used to fit the WT output. The Weibull probability density distribution of wind speed is expressed as: Where: f1(v) represents the probability density function of wind speed; k1 and c represent the shape parameters of the probability density function of wind speed; It represents the attenuation characteristics of wind speed v under the action of shape parameters c and k1; 2) Photovoltaic model: Photovoltaic (PV) output is mainly affected by the intensity of solar radiation. The relationship between PV output power and solar radiation intensity is as follows: P PV =ηAg1 Where: P PV represents PV output power; η represents the photoelectric conversion efficiency of the photovoltaic panel; A represents the area of ​​the photovoltaic panel; g1 represents the solar radiation intensity; PV output is affected by changes in solar radiation intensity and has a large degree of randomness. The Beta distribution function is used to approximate the solar radiation situation: Where: f2(g1) represents the probability density function of solar radiation; g 1,max represents the maximum radiation intensity of sunlight; Γ(i) represents the gamma distribution; α and β represent the shape parameters of the beta distribution function; 3) Energy storage model: The charging and discharging model of the energy storage device is as follows: Where: P c,t and P d,t Respectively represent the charging and discharging power of energy storage during time period t; P c max and P d max Respectively represent the maximum charging and discharging power of energy storage; SOC t Indicates the remaining storage capacity during period t; SOC max and SOC min Respectively represent the maximum and minimum power allowed by energy storage; P fc,t , P gc,t They represent the output of the wind storage system and the solar storage system at time t respectively; 4) Load model: a) Load time-varying model: According to different uses, the microgrid load can be divided into three categories, namely commercial, residential and industrial loads. For the load, the definite integral calculation is performed in adjacent time periods to obtain the power demand of different types of loads in the period; Where: L i,t represents the load demand on node i from t to t+1; f i (x) represents the load curve function at node i; b) Load demand model: During the microgrid fault recovery process, the priority restoration coefficient W is used to consider the time-varying load demand. i,t To measure the recovery priority of each node load in different time periods; W i,t =D i,t ω Where: D i,t It represents the time demand of the load on node i at time t. The demand of the first-level load is set to 10. The demand of other loads at the corresponding time is obtained according to the load demand ratio at different times. ω represents the load level coefficient, which is 100, 10, and 1 according to the load level classification. (2) Build a fault recovery model: Construct the objective function of the fault recovery model: Set the objective function F as: minF=(F1+F2+F3)△t Where: △t represents the unit time length; F represents the fault recovery cost; F1, F2, and F3 represent the system power loss load loss cost, system network loss cost, and system switch operation equivalent loss cost respectively; c1, c2, and c3 represent the load loss coefficient, system network loss loss coefficient, and switch operation equivalent loss coefficient respectively; N, M, O, and L represent the set of microgrid nodes, the set of branches, the set of section switches, and the set of tie lines respectively; x i It is a 0-1 binary value, used to indicate the access status of node i, x i =0 means the node load is not removed, x i =1 means the node load is cut off; P i cut represents the load loss on node i; k s It is a 0-1 binary value, used to indicate the access status of branch s, k s =0 means the branch is not connected to the main network, k s =1 means the branch is connected to the main network; P s and Q s Respectively represent the active and reactive power on branch s; U s represents the voltage at the end node of branch s; R s represents the reactance of branch s; k p and k q It is a 0-1 binary value, which is used to indicate the opening and closing status of the section switch and the tie switch, 0 indicates that the switch is open, and 1 indicates that the switch is closed.

3. The microgrid fault recovery method based on DASAC according to claim 2 is characterized in that: Constructing the fault recovery model constraints: a) Island power constraints: Where: D represents the node set in the island; P i,t represents the active power of node i in the island during period t; P loss,t represents the island network loss during period t; b) Branch power constraints: P s,t ≤P s max Where: P s,t Represents the active power value of branch s; P s max Indicates the maximum active power allowed to flow through branch s; c) Node voltage constraints: Where: U i,t represents the voltage value of node i at time t; Indicates the maximum and minimum voltage allowed on node i; d) Power flow constraints: Where: P i,t , Q i,t They represent the active power and reactive power of the load at node i at time t respectively; U i,t , U j,t Respectively represent the voltage amplitude of node i and node j at time t; θ ij represents the voltage phase difference between nodes i and j; G ij , B ij denote the conductance and susceptance between nodes i and j respectively; e) Radial operation constraints: g 2,t ∈G Where: G represents the set of radial topological structures; g 2,t represents the topology of the microgrid; f) Load constraints: The loads in a microgrid can be divided into controllable loads and uncontrollable loads. When a microgrid encounters a fault, the controllable loads can be partially removed to ensure its power balance and stable power supply to important loads. The load removal amount during a fault is subject to the following constraints: Where: represents the load shedding amount at node i during period t; Represents the maximum load shedding factor at node i.

4. The microgrid fault recovery method based on DASAC according to claim 1 is characterized in that: In step 2, DARL is a process in which two agents learn actions in order to obtain their respective maximum rewards and achieve specific goals during the process of perceiving and interacting with the environment. i ,A i ,P i ,R i ,γ i ) represents; where: S represents all environmental state spaces; O i A represents the local state space that agent i can perceive; i represents the action space of agent i; P i :S×A i ×S→[0,1] represents the state transition probability of agent i; R i :S×A i →R i Represents the reward obtained after the agent takes action; γ i ∈[0,1) represents the discount factor, which is used to adjust the influence of short-term and long-term rewards on the current decision-making of different agents.

5. The microgrid fault recovery method based on DASAC according to claim 4 is characterized in that: The microgrid recovery process includes two problems: network topology reconstruction and node load shedding. Among them, network topology reconstruction is represented by switch states and belongs to a discrete action solution problem; while node load shedding belongs to a continuous action solution problem. In order to cope with the situation where both continuous and discrete actions exist, the microgrid fault recovery problem is modeled as a dual-agent learning task. Two agents are responsible for the control of discrete actions and continuous actions respectively. The two agents have their own experience buffers D1 and D2, which are updated through iterative learning, and the output actions are jointly applied to the microgrid fault recovery scenario. The Markov decision process considering the mapping of the microgrid fault recovery problem is formulated as follows: (1) Discrete Agents: 1) Environment state space: the environment state O perceived by the discrete agent 1,t It includes: micro DG output at time t, load power demand, line fault status, time-varying comprehensive weight coefficient of load, power difference within the island, branch power, node voltage and topology structure, expressed as: O 1,t ={P fc,t ,P gc,t ,L i,t ,Z t ,W i,t ,△P D,t ,P s,t ,U i,t ,g 2,t } Where: Z t represents the fault branch at time t; △P D,t represents the power difference within island D at time t; 2) Action space: The discrete agent is used to control the opening and closing of the microgrid section switches and tie switches. Therefore, the action space A of the discrete agent is 1,t for: A 1,t ={k p,t ,k q,t } Where: k p,t represents the opening of the pth section switch at time t; k q,t represents the opening of the qth tie switch at time t; 3) Reward function: The reward of the discrete agent also includes the reward obtained by the continuous agent; therefore, the reward function R obtained by the discrete agent at time t 1,t Set to: R 1,t =R 2,t -F 2,t -F 3,t Where: R 2,t represents the reward obtained by the continuous action agent at time t; F 2,t represents the network loss cost of the discrete agent at time t; F 3,t represents the equivalent loss cost of the switching operation of the discrete agent at time t; (2) Continuous Action Agents: ①, Environmental state space: the environmental state O that the continuous agent can perceive 2,t Same as for discrete agents; ② Action space: The continuous action agent is used to control the removal of node loads, so the action space A of the continuous action agent is 2,t for: ③Reward function: Reward function R of continuous action agent 2,t It includes negative continuous agent action costs and action constraint violation penalties; therefore, the reward function is set as: R 2,t =-F 1,t -F Where: F 1,t represents the power loss load loss cost of the continuous action agent at time t; Φ represents the constraint violation penalty item.

6. The microgrid fault recovery method based on DASAC according to claim 1, characterized in that: In step 3, the DASAC used aims to maximize the cumulative reward and policy entropy at the same time. The objective functions for training discrete agents and continuous action agents are: Where: represents the optimal strategy of agent y; In the strategy π y Under all states s y,t and action a y,t Distribution The expected value of y,t represents the state space of agent y at time t; a y,t represents the action space of agent y at time t; r y,t (s y,t ,a y,t ) represents the agent y at time t and s y,t In state a, take action a y,t The immediate rewards received; E represents mathematical expectation; y represents the agent number. When y is 1, it represents a discrete action agent, and when y is 2, it represents a continuous action agent; π y represents the strategy of agent y; Represents strategy π y Lower track (s y ,a y ) distribution; H y (·) represents the policy entropy of agent y; ν y Represents strategy π y The entropy coefficient is used to balance the reward size and the influence of entropy.

7. The microgrid fault recovery method based on DASAC according to claim 6 is characterized in that: Discrete agents and continuous action agents are trained under the framework of SAC-D and SAC respectively. Both contain their own experience pools. The training process includes: First, define the neural network structure and initialize the policy network and Q network parameters; Next, the policy network executes the current policy, records the current state, action, reward, next state and other experiences and stores them in the experience pool; Then, update the value network and policy network and continue to select actions based on the current state and the learned policy; Finally, repeat the above steps until convergence.

Images