Island micro-grid distributed low frequency load shedding method based on improved multi-agent collaborative algorithm

Abstract

The application discloses an island micro-grid distributed low-frequency load shedding method based on an improved multi-agent collaborative algorithm, and comprises the following steps: constructing a load shedding contribution factor considering load frequency characteristics and load importance, which is used for quantifying load shedding loss; establishing a low-frequency load shedding optimization model for minimizing load shedding loss and system frequency deviation; converting the distributed low-frequency load shedding problem into a Markov game process, regarding each load node as an independent agent, and defining the state space, action space and reward function of each agent according to the established low-frequency load shedding optimization model; and training each agent by using a multi-agent deep deterministic policy gradient (MADDPG) algorithm with a priority experience replay mechanism, so that the agent learns a low-frequency load shedding strategy depending on local real-time information. The method is based on the design of the island micro-grid distributed low-frequency load shedding strategy, and provides an effective measure for reducing load shedding loss and improving system frequency stability.

Description

Technical Field

[0001] This invention relates to the field of low-frequency load shedding control in islanded microgrids, and specifically to a distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm. Background Technology

[0002] Microgrids, as an important component of new power systems, integrate distributed power sources, energy storage, and loads to achieve local consumption of new energy sources and possess the capability for grid-connected operation and islanded operation under fault conditions. However, due to the limited capacity and low equivalent inertia of islanded microgrid systems, frequency shifts are more severe when power deficits occur. Simultaneously, the strong intermittency and volatility of distributed power output make accurate calculation of load shedding difficult, easily leading to over-shearing or under-shearing phenomena. Existing centralized low-frequency load shedding schemes have very high requirements for the real-time performance, synchronization, and reliability of communication networks, which still presents certain difficulties in applications of islanded microgrids with rapidly changing source-load disturbances; moreover, centralized architectures are often accompanied by high construction and maintenance costs. In contrast, while existing distributed low-frequency load shedding schemes reduce dependence on communication conditions, they generally lack coordination mechanisms between load nodes and do not fully consider the differences in contribution of different loads during the load shedding process, making it difficult to achieve global coordination and optimization of load shedding effects. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm. This method first considers the frequency characteristics and importance of the load, establishing a low-frequency load shedding optimization model that minimizes load shedding losses and system frequency deviation. Then, the distributed low-frequency load shedding problem is transformed into a Markov game process, and a priority experience replay mechanism is introduced to improve experience utilization efficiency. Under a framework of centralized training and distributed execution, each agent learns the optimal load shedding strategy based solely on local real-time load information through continuous interaction with the environment. This method, based on the distributed low-frequency load shedding strategy design for islanded microgrids, provides effective measures to reduce load shedding losses and improve system frequency stability.

[0004] The technical solution adopted in this invention is as follows:

[0005] A distributed low-frequency load shedding method for isolated microgrids based on an improved multi-agent cooperative algorithm includes the following steps: Step 1: Construct a load shedding contribution factor that takes into account load frequency characteristics and load importance, in order to quantify load shedding losses; Step 2: Establish a low-frequency load shedding optimization model that minimizes load shedding losses and system frequency deviation; Step 3: Transform the distributed low-frequency load shedding problem into a Markov game process, where each load node is regarded as an independent agent, and define the state space, action space and reward function of each agent based on the low-frequency load shedding optimization model established in Step 2. Step 4: Train each agent using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm with a priority experience replay mechanism, so that the agents learn low-frequency load reduction strategies that rely on local real-time information.

[0006] In step 1: 1) Analyze the load frequency characteristics and load importance: When the system power balance is disturbed and causes a frequency shift, the load will produce a corresponding power change according to its own frequency characteristics. This response is described by the load's frequency regulation effect coefficient. (1); In equation (1), This is the frequency modulation effect coefficient; This is the per-unit value of the active power of the load; This is the per-unit value of the system frequency; To match the system's rated frequency The proportionality coefficient of the load to the rated load, which is directly proportional to the power of the load.

[0007] For two different frequency regulation effect loads When the frequency changes from Down to When, its power changes ,Right now Therefore, when the system frequency decreases, Larger loads absorb more active power from the system; when the system faces low-frequency load shedding, priority is given to disconnecting power. Smaller load, retain Larger loads help reduce load shedding and shedding losses. Load importance is used to quantify the differences in economic losses caused by power outages; the higher the importance, the greater the loss per unit of power lost.

[0008] 2): Incorporating load frequency response characteristics into load reduction decisions. Importance of load Constructing load reduction contribution factors Specifically, loads with smaller frequency regulation effect coefficients are prioritized for shelving to improve the system's frequency recovery capability; at the same time, loads of lower importance are prioritized for shelving to reduce the economic losses caused by load shelving to the system operation.

[0009] Load reduction contribution factor Calculation: This invention employs an approximation-ideal-solution method to comprehensively evaluate various indicators of each load node. Essentially, this method normalizes each indicator, constructs a positive ideal solution composed of the optimal indicator values, and a negative ideal solution composed of the worst indicator values. The distance between each evaluation indicator and these two ideal solutions is calculated to measure whether it is closer to the optimal or worst solution, thus reflecting its overall quality. The specific steps are as follows: First, the evaluation indicators are normalized: (2); In equation (2), The index value is the normalized value; Indicates the first individual load indicators The attribute value; They represent the first The minimum and maximum values ​​of each indicator across the entire set of load nodes.

[0010] Introducing indicator weights Constructing a weighted decision matrix ,in: , Indicates the number of load nodes. This indicates the quantity of load indicators.

[0011] Then, the positive and negative ideal solutions are determined separately, and the Euclidean distances from each load node to the positive and negative ideal solutions are calculated. The positive ideal solution represents the optimal value of each index among all load nodes, while the negative ideal solution represents the worst value. (3); In equation (3), These are the positive ideal solution and the negative ideal solution, respectively. denoted as Euclidean distances for the positive and negative ideal solutions, respectively.

[0012] Finally, the load reduction contribution factor of the load node is defined as follows: (4); In equation (4), The value ranges from 0 to 1. The smaller the value, the lower the load contribution and the higher the priority of shedding in load reduction decisions.

[0013] In step 2, in an islanded microgrid, the optimization objective is to minimize load shedding losses and system frequency deviation; the objective function for low-frequency load shedding decision is described as follows: (5); In equation (5), Maximum time step per simulation round; The number of load nodes; For load nodes Contribution weight; For the first The load is unloaded at the time of load reduction. The amount of load shedding; The system's rated frequency; For the first The load is unloaded at the time of load reduction. The actual frequency; For the first The load is unloaded at the time of load reduction. The percentage of shear load; For the first The rated active power of each load.

[0014] 1) Power balance constraints: To ensure system stability, it is necessary to ensure that any system... The active power output is always balanced with the total load and load loss.

[0015] (6); In formula (6): The first The output power of a miniature gas turbine, wind turbine, and photovoltaic system; , The first The charging and discharging power of the energy storage device; These are the quantities of micro gas turbines, wind turbines, photovoltaic systems, and energy storage, respectively. For the first The rated active power of each load; For the first The load is unloaded at the time of load reduction. The amount of load shedding; Energy lost by the network.

[0016] 2) Output constraints of distributed power sources: (7); In equation (7): These are the upper and lower limits of the active power of the micro gas turbine, respectively. These are the upper limits of the output power of wind turbines and photovoltaics, respectively.

[0017] 3) Energy storage constraints: (8); In equation (8): , The first The charging and discharging power of the energy storage device; These represent the charging and discharging states of energy storage. These are the upper limits for energy storage charging and discharging power, respectively. , They are time points and time Energy storage capacity; These are the charging and discharging efficiencies of energy storage, respectively. , They are time points and time The energy storage state of charge; This represents the change in energy storage capacity. This refers to the rated capacity of the energy storage. These represent the minimum and maximum states of charge of the energy storage unit, respectively.

[0018] 4) Frequency recovery constraint: (9); In equation (9): This represents the minimum value of the system's transient frequency; This refers to the steady-state frequency deviation of the system. This represents the maximum steady-state frequency deviation of the system.

[0019] 5) Voltage safety constraints: To ensure that the voltage does not remain excessively low or high for an extended period while the frequency is restored, a safe voltage range is set: (10); In formula (10): For nodes The actual voltage at that location They are nodes The minimum and maximum voltage values ​​allowed for operation under load.

[0020] In step 3, the Markov game process includes a set of agents. State space Action space and reward function ; At any load reduction moment The interaction process between each intelligent agent and the microgrid environment is as follows: First of all, the An agent observes the local load status. and through strategy Generate corresponding control actions ; Then, the actions of all agents are combined to form a joint action. It operates in the microgrid environment; Subsequently, the system based on joint actions Provide immediate rewards to each agent and transition to the next state. .

[0021] 1) State space: The local state of each load node consists of the load node frequency deviation, frequency change rate, and bus voltage. The global state, combining the local states of all agents, the output power of each distributed power source, the charging and discharging power of the energy storage unit, and the state of charge of the energy storage itself, describes the overall operating environment of the microgrid. For each given agent... , That is, the number of load nodes, at time... The local state and the global state of the system are represented as follows: (11); In equation (11): , These represent the local state and the global state of the agent, respectively. , , They are the load nodes. At any moment Frequency deviation, rate of frequency change, and voltage; , , They were respectively in the second Taiwan's micro gas turbines, wind turbines, and photovoltaics are constantly ; output power; , The first Taiwan's energy storage at any time The charging and discharging power; For a moment The state of charge of energy storage.

[0022] 2) Motion space: Each agent, based on its current state and according to the policy, Output action, which is defined as the load node At any moment The load shedding ratio, i.e., the operating space. , For the first The load is unloaded at the time of load reduction. The percentage of load shedding.

[0023] 3) Reward and punishment functions: In the load reduction strategy, all agents cooperate fully and share the same interests, achieving a common goal through collaboration to obtain the same reward; therefore, the reward function is set consistently. Combining this with a low-frequency load reduction optimization model, the first... Reward function for each load node This includes both target-based rewards and constraint-based rewards; The target reward includes a load shedding loss reward. and frequency offset reward ; The constraint reward is constructed from equations (6) to (10). , respectively represented as: (12); In equation (12): These are the weighting coefficients for each reward; The penalty coefficient for restoring the system when the constraints are not met; For load nodes Contribution weight; For the first The load is unloaded at the time of load reduction. The amount of load shedding; The system's rated frequency; For the first The load is unloaded at the time of load reduction. The actual frequency.

[0024] To link immediate rewards with long-term returns, a value function is constructed for each agent. Characterizing intelligent agents Starting from the initial state, follow the strategy. The expected cumulative return that can be obtained is shown in equation (13): (13); In equation (13): The desired value is the weighted average over multiple simulation rounds. This is the cumulative reward discount coefficient; The maximum time step for each simulation round; For the first Each load node at time... The reward function.

[0025] Step 4 includes: 4.1: Solving the Markov game model for distributed low-frequency load shedding: The Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG) configures an independent action network for each agent. and value network and its corresponding target network , ; in: This represents the local state space of the agent. It is the set of global state spaces for all intelligent agents; , , , These are the parameters for the action network, value network, and their corresponding target network, respectively. It is the set of action spaces for all intelligent agents.

[0026] Action networks are used for approximate deterministic policy functions In contrast, value networks, under centralized training conditions, evaluate the value of actions based on global information.

[0027] During the training phase, the policy function is trained using a deep neural network. and The value functions are jointly fitted, and the value function is maximized. Update network parameters to achieve continuous optimization of the strategy in a high-dimensional continuous action space; The value represents the combined loss caused by load shedding and frequency shift, and provides gradient guidance for policy updates.

[0028] This paper introduces a Priority Experience Replay (PER) mechanism based on the MADDPG algorithm. The importance of each experience is measured by its temporal aldifference (TD) error, and the sampling probability is dynamically adjusted to ensure that high-value experiences are utilized more extensively during training. The expression for calculating the TD error is as follows: (14); In equation (14): the larger the TD error, the greater the difference between the value network and the target network. The more significant the value deviation, the higher the corresponding empirical sampling probability.

[0029] A ranking-based priority mechanism is adopted, using the network training loss as an empirical importance indicator. The formula for calculating the empirical probability priority is as follows: (15); In equation (15): The first The probability and priority of each experience being sampled; Indicates based on experience The experience replay buffer is sorted from largest to smallest, with a smaller sequence number indicating higher priority; This is the sampling priority parameter. The larger the value, the more it relies on priority for sampling; For the first The relative weights of each empirical sample; This represents the sum of the weights of all samples in the experience replay pool; This represents the total number of samples stored in the experience replay pool.

[0030] Importance sampling weights are introduced to adjust the update magnitude of high-priority experiences: (16); In equation (16): For the first The importance sampling weight of each experience sample; No. Normalized sampling weights for empirical samples; It is the size of the sample storage capacity in the experience pool; Hyperparameters representing the influence of priority samples on the convergence result; This represents the maximum weight of all samples in the sampling batch.

[0031] Based on this, the agent's neural network is trained during its interaction with the environment. For samples taken from the priority experience replay pool... In the batch of samples, each agent The target action network is based on the next time step state. Generate predicted actions And the target value network calculates the corresponding target. value: (17); In equation (17): For the first The goal of an intelligent agent value; For the first The instant reward obtained by an intelligent agent; Discount factor; For the first A network of target values ​​for individual agents; , These are the set of the agent's state space and action space for the next moment, respectively. These are the parameters of the target value network.

[0032] Value network parameters The update is performed by minimizing the following loss function: (18); In formula (18): This is the loss function for the value network, used to update the value network; This indicates the batch size of samples taken from the experience replay pool; No. Normalized sampling weights for empirical samples; For the first A value network of intelligent agents; These are the parameters of the value network.

[0033] Action network parameters Use the following gradient ascent strategy for gradient updates: (19); In equation (19): Value function Action network parameters Find the partial derivatives; Value network Action Find the partial derivative; Action network Its parameters Find the partial derivative.

[0034] Parameters of the Action and Value Target Network and Use the following soft update method: (20); In equation (20): It is a constant much smaller than 1, used to reduce the variation of neural network parameters, making the model easier to converge.

[0035] 4.2: Training process for solving Markov game models: Training process of the distributed low-frequency load shedding strategy based on improved MADDPG in an islanded microgrid simulation environment: In each round of training, each agent constructs its current state using local measurement information and uses the action network to select the currently executable deload action. In the islanded microgrid simulation environment, the system completes the state transition based on the load reduction actions of all agents, and obtains the new node frequency, voltage and system power balance. Subsequently, the system operating status is evaluated based on the objective function constructed based on the load shedding loss and frequency offset, and the instantaneous reward value of each agent is calculated according to Equation (12); Each agent constructs an experience sample from its current state, action, reward value, and next state, and stores it in the experience replay pool. Each agent assigns priority to the experience samples according to the TD error and calculates the experience sampling probability distribution according to equations (12) to (15). Each agent samples experience in batches from the experience replay pool and updates the action network, value network and its corresponding target network according to equations (16) to (20).

[0036] When time Reaching the maximum time step When the current round ends and the next round begins; during training rounds Reach the preset maximum number of rounds In this way, the intelligent agents can form a stable cooperative strategy and achieve optimal distributed load reduction control under conditions of no communication.

[0037] This invention discloses a distributed low-frequency load shedding method for isolated microgrids based on an improved multi-agent cooperative algorithm, with the following technical advantages: 1) This invention proposes a distributed low-frequency load shedding method for isolated microgrids. By introducing a load shedding contribution factor and combining it with multi-agent reinforcement learning, it achieves distributed collaborative decision-making. While reducing communication dependence, it provides effective measures to reduce load shedding losses and improve system frequency stability.

[0038] 2) Step 1 of the present invention constructs a load reduction contribution factor that takes into account the load frequency characteristics and load importance, and quantitatively characterizes the load shedding loss, providing an evaluation basis for differentiated decision-making on low-frequency load reduction.

[0039] 3) Step 2 of the present invention establishes a low-frequency load reduction optimization model that simultaneously considers load shedding loss and minimizes system frequency deviation based on the above-mentioned load reduction contribution factors, providing a clear objective function and constraints for the optimization solution of the load reduction strategy.

[0040] 4) Step 3 of the present invention models the distributed low-frequency load shedding process as a Markov game, and constructs the state, action and reward function of each agent based on the optimization model, providing a modeling framework for collaborative decision-making and strategy learning among multiple load nodes.

[0041] 5) Step 4 of the present invention uses MADDPG with a priority experience replay mechanism to train each agent, enabling it to achieve distributed low-frequency load shedding control under the condition of relying only on local information, thereby reducing the dependence on communication and centralized control and improving the frequency stability and operational reliability of the islanded microgrid under complex disturbances. Attached Figure Description

[0042] The present invention will be further described below with reference to the accompanying drawings and examples; Figure 1 This is a schematic diagram of a distributed low-frequency load reduction strategy framework for multi-agent collaboration.

[0043] Figure 2 This is the active-frequency response curve of the load.

[0044] Figure 3 A schematic diagram of the basic structure of the improved MADDPG algorithm.

[0045] Figure 4 This is a flowchart of the agent training process.

[0046] Figure 5 Diagram of the improved IEEE-13 node microgrid system architecture.

[0047] Figure 6 The cumulative reward change curves for three different algorithms are shown.

[0048] Figure 7 The frequency dynamic curves are for different low-frequency load reduction strategies in operating condition 1.

[0049] Figure 8 The frequency dynamic curves are for different low-frequency load reduction strategies in operating condition 2.

[0050] Figure 9 This is a comparison chart of start-up time and load reduction under fault condition 1.

[0051] Figure 10 This is a comparison chart of start-up time and load reduction under fault condition 1. Detailed Implementation

[0052] A distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm is proposed. First, considering the frequency characteristics and importance of the load, a low-frequency load shedding optimization model is established, taking into account load shedding losses and minimizing system frequency deviation. Then, the distributed low-frequency load shedding problem is transformed into a Markov game process, and a priority experience replay mechanism is introduced to improve experience utilization efficiency. Each agent, under a centralized training and distributed execution framework, learns the optimal load shedding strategy based solely on local real-time load information through continuous interaction with the environment. Finally, the effectiveness and feasibility of the proposed strategy are verified using an improved IEEE-13-node microgrid system. Figure 1 The diagram illustrates a distributed low-frequency load shedding strategy framework based on multi-agent collaboration, comprising the following steps: Step 1: Construct a load shedding contribution factor that takes into account load frequency characteristics and load importance, in order to quantify load shedding losses; Step 2: Establish a low-frequency load shedding optimization model that minimizes load shedding losses and system frequency deviation; Step 3: Transform the distributed low-frequency load shedding problem into a Markov game process, where each load node is regarded as an independent agent. Based on Step 2, establish a low-frequency load shedding optimization model to define the state space, action space, and reward function of each agent, which are used to describe the interaction process between the agent and the simulation environment. Step 4: Train each agent using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, which introduces a priority experience replay mechanism, so that the agents learn low-frequency load reduction strategies that rely only on local real-time information.

[0053] In step 1, 1) Analyze the load frequency characteristics and load importance: When the system power balance is disturbed and causes a frequency shift, the load will produce a corresponding power change according to its own frequency characteristics. This response is usually described by the load's frequency regulation effect coefficient. (1); In equation (1), This is the frequency modulation effect coefficient; This is the per-unit value of the active power of the load; This is the per-unit value of the system frequency; To match the system's rated frequency The proportionality coefficient of the load to the rated load, which is directly proportional to the power of the load.

[0054] like Figure 2 The figure shows the active power-frequency characteristic curve of the load, representing two different frequency regulation effects of the load. When the frequency changes from Down to When, its power changes ,Right now Therefore, when the system frequency decreases, Larger loads absorb more active power from the system; when the system faces low-frequency load shedding, priority should be given to disconnecting power. Smaller load, retain A larger load helps reduce the amount of load shedding and the losses from load shedding. The importance of the load is used to quantify the differences in economic losses caused by power outages; the higher the importance, the greater the loss per unit of power outage.

[0055] 2): In summary, load frequency response characteristics are introduced into the load reduction decision. Importance of load Constructing load reduction contribution factors Specifically, loads with smaller frequency regulation effect coefficients are prioritized for shelving to improve the system's frequency recovery capability; at the same time, loads of lower importance are prioritized for shelving to reduce the economic losses caused by load shelving to the system operation.

[0056] Load reduction contribution factor Calculation: This invention employs an approximation-ideal-solution method to comprehensively evaluate various indicators of each load node. Essentially, this method normalizes each indicator, constructs a positive ideal solution composed of the optimal indicator values, and a negative ideal solution composed of the worst indicator values. The distance between each evaluation indicator and these two ideal solutions is calculated to measure whether it is closer to the optimal or worst solution, thus reflecting its overall quality. The specific steps are as follows: First, the evaluation indicators are normalized: (2); In equation (2), The index value is the normalized value; Indicates the first individual load indicators The attribute value; They represent the first The minimum and maximum values ​​of each indicator across the entire set of load nodes.

[0057] Introducing indicator weights Constructing a weighted decision matrix ,in: , Indicates the number of load nodes. This indicates the quantity of load indicators.

[0058] Then, the positive and negative ideal solutions are determined separately, and the Euclidean distances from each load node to the positive and negative ideal solutions are calculated. The positive ideal solution represents the optimal value of each index among all load nodes, while the negative ideal solution represents the worst value. (3); In equation (3), These are the positive ideal solution and the negative ideal solution, respectively. denoted as Euclidean distances for the positive and negative ideal solutions, respectively.

[0059] Finally, the load reduction contribution factor of the load node is defined as follows: (4); In equation (4), The value ranges from 0 to 1. The smaller the value, the lower the load contribution and the higher the priority of shedding in load reduction decisions.

[0060] In step 2, maximizing operational efficiency in an islanded microgrid hinges on the rational optimization of various operating costs. The optimization objective of this invention is to minimize load shedding losses and system frequency deviation. The objective function for low-frequency load shedding decisions should primarily consider the aforementioned two aspects, mathematically described as follows: (5); In equation (5), Maximum time step per simulation round; The number of load nodes; For load nodes Contribution weight; For the first The load is unloaded at the time of load reduction. The amount of load shedding; The system's rated frequency; For the first The load is unloaded at the time of load reduction. The actual frequency; For the first The load is unloaded at the time of load reduction. The percentage of shear load; For the first The rated active power of each load.

[0061] 1) Power balance constraints: To ensure system stability, it is necessary to ensure that any system... The active power output is always balanced with the total load and load loss.

[0062] (6); In formula (6): The first The output power of a miniature gas turbine, wind turbine, and photovoltaic system; , The first The charging and discharging power of the energy storage device; These are the quantities of micro gas turbines, wind turbines, photovoltaic systems, and energy storage, respectively. For the first The rated active power of each load; For the first The load is unloaded at the time of load reduction. The amount of load shedding; Energy lost by the network.

[0063] 2) Output constraints of distributed power sources: (7); In equation (7): These are the upper and lower limits of the active power of the micro gas turbine, respectively. These are the upper limits of the output power of wind turbines and photovoltaics, respectively.

[0064] 3) Energy storage constraints: (8); In equation (8): , The first The charging and discharging power of the energy storage device; These represent the charging and discharging states of energy storage. These are the upper limits for energy storage charging and discharging power, respectively. , They are time points and time Energy storage capacity; These are the charging and discharging efficiencies of energy storage, respectively. , They are time points and time The energy storage state of charge; This represents the change in energy storage capacity. This refers to the rated capacity of the energy storage. These represent the minimum and maximum states of charge of the energy storage unit, respectively.

[0065] 4) Frequency recovery constraint: (9); In equation (9): This represents the minimum value of the system's transient frequency; This refers to the steady-state frequency deviation of the system. This represents the maximum steady-state frequency deviation of the system.

[0066] 5) Voltage safety constraints: To ensure that the voltage does not remain excessively low or high for an extended period while the frequency is restored, a safe voltage range is set: (10); In formula (10): For nodes The actual voltage at that location They are nodes The minimum and maximum voltage values ​​allowed for operation under load.

[0067] In step 3, the main elements of the Markov game process include a set of agents. State space Action space and reward function ; At any load reduction moment The interaction process between each intelligent agent and the microgrid environment is as follows: First of all, the An agent observes the local load status. and through strategy Generate corresponding control actions ; Then, the actions of all agents are combined to form a joint action. It operates in the microgrid environment; Subsequently, the system based on joint actions Provide immediate rewards to each agent and transition to the next state. .

[0068] 1) State space: The local state of each load node consists of the load node frequency deviation, frequency change rate, and bus voltage. The global state, combining the local states of all agents, the output power of each distributed power source, the charging and discharging power of the energy storage unit, and the state of charge of the energy storage itself, describes the overall operating environment of the microgrid. For each given agent... , That is, the number of load nodes, at time... The local state and the global state of the system are represented as follows: (11); In equation (11): , These represent the local state and the global state of the agent, respectively. , , They are the load nodes. At any moment Frequency deviation, rate of frequency change, and voltage; , , They were respectively in the second Taiwan's micro gas turbines, wind turbines, and photovoltaics are constantly ; output power; , The first Taiwan's energy storage at any time The charging and discharging power; For a moment The state of charge of energy storage.

[0069] 2) Motion space: Each agent, based on its current state and according to the policy, Output action, which is defined as the load node At any moment The load shedding ratio, i.e., the operating space. , For the first The load is unloaded at the time of load reduction. The percentage of load shedding.

[0070] 3) Reward and punishment functions: In the load reduction strategy, all agents cooperate fully and share the same interests, achieving a common goal through collaboration to obtain the same reward; therefore, the reward function is set consistently. Combining this with a low-frequency load reduction optimization model, the first... Reward function for each load node This includes both target-based rewards and constraint-based rewards; The target reward includes a load shedding loss reward. and frequency offset reward ; The constraint reward is constructed from equations (6) to (10). , respectively represented as: (12); In equation (12): These are the weighting coefficients for each reward; The penalty coefficient for restoring the system when the constraints are not met; For load nodes Contribution weight; For the first The load is unloaded at the time of load reduction. The amount of load shedding; The system's rated frequency; For the first The load is unloaded at the time of load reduction. The actual frequency.

[0071] To link immediate rewards with long-term returns, a value function is constructed for each agent. Characterizing intelligent agents Starting from the initial state, follow the strategy. The expected cumulative return that can be obtained is shown in equation (13): (13); In equation (13): The desired value is the weighted average over multiple simulation rounds. This is the cumulative reward discount coefficient; The maximum time step for each simulation round; For the first Each load node at time... The reward function.

[0072] Step 4 includes: 4.1: Solving the Markov game model for distributed low-frequency load shedding: like Figure 3 The diagram shows the basic structure of the improved MADDPG algorithm. The Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG) configures an independent action network for each agent. and value network and its corresponding target network , ; in: This refers to the local state space of the agent. It is the set of global state spaces for all intelligent agents; , , , These are the parameters for the action network, value network, and their corresponding target network, respectively. It is the set of action spaces for all intelligent agents.

[0073] Action networks are used for approximate deterministic policy functions In contrast, value networks, under centralized training conditions, evaluate the value of actions based on global information.

[0074] During the training phase, the policy function is trained using a deep neural network. and The value functions are jointly fitted, and the value function is maximized. Update network parameters to achieve continuous optimization of the strategy in a high-dimensional continuous action space; The value represents the combined loss caused by load shedding and frequency shift, and provides gradient guidance for policy updates.

[0075] To further improve training efficiency and optimize multi-agent cooperation strategies, this invention introduces a Priority Experience Replay (PER) mechanism based on the MADDPG algorithm. The importance of each experience is measured by its temporal aldifference (TD) error, and the sampling probability is dynamically adjusted to ensure that high-value experiences are utilized more extensively during training. The expression for calculating the TD error is as follows: (14); In equation (14): the larger the TD error, the greater the difference between the value network and the target network. The more significant the value deviation, the higher the sampling probability of the corresponding experience, which helps to update the network evaluation value and optimize the strategy.

[0076] This invention employs a ranking-based priority mechanism, using the network training loss as an empirical importance indicator. The formula for calculating the empirical probability priority is as follows: (15); In equation (15): The first The probability and priority of each experience being sampled; Indicates based on experience The experience replay buffer is sorted from largest to smallest, with a smaller sequence number indicating higher priority; This is the sampling priority parameter. The larger the value, the more it relies on priority for sampling; For the first The relative weights of each empirical sample; This represents the sum of the weights of all samples in the experience replay pool; This represents the total number of samples stored in the experience replay pool.

[0077] However, frequently replaying experiences with high TD errors can alter the sample distribution, easily leading to training oscillations or even divergence. Therefore, importance sampling weights are introduced to adjust the update magnitude of high-priority experiences: (16); In equation (16): For the first The importance sampling weight of each experience sample; No. Normalized sampling weights for empirical samples; It is the size of the sample storage capacity in the experience pool; Hyperparameters representing the influence of priority samples on the convergence result; This represents the maximum weight of all samples in the sampling batch.

[0078] Based on this, the agent's neural network is trained during its interaction with the environment. For samples taken from the priority experience replay pool... In the batch of samples, each agent The target action network is based on the next time step state. Generate predicted actions And the target value network calculates the corresponding target. value: (17); In equation (17): For the first The goal of an intelligent agent value; For the first The instant reward obtained by an intelligent agent; Discount factor; For the first A network of target values ​​for individual agents; , These are the set of the agent's state space and action space for the next moment, respectively. These are the parameters of the target value network.

[0079] Value network parameters The update is performed by minimizing the following loss function: (18); In formula (18): This is the loss function for the value network, used to update the value network; This indicates the batch size of samples taken from the experience replay pool; No. Normalized sampling weights for empirical samples; For the first A value network of intelligent agents; These are the parameters of the value network.

[0080] Action network parameters Use the following gradient ascent strategy for gradient updates: (19); In equation (19): Value function Action network parameters Find the partial derivatives; Value network Action Find the partial derivative; Action network Its parameters Find the partial derivative.

[0081] Parameters of the Action and Value Target Network and Use the following soft update method: (20); In equation (20): It is a constant much smaller than 1, used to reduce the variation of neural network parameters, making the model easier to converge.

[0082] 4.2: Training process for solving Markov game models: According to the algorithm described above, as Figure 4 Training process for a distributed low-frequency load shedding strategy based on improved MADDPG in an islanded microgrid simulation environment: In each round of training, each agent constructs its current state using local measurement information and uses the action network to select the currently executable deload action. In the islanded microgrid simulation environment, the system completes the state transition based on the load reduction actions of all agents, and obtains the new node frequency, voltage and system power balance. Subsequently, the system operating status is evaluated based on the objective function constructed based on the load shedding loss and frequency offset, and the instantaneous reward value of each agent is calculated according to Equation (12); Each agent constructs an experience sample from its current state, action, reward value, and next state, and stores it in the experience replay pool. Each agent assigns priority to the experience samples according to the TD error and calculates the experience sampling probability distribution according to equations (12) to (15). Each agent samples experience in batches from the experience replay pool and updates the action network, value network, and their corresponding target network according to equations (16) to (20). When time... Reaching the maximum time step When the current round ends and the next round begins; during training rounds Reach the preset maximum number of rounds In this way, the intelligent agents can form a stable cooperative strategy and achieve optimal distributed load reduction control under conditions of no communication.

[0083] Its effectiveness and feasibility were verified through simulation. This invention employs an improved IEEE-13 node microgrid system model for simulation verification in MATLAB / Simulink, and its topology is as follows: Figure 5 As shown in the figure, the system consists of one 8MVA micro gas turbine, two 2MW photovoltaic arrays, two 1.5MW wind turbines, two 0.5MW battery energy storage units, and nine load nodes. The rated voltage of the microgrid is 10kV. The importance of each load node, the frequency regulation effect coefficient, and the load reduction contribution factor are set as shown in Table 1. Through extensive parameter optimization and experimental verification, the optimal parameter configuration of the algorithm was determined as shown in Table 2.

[0084]

[0085]

[0086] This invention is based on a distributed low-frequency load shedding method for isolated microgrids using an improved multi-agent cooperative algorithm. The effectiveness is verified as follows: 1) Performance analysis of the improved MADDPG algorithm: The distributed MADDPG algorithm and the centralized DDPG algorithm were selected as comparison algorithms. By adjusting the generator output and load level, the system load level was randomly set to 90% to 110% of the rated load during the simulation initialization phase, and generator tripping faults were applied at the same time to generate sufficient training samples to simulate the frequency disturbances of the islanded microgrid.

[0087] Figure 6The global average reward curves of the three algorithms were obtained through offline training for 5000 rounds. The improved MADDPG algorithm showed an improvement in the global average reward value in multi-agent cooperative decision-making compared to the original MADDPG algorithm, and its convergence value was also significantly higher. Therefore, the improved MADDPG algorithm exhibits significant advantages in both overall training performance and policy stability.

[0088] 2) Comparative analysis of the decision-making effects of different low-frequency load reduction schemes: This invention sets two relatively severe fault conditions when the system is running stably for 5 seconds: Condition 1 causes the microgrid to disconnect from the main grid due to a main grid fault, resulting in a 2MW power deficit, thereby simulating unplanned islanding operation of the microgrid; Condition 2 adds wind turbine disconnection to Condition 1, increasing the power deficit to 3.5MW.

[0089] Depend on Figure 7 and Figure 9 It can be seen that the load reduction scheme proposed in operating condition 1 can still meet the system frequency stability requirements while reducing the load shedding amount. In the initial stage of the fault, the load reduction action has a short start time, which timely suppresses the frequency drop process and significantly increases the frequency minimum point. In the subsequent recovery stage, the frequency curve changes smoothly, and the steady-state frequency deviation is maintained at a low level. This proves that the distributed control structure effectively reduces the impact of communication delay, enabling each agent to quickly complete collaborative decision-making under the drive of local information, thereby improving the system stability and dynamic response speed.

[0090] Depend on Figure 8 It can be seen that the lowest frequency of the improved MADDPG method in fault condition 2 is 49.00Hz, which is also better than the other two schemes; in the case of fault condition 2, the lowest frequency of the improved MADDPG method is 49.00Hz. Figure 10 A comparison of load response information shows that the action time is 113ms and the load reduction is 2.69MW, the smallest among the three schemes. This indicates that even under severe fault conditions, this scheme can still maintain stable frequency control and has good adaptability. However, as shown in Table 3, in condition 2 with large initial system disturbances, the MADDPG and DDPG algorithms need to cut off more loads to meet higher frequency requirements, cutting off loads with higher weights, LD8 and LD2, respectively. In contrast, the improved MADDPG algorithm, while meeting the system's transient steady-state frequency deviation requirements, prioritizes cutting off loads with lower importance, effectively balancing the power supply reliability of critical loads. This result shows that the distributed agent cooperative mechanism can more rationally consider the load importance and load frequency response characteristics, demonstrating higher control sensitivity, better frequency recovery performance, and a more reasonable load allocation strategy. .

Claims

1. A distributed low-frequency load shedding method for isolated microgrids based on an improved multi-agent cooperative algorithm, characterized in that... Includes the following steps: Step 1: Construct a load shedding contribution factor that takes into account load frequency characteristics and load importance, in order to quantify load shedding losses; Step 2: Establish a low-frequency load shedding optimization model that minimizes load shedding losses and system frequency deviation; Step 3: Transform the distributed low-frequency load shedding problem into a Markov game process, where each load node is regarded as an independent agent, and define the state space, action space and reward function of each agent based on the low-frequency load shedding optimization model established in Step 2. Step 4: Train each agent using the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm with a priority experience replay mechanism, so that the agents learn low-frequency load reduction strategies that rely on local real-time information.

2. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 1, characterized in that: Step 1 includes: 1) Analyze the load frequency characteristics and load importance: When the system power balance is disturbed and causes a frequency shift, the load will produce a corresponding power change according to its own frequency characteristics. This response is described by the load's frequency regulation effect coefficient. （1）； In equation (1), This is the frequency modulation effect coefficient; This is the per-unit value of the active power of the load; This is the per-unit value of the system frequency; To match the system's rated frequency The proportionality coefficient of the load to the rated load that is directly proportional to the power of the load; For two different frequency regulation effect loads When the frequency changes from Down to When, its power changes ,Right now Therefore, when the system frequency decreases, Larger loads absorb more active power from the system; when the system faces low-frequency load shedding, priority is given to disconnecting power. Smaller load, retain Larger loads; load importance is used to quantify the differences in economic losses caused by power outages, with higher importance resulting in greater losses per unit of power outage; 2): Incorporating load frequency response characteristics into load reduction decisions. Importance of load Constructing load reduction contribution factors Specifically, loads with smaller frequency regulation effect coefficients are prioritized for shelving to improve the system's frequency recovery capability; at the same time, loads with lower importance are prioritized for shelving to reduce the economic losses caused by load shelving to the system operation.

3. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 2, characterized in that: Load reduction contribution factor The calculation is as follows: The approximation ideal solution method is used to comprehensively evaluate each index of each load node. In essence, this method normalizes each index and constructs a positive ideal solution composed of the optimal index values ​​and a negative ideal solution composed of the worst index values. By calculating the distance between each evaluation index and the two ideal solutions, it measures whether it is closer to the optimal or the worst, thereby reflecting its overall quality.

4. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 3, characterized in that: Load reduction contribution factor The calculation includes the following steps: First, the evaluation indicators are normalized: （2）； In equation (2), The index value is the normalized value; Indicates the first individual load indicators The attribute value; They represent the first The minimum and maximum values ​​of each indicator across the entire set of load nodes; Introducing indicator weights Constructing a weighted decision matrix ,in: , Indicates the number of load nodes. Indicates the quantity of load indicators; Then, the positive and negative ideal solutions are determined separately, and the Euclidean distances from each load node to the positive and negative ideal solutions are calculated. The positive ideal solution represents the optimal value of each index among all load nodes, while the negative ideal solution represents the worst value. （3）； In equation (3), These are the positive ideal solution and the negative ideal solution, respectively. , respectively, are the Euclidean distances between the positive and negative ideal solutions; Finally, the load reduction contribution factor of the load node is defined as follows: （4）； In equation (4), The value ranges from 0 to 1. The smaller the value, the lower the load contribution and the higher the priority of shedding in load reduction decisions.

5. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 4, characterized in that: In step 2, in an islanded microgrid, the optimization objective is to minimize load shedding losses and system frequency deviation; the objective function for low-frequency load shedding decision is described as follows: （5）； In equation (5), Maximum time step per simulation round; The number of load nodes; For load nodes Contribution weight; For the first The load is unloaded at the time of load reduction. The amount of load shedding; The system's rated frequency; For the first The load is unloaded at the time of load reduction. The actual frequency; For the first The load is unloaded at the time of load reduction. The percentage of shear load; For the first The rated active power of each load.

6. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 5, characterized in that: The objective function for low-frequency load shedding decisions includes: 1) Power balance constraints: To ensure system stability, it is necessary to ensure that any system... The active power output is always balanced with the total load and load loss; （6）； In formula (6): The first The output power of a miniature gas turbine, wind turbine, and photovoltaic system; , The first The charging and discharging power of the energy storage device; These are the quantities of micro gas turbines, wind turbines, photovoltaic systems, and energy storage, respectively. For the first The rated active power of each load; For the first The load is unloaded at the time of load reduction. The amount of load shedding; Energy lost by the network; 2) Output constraints of distributed power sources: （7）； In equation (7): These are the upper and lower limits of the active power of the micro gas turbine, respectively. These are the upper limits of the output power of wind turbines and photovoltaic systems, respectively. 3) Energy storage constraints: （8）； In equation (8): , The first The charging and discharging power of the energy storage device; These represent the charging and discharging states of energy storage, respectively. These are the upper limits for energy storage charging and discharging power, respectively. , They are time points and time Energy storage capacity; These are the charging and discharging efficiencies of energy storage, respectively. , They are time points and time The energy storage state of charge; This represents the change in energy storage capacity. This refers to the rated capacity of the energy storage. These are the minimum and maximum states of charge of the energy storage unit, respectively. 4) Frequency recovery constraint: （9）； In equation (9): This represents the minimum value of the system's transient frequency; This refers to the steady-state frequency deviation of the system. This represents the maximum steady-state frequency deviation of the system. 5) Voltage safety constraints: To ensure that the voltage does not remain excessively low or high for an extended period while the frequency is restored, a safe voltage range is set: （10）； In formula (10): For nodes The actual voltage at that location They are nodes The minimum and maximum voltage values ​​allowed for operation under load.

7. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 6, characterized in that: In step 3, the Markov game process includes a set of agents. State space Action space and reward function At any time of load reduction The interaction process between each intelligent agent and the microgrid environment is as follows: First of all, the An agent observes the local load status. and through strategy Generate corresponding control actions ; Then, the actions of all agents are combined to form a joint action. It operates in the microgrid environment; Subsequently, the system based on joint actions Provide immediate rewards to each agent and transition to the next state. .

8. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 7, characterized in that: In step 3, 1) State space: The local state of each load node consists of the load node frequency deviation, frequency change rate, and bus voltage, while the global state combines the local states of all agents, the output power of each distributed power source, the charging and discharging power of the energy storage unit, and the state of charge of the energy storage itself, to describe the overall operating environment of the microgrid; for each given agent... , That is, the number of load nodes, at time... The local state and the global state of the system are represented as follows: （11）； In equation (11): , These represent the local state and the global state of the agent, respectively. , , They are the load nodes. At any moment Frequency deviation, rate of frequency change, and voltage; , , They were respectively in the second Taiwan's micro gas turbines, wind turbines, and photovoltaics are constantly ; output power; , The first Taiwan's energy storage at any time The charging and discharging power; For a moment State of charge of energy storage; 2) Motion space: Each agent, based on its current state and according to the policy, Output action, which is defined as the load node At any moment The load shedding ratio, i.e., the operating space. , For the first The load is unloaded at the time of load reduction. The percentage of shear load; 3) Reward and punishment functions: In the load reduction strategy, all agents cooperate fully and share the same interests, achieving a common goal through collaboration to obtain the same reward; therefore, the reward function is set consistently. Combined with the low-frequency load reduction optimization model, the first... Reward function for each load node This includes both target-based rewards and constraint-based rewards; The target reward includes a load shedding loss reward. and frequency offset reward ; The constraint reward is constructed from equations (6) to (10). , respectively represented as: （12）； In equation (12): These are the weighting coefficients for each reward; The penalty coefficient for restoring the system when the constraints are not met; For load nodes Contribution weight; For the first The load is unloaded at the time of load reduction. The amount of load shedding; The system's rated frequency; For the first The load is unloaded at the time of load reduction. The actual frequency; To link immediate rewards with long-term returns, a value function is constructed for each agent. Characterizing intelligent agents Starting from the initial state, follow the strategy. The expected cumulative return that can be obtained is shown in equation (13): （13）； In equation (13): The desired value is the weighted average over multiple simulation rounds. This is the cumulative reward discount coefficient; The maximum time step for each simulation round; For the first Each load node at time... The reward function.

9. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 8, characterized in that: Step 4 includes solving the Markov game model for distributed low-frequency load shedding: The Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG) configures an independent action network for each agent. and value network and its corresponding target network , ; in: This represents the local state space of the agent. It is the set of global state spaces for all intelligent agents; , , , These are the parameters for the action network, value network, and their corresponding target network, respectively. It is the set of action spaces for all intelligent agents; Action networks are used for approximate deterministic policy functions In contrast, value networks, under centralized training conditions, evaluate the value of actions based on global information. During the training phase, the policy function is trained using a deep neural network. and The value functions are jointly fitted, and the value function is maximized. Update network parameters to achieve continuous optimization of the strategy in a high-dimensional continuous action space; The value represents the combined loss caused by the load shedding and frequency shift, and provides gradient guidance for policy updates; Based on the MADDPG algorithm, a Priority Experience Replay (PER) mechanism is introduced. The importance of each experience is measured by its temporal aldifference (TD) error, and the sampling probability is dynamically adjusted to ensure that high-value experiences are utilized more during training. The expression for calculating the TD error is as follows: （14）； In equation (14): the larger the TD error, the greater the difference between the value network and the target network. The more significant the value deviation, the higher the corresponding empirical sampling probability; A ranking-based priority mechanism is adopted, using the network training loss as an empirical importance indicator; the empirical probability priority calculation formula is as follows: （15）； In equation (15): The first The probability and priority of each experience being sampled; Indicates based on experience The experience replay buffer is sorted from largest to smallest, with a smaller sequence number indicating higher priority; This is the sampling priority parameter. The larger the value, the more it relies on priority for sampling; For the first The relative weights of each empirical sample; This represents the sum of the weights of all samples in the experience replay pool; This represents the total number of samples stored in the experience replay pool. Importance sampling weights are introduced to adjust the update magnitude of high-priority experiences: （16）； In equation (16): For the first The importance sampling weight of each experience sample; No. Normalized sampling weights for empirical samples; It is the size of the sample storage capacity in the experience pool; Hyperparameters representing the influence of priority samples on the convergence result; This represents the maximum weight of all samples in the sampling batch; Based on this, the agent's neural network is trained during its interaction with the environment; for samples taken from the priority experience replay pool containing... In the batch of samples, each agent The target action network is based on the next time step state. Generate predicted actions And the target value network calculates the corresponding target. value: （17）； In equation (17): For the first The goal of an intelligent agent value; For the first The instant reward obtained by an intelligent agent; Discount factor; For the first A network of target values ​​for individual agents; , These are the set of the agent's state space and action space for the next moment, respectively. The parameters of the target value network; Value network parameters The update is performed by minimizing the following loss function: （18）； In formula (18): This is the loss function for the value network, used to update the value network; This indicates the batch size of samples taken from the experience replay pool; No. Normalized sampling weights for empirical samples; For the first A value network of intelligent agents; These are the parameters of the value network; Action network parameters Use the following gradient ascent strategy for gradient updates: （19）； In equation (19): Value function Action network parameters Find the partial derivatives; Value network Action Find the partial derivative; Action network Its parameters Find the partial derivative; Parameters of the Action and Value Target Network and Use the following soft update method: （20）； In equation (20): It is a constant much smaller than 1, used to reduce the variation of neural network parameters, making the model easier to converge.

10. The distributed low-frequency load shedding method for islanded microgrids based on an improved multi-agent cooperative algorithm according to claim 9, characterized in that: Step 4 includes the training process of the distributed low-frequency load shedding strategy based on the improved MADDPG in an islanded microgrid simulation environment: In each round of training, each agent constructs its current state using local measurement information and uses the action network to select the currently executable deload action. In the islanded microgrid simulation environment, the system completes the state transition based on the load reduction actions of all agents, and obtains the new node frequency, voltage and system power balance. Subsequently, the system operating status is evaluated based on the objective function constructed based on the load shedding loss and frequency offset, and the instantaneous reward value of each agent is calculated according to Equation (12); Each agent constructs an experience sample from its current state, action, reward value, and next state, and stores it in the experience replay pool. Each agent assigns priority to the experience samples according to the TD error and calculates the experience sampling probability distribution according to equations (12) to (15). Each agent samples experience in batches from the experience replay pool and updates the action network, value network and its corresponding target network according to equations (16) to (20); When time Reaching the maximum time step When the current round ends and the next round begins; during training rounds Reach the preset maximum number of rounds In this way, the intelligent agents can form a stable cooperative strategy and achieve optimal distributed load reduction control under conditions of no communication.

Images