A full-distributed load frequency control method for island microgrid

Abstract

The application relates to a full-distributed load frequency control method for an island micro-grid, which comprises the following steps: step S1, island micro-grid modeling: Markov decision process modeling is carried out on the island micro-grid, each generator set is set as an agent, and an action space and a state space are initialized; step S2, agent pre-learning: the agents are coordinately trained by using a centralized training principle, and a distributed global optimal coordination strategy is obtained; and step S3, load frequency control: the frequency state and the output state of the micro-grid are collected on site by each distributed generator set, a decentralized execution strategy is adopted to decide the power generation power instruction of the self unit, and load frequency control (LFC) is carried out. Compared with the prior art, the application has the advantages of high decision accuracy, high efficiency and high robustness.

Description

Technical Field

[0001] This invention relates to the field of distributed load frequency control in microgrids, and in particular to a fully distributed load frequency control method for islanded microgrids. Background Technology

[0002] Due to the advantages of distributed new energy and energy storage facilities, such as flexible power supply, no environmental pollution, and high energy utilization, and the complementary advantages of distributed new energy and distributed energy storage facilities, microgrid systems composed of renewable energy power generation devices, micro gas turbines, fuel cells, diesel generators, etc. have received increasing attention from the industry in recent years and have been gradually promoted and applied to power demand response to meet the diverse energy needs of different regions.

[0003] Microgrids operate in islanded mode, lacking the support of the main power grid, and rely on micro-energy sources and energy storage units to maintain system frequency and voltage stability. The intermittent nature of distributed renewable energy generation and the increasing number of distributed power sources connected to the grid will inevitably lead to significant fluctuations in the microgrid system frequency, adversely affecting the stable operation of the microgrid.

[0004] Therefore, there is an urgent need to design a fully distributed load frequency control method for islanded microgrids to solve the problems of reduced generation costs and frequency regulation performance caused by LFC decision errors in load frequency control in islanded microgrids due to communication errors and delays, and the waste of frequency regulation resources caused by the inability of the controller and power divider to cooperate. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a fully distributed load frequency control method for islanded microgrids that has high decision-making accuracy, high efficiency and high robustness.

[0006] The objective of this invention can be achieved through the following technical solutions:

[0007] This invention provides a fully distributed load frequency control method for islanded microgrids, which includes the following steps:

[0008] Step S1, Islanded Microgrid Modeling: Model the islanded microgrid using a Markov decision process, setting each generator set as an agent and initializing the action space and state space;

[0009] Step S2, Agent pre-learning: The agent is trained in a coordinated manner using the principle of centralized training to obtain a distributed global optimal coordination strategy;

[0010] Step S3, Load Frequency Control: Each distributed generator unit locally collects the frequency status and output status of the microgrid, and uses a distributed execution strategy to decide and output its own generator power command to perform load frequency control (LFC).

[0011] Preferably, step S1 specifically includes:

[0012] 1) Construct a multi-agent reinforcement learning stochastic game framework <S,A1,...,A1> based on Markov decision processes. N ,R1,...,R N ,P,γ>,where S is the state of the islanded microgrid, A i For the action of the i-th agent, R i Let be the reward for the i-th agent, N be the number of agents, P be the state transition probability, and γ be the discount factor;

[0013] 2) Determine the action space and state space;

[0014] 3) Determine the objective function and the comprehensive reward function.

[0015] Preferably, the action space and state space are specifically:

[0016] Action space A: The power generation command ΔP of the i-th generator set. order-i Action A of the i-th agent i ;

[0017] State space S: Where Δf is the frequency deviation of the islanded microgrid. This is the integral of time over frequency deviation. For the additional power output of the i-th generator set, ΔP order-i (t-1) represents the magnitude of the power output command for generator set i at time t-1.

[0018] Preferably, determining the objective function and the comprehensive reward function specifically involves:

[0019] Objective function: The objective function is to minimize the sum of the cumulative total frequency deviation and the cumulative total power generation cost. The expression is as follows:

[0020]

[0021] In the formula, α i β i γ i These are the dynamic coefficients ΔP and ΔP, representing the power generation cost of the i-th generator unit after considering power disturbances. Gi Let C be the additional power output of the i-th generator set. totalWhere T is the power generation cost, T is the total operating cycle, and N is the number of generator sets;

[0022] The comprehensive reward function considers the absolute value of the frequency deviation and the total power generation cost, and incorporates a control penalty term to guide the agent towards the optimal frequency. The expression for the comprehensive reward function is:

[0023]

[0024]

[0025] In the formula, r is the reward function value, Δf is the frequency deviation, and C i Let be the power generation cost of the i-th generator set, μ1 and μ2 be the corresponding weighting coefficients, B be the penalty function, and c be the set frequency deviation threshold.

[0026] Preferably, the state transition of the islanded microgrid is performed using a joint strategy, expressed as:

[0027]

[0028] In the formula, π i (A i |S) is the policy function of the i-th agent, A i represents the action of the i-th agent, S represents the state of the islanded microgrid, and N is the number of agents;

[0029] The reward value expression for each agent is:

[0030]

[0031] In the formula, A is the reward for the i-th agent at step t. t,i is the action of the i-th agent at step t; s and a are the state value and action value of the agent, respectively.

[0032] The Bellman equations for the state-value function and the state-action-value function in a multi-agent system are as follows:

[0033] V i (s)=E i [R t+1 +γV i (S t+1 )∣S t =s] (6)

[0034] Q i (s,a)=E i [R t+1 +γQ i (S t+1 A t+1)∣S t =s,A t =a] (7)

[0035] In the formula, V i (s) is the action of the i-th agent in state s, E i S is the mathematical expectation of the i-th agent. t+1 Let A be the state of the islanded microgrid at step t+1. t+1 For the joint action of the islanded microgrid at step t+1, Q i Let γ be the Q-value function of the i-th agent, and γ be the discount factor.

[0036] Preferably, in step S2, the agent is trained in a coordinated manner using the principle of centralized training. Specifically, the agent is trained in a centralized and coordinated manner using the Distributed Deep Reinforcement Learning-Distributed Decomposition Multi-role Multi-agent Deep Deterministic Policy Gradient Algorithm (DDMR-MADDPG).

[0037] Preferably, the Distributed Deep Reinforcement Learning-Distributed Decomposition Multi-Agent Deep Deterministic Policy Gradient Algorithm (DDMR-MADDPG) employs a multi-agent deep reinforcement learning model based on an actor-critic framework. The actor network outputs actions according to the current observation state, and the critic network calculates the Q-value based on the state-action pair and feeds the calculated Q-value back to the actor network to adjust the action policy.

[0038] Preferably, the structure of the Distributed Deep Reinforcement Learning-Distributed Decomposition Multi-Agent Deep Deterministic Policy Gradient Algorithm (DDMR-MADDPG) includes:

[0039] Parallel systems: Each parallel system is injected with load disturbances of different amplitudes and durations. The amplitude of the random disturbances is set according to the course learning strategy and increases as the number of episodes increases.

[0040] Explorers explore the environment of their respective parallel systems and provide the exploration samples obtained to the leader. Each explorer has only one actor network, and different explorers use different exploration principles to conduct distributed exploration of the environment.

[0041] Demonstrator: Based on the idea of ​​imitation learning, the demonstrator is equipped with a controller or a distributor. The controllers or distributors in different parallel systems have obtained preset performance through parameter tuning. They interact with the environment and obtain demonstration samples, which are then provided to the leader for learning.

[0042] Leader: This is the agent that will eventually be installed in the load frequency control LFC coordination strategy in the online application; In each step, the leader randomly takes samples from the common experience pool and updates its own network parameters through the classification experience replay mechanism, and then periodically transmits these network parameters to the explorer for the explorer to update;

[0043] Common Experience Pool: A classification experience replay strategy is adopted. Each leader has two independent experience pools. The first common experience pool stores samples from explorers, and the second common experience pool stores samples from demonstrators. The leader selects samples from the first common experience pool with a set variable probability ξ, and selects samples from the second common experience pool with 1-ξ. The probability changes as the number of episodes increases.

[0044] Preferably, the agent pre-learning process specifically includes the following sub-steps:

[0045] 1) Optimize the demonstrator's controller and allocator, and initialize the parameters of the leader and explorer;

[0046] 2) Initialize the episodes;

[0047] 3) Initialize the exploration principle and obtain the initial state;

[0048] 4) Select a random perturbation for each parallel system, calculate the explorer action for each system, and output the action for each demonstrator based on its principle;

[0049] 5) Based on the classification experience, the samples are stored in the first public experience pool or the second public experience pool. Each leader draws samples from the first public experience pool by setting a probability ξ, or draws samples from the second public experience pool by 1-ξ.

[0050] 6) Each leader determines the target value for each sample in the mini-batch, calculates the loss function, and updates its own parameters according to the policy gradient;

[0051] 7) The leader updates the target network and distributes parameters to the explorers;

[0052] 8) Determine if the current time has reached the set maximum step size. If not, increment the step size by 1 and go to step 4); otherwise, go to step 9.

[0053] 9) Determine if the current episode count has reached the set number of episodes. If not, increment the episode count by 1 and proceed to step 3); otherwise, proceed to step 10.

[0054] 10) Output the training results.

[0055] Preferably, the explorer employs any of the following strategies for action exploration:

[0056] The explorer uses a greedy strategy, and the corresponding exploration action expression is:

[0057]

[0058] In the formula, This is the action of the lth explorer. It is the policy function of the l-th explorer. It is a random action;

[0059] The explorer employs an OU noise detection strategy, and the corresponding detection action expression is:

[0060]

[0061] In the formula, This is the action of the j-th explorer. It is the policy function of the j-th explorer. This is OU noise;

[0062] The explorer employs a Gaussian noise detection strategy, and the corresponding exploration action expression is:

[0063]

[0064] In the formula, This is the action of the m-th explorer. It is the policy function of the m-th explorer. It is Gaussian noise.

[0065] Compared with the prior art, the present invention has the following advantages:

[0066] 1) This invention mimics the distributed neural network and decision-making of starfish, equating each distributed power source with an intelligent agent. It considers global optimality and sets reward functions for each intelligent agent. By training all the intelligent agents in a centralized manner, the coordination and control of multiple intelligent agents are considered in the pre-learning process, thereby obtaining the global optimal strategy. When running online, each distributed power source unit only needs to collect the frequency status of the microgrid and its own output status to make a globally optimal decision without any mutual communication, thus completely solving the control problem caused by communication failure.

[0067] 2) This invention considers the mutual coupling and influence between the control process of the controller and the distribution process of the distributor. Through distributed decision theory, it improves the frequency regulation performance of the microgrid and reduces the waste of frequency regulation resources.

[0068] 3) This invention adopts the DDMR-MADDPG algorithm, introduces multi-role training theory and combines various techniques from imitation learning and course learning to improve robustness and adaptability. It can effectively improve the frequency regulation performance of load frequency control and reduce power generation costs, achieving multi-objective comprehensive optimization. Attached Figure Description

[0069] Figure 1 This is a flowchart of the method of the present invention;

[0070] Figure 2 A schematic diagram of the agent's pre-learning process;

[0071] Figure 3 This is a schematic diagram of the starfish neural network structure;

[0072] Figure 4 A schematic diagram of the LFC framework structure for a fully distributed SFD-load frequency control system for "starfish". Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0074] Example

[0075] like Figure 1 As shown in the figure, this embodiment presents a fully distributed load frequency control method for islanded microgrids, which includes the following steps:

[0076] Step S1, Islanded Microgrid Modeling: Model the islanded microgrid using a Markov decision process, setting each generator set as an agent and initializing the action space, state space, and reward function;

[0077] Step S2, Agent pre-learning: The agent is trained in a coordinated manner using the principle of centralized training to obtain a distributed global optimal coordination strategy;

[0078] Step S3, Load Frequency Control: Each distributed generator unit locally collects the frequency status and output status of the microgrid, and uses a distributed execution strategy to decide and output its own generator power command to perform load frequency control (LFC).

[0079] The method of this embodiment will now be described in detail.

[0080] Step S1 specifically includes:

[0081] 1) Construct a stochastic game framework for multi-agent reinforcement learning based on Markov decision processes:

[0082] <S,A1,...,A N ,R1,...,R N ,P,γ> (1)

[0083] Where S represents the state of the islanded microgrid, A i R represents the action of the i-th agent. i γ represents the reward for the i-th agent, N is the number of agents, P represents the state transition probability, and γ represents the discount factor.

[0084] In a multi-agent system, state transition is the result of all agents jointly performing actions, and the reward obtained depends on the joint policy, which can be represented as follows:

[0085]

[0086] In the formula, π i (a i |s) is the policy function of the i-th agent, s is the state of the islanded microgrid, and a i Let be the action of the i-th agent.

[0087] The joint policy is the sum of the joint decision policies of all agents, and the reward for each agent is as follows:

[0088]

[0089] In the formula, It is the action of the i-th agent at step t, A t , i It is the action of the i-th agent at step t.

[0090] The Bellman equations for the state-value function and the state-action-value function in a multi-agent system are shown in (4) and (5).

[0091] V i (s)=E i [R t+1 +γV i (S t+1 )∣S t =s] (4)

[0092] Q i (s,a)=E i [R t+1 +γQ i (S t+1 A t+1 )∣S t =s,At =a] (5)

[0093] In the formula, V i (s) is the action of the i-th agent in state s, E i S is the mathematical expectation of the i-th agent. t+1 Let A be the state of the islanded microgrid at step t+1. t+1 For the joint action of the islanded microgrid at step t+1, Q i Let be the Q-value function of the i-th agent. The Markov decision process modeling includes the setting of the state space, action space, and reward function, as detailed below:

[0094] 1) Action Space

[0095] Considering that the role of the agent is to output the power generation command of the unit, so that the agent can use a more standard noise sequence during the exploration process, for any time t, the power generation command of unit i is used as the action of the agent, and there is one action in total, as shown in the following formula.

[0096] [ΔP order-i / 10] (6)

[0097] In the formula, ΔP order-i This is the power generation command for the i-th intelligent agent.

[0098] 2) State Space

[0099] The state space includes the frequency deviation, the integral of time over the frequency deviation Δf, the actual power output of unit i, and the magnitude of the power output command at the previous moment. There are a total of 4 states, as shown in equation (7):

[0100]

[0101] In the formula, Δf represents the frequency deviation of the islanded microgrid. This is the integral of time over frequency deviation. For the additional output power of actual unit i, ΔP order-i (t-1) represents the magnitude of the power generation command of unit i at the previous moment.

[0102] 3) Objective function and comprehensive reward function

[0103] If the system has n AGC units, the objective function for the coordinated control of its LFC (Local Frequency Control) units, considering the cumulative total frequency deviation and the cumulative total generation cost, can be described as follows:

[0104]

[0105] In the formula, α i ,β iγ i These are the dynamic coefficients ΔP and ΔP, representing the power generation cost of the i-th generating unit after considering power disturbances. Gi C is the additional power output of the i-th unit. total T is the cost of electricity generation, and T is the total operating cycle.

[0106] According to the objective function of formula (8), the reward function considers the absolute value of the frequency deviation and the total power generation cost, and a control penalty term is used to guide the agent to explore in the direction of optimal frequency. The comprehensive reward function is as follows:

[0107]

[0108]

[0109] In the formula, r is the reward function, and C i Let B be the power generation cost of the i-th unit, and let B be the penalty function.

[0110] In step S2, the agent is coordinated and trained using the principle of centralized training, specifically as follows:

[0111] The agents in an isolated microgrid adopt a centralized training principle. During the learning process, each agent can stabilize the training environment by observing the actions of other agents, as detailed below:

[0112] During pre-learning, the agents adopt a centralized training principle, mainly using a distributed deep reinforcement learning algorithm suitable for multi-agent systems based on the Multi-Agent Deep Deterministic Policy Gradient Algorithm (MADDPG) – the Distributed Decomposition Multi-Role Multi-Agent Deep Deterministic Policy Gradient Algorithm (DDMR-MADDPG).

[0113] This algorithm employs an actor-critic framework. The actor network outputs actions based on the current observed state, while the critic network calculates Q-values ​​based on state-action pairs and feeds these Q-values ​​back to the actor network for adjusting its action policy. During training, the critic network calculates not only its own state-action Q-value but also the influence of other agents' state-action pairs in the environment. In other words, each agent's critic network considers the state-action pairs of all agents in the environment when calculating its Q-value. This allows each agent to continuously adjust its actor network's policy based on its relationships with the environment and with other agents to complete the design task. The centralized evaluation function allows for the calculation of other agents' policies, representing centralized training. This enables agents to estimate the actions and policies of other agents during training, facilitating effective use of this information for complex action coordination and policy communication during decision-making.

[0114] In this framework, each agent obtains the actions required to reach the current state based on its own policy. It also gains experience samples by interacting with its environment and adds them to its own experience pool. The agent observes and acts, and updates the actors by outputting the Q-factor of each individual actor. For each agent, random minimum batches of samples are sampled from the experience pool into the critic network.

[0115] The DDMR-MADDPG algorithm employs a distributed multi-agent deep reinforcement learning training framework based on multiple roles. The algorithm's structure includes: a parallel system, explorers, demonstrators, a leader, and a shared experience pool, specifically:

[0116] 1) Parallel Systems: This framework contains 12 parallel systems, each injected with load disturbances of varying amplitude and duration. The first six parallel systems each have *i* agents, representing different generator units. These *i* agents can be multiple ε-explorers using a greedy strategy, multiple G-explorers using Gaussian noise, or multiple OU-explorers using OU noise. In the latter six parallel systems, each has two demonstrators, representing a controller and a distributor respectively. These demonstrators employ control and distribution algorithms based on different principles.

[0117] The environment in each parallel system is random, and each region corresponds to a different random perturbation. The amplitude of this random perturbation increases with the number of episodes, thus employing a course learning strategy to enrich the diversity of the samples.

[0118] 2) Explorers: The main responsibility of explorers is to explore the environment of their respective parallel systems and provide the obtained exploration samples to the leader. There is only one actor network in each explorer, and different explorers use different exploration principles to conduct distributed exploration of the environment.

[0119] In this embodiment, the explorer structure includes only one actor network and uses a different network model. The explorers of different parallel systems employ different exploration principles, including three different exploration principles: greedy strategy, Gaussian noise, and OU noise.

[0120] like Figure 4 As shown, the explorer in parallel systems 1-4 adopts a greedy strategy (called the ε-explorer), and the exploration actions are as follows:

[0121]

[0122] in, This is the action of the lth explorer. It is the policy function of the l-th explorer. It is a random action.

[0123] In parallel system 5-8, the explorer uses an OU noise detection strategy. The OU explorer is an explorer whose detection actions are as follows:

[0124]

[0125] in, This is the action of the j-th explorer. It is the policy function of the j-th explorer. This is OU noise.

[0126] In parallel systems 9-12, the explorers use a Gaussian noise detection strategy; therefore, these explorers are called Gaussian explorers. The exploration actions are as follows:

[0127]

[0128] in, This is the action of the m-th explorer. It is the policy function of the m-th explorer. It is Gaussian noise.

[0129] 3) Demonstrators: Based on the idea of ​​imitation learning, demonstrators may include a controller or an allocator. The controllers or allocators in different parallel systems employ different algorithms and have achieved good performance through parameter tuning. The main responsibility of these demonstrators is to obtain demonstrative samples through effective interaction with the environment and provide them to the leader for learning. These demonstrative samples are high-value samples that can effectively guide the leader's training.

[0130] 4) Leader: The leader is the agent that will ultimately be installed in the load frequency control (LFC) coordination strategy in the online application. Therefore, the role of all other agents is to generate samples for the leader to learn from. The leader's main responsibility is to randomly select samples from the shared experience pool at each step and update its own network parameters through a classification experience replay mechanism. These network parameters are then periodically transmitted to the explorers for updates. A comprehensive Q-value overestimation prevention strategy is employed, utilizing techniques in the reviewers to improve the algorithm's robustness.

[0131] 5) Common Experience Pool: The common experience pool adopts a classification experience replay strategy. Each leader has two independent experience pools. Common experience pool 1 stores samples from explorers and common experience pool 2 stores samples from demonstrators. The leader selects samples from common experience pool 1 with a certain probability ξ and selects samples from common experience pool 2 with 1-ξ. The probability changes as the number of episodes increases.

[0132] This invention employs a centralized training approach. The final pre-trained agent serves as the leader, and i leaders are deployed as agents in each unit. During online operation, they are responsible for outputting the corresponding unit's power generation commands and observing the system status. In pre-learning, the leader is responsible for collecting samples for centralized training, while explorers and demonstrators are responsible for real-time processing and execution, as well as exploring the environment. This multi-role exploration strategy improves training efficiency.

[0133] The training process can be summarized as follows:

[0134] Since the actions between agents can affect the control effect of load frequency control (LFC), during intensive training, each agent can observe the actions of other agents to adjust its own actions more precisely, thereby coordinating with other agents in different regions to achieve a balanced state and obtain a balanced optimal solution.

[0135] like Figure 2 As shown, the agent pre-learning process specifically includes the following sub-steps:

[0136] 1) Optimize the demonstrator's controller and allocator, and initialize the parameters of the leader and explorer;

[0137] 2) Initialize the episodes;

[0138] 3) Initialize the exploration principle and obtain the initial state;

[0139] 4) Select a random perturbation for each parallel system, calculate the explorer action for each system, and output the action for each demonstrator based on its principle;

[0140] 5) Based on the classification experience, the samples are stored in the first public experience pool or the second public experience pool. Each leader draws samples from the first public experience pool by setting a probability ξ, or draws samples from the second public experience pool by 1-ξ.

[0141] 6) Each leader determines the target value for each sample in the mini-batch, calculates the loss function, and updates its own parameters according to the policy gradient;

[0142] 7) The leader updates the target network and distributes parameters to the explorers;

[0143] 8) Determine if the current time has reached the set maximum step size. If not, increment the step size by 1 and go to step 4); otherwise, go to step 9.

[0144] 9) Determine if the current episode count has reached the set number of episodes. If not, increment the episode count by 1 and proceed to step 3); otherwise, proceed to step 10.

[0145] 10) Output the training results.

[0146] In online applications, a distributed execution strategy is adopted based on the trained algorithm. The decision-making strategy uses biomimicry, mimicking the decision-making form of a distributed neural network similar to a starfish. Specifically, as follows... Figure 3 As shown, the SFD-LFC strategy references the distributed neural network structure of starfish.

[0147] The nervous systems of starfish and all echinoderms are dispersed, without forming ganglia or nerve centers, and are tightly connected to epithelial cells. Figure 3 As shown, it possesses an advanced processing system. Unlike the human body, which transmits information from the entire body to the brain for processing, the starfish's nervous system lacks a central processing neural structure. Instead, all neural structures are distributed across its tube feet, allowing information within the starfish to be shared through the nervous system. The nervous systems of all the tube feet appear to control movement through the physical interaction between the body and its environment. This creates a mechanism for the mechanical transmission of "information" between the tube feet. A single tube foot only needs to perceive its own state (proprioception) and react accordingly. Because its perception is mechanically coupled to other tube feet, they work together. When a tube foot begins to move, each tube foot generates an individual force, becoming part of the collective. Each tube foot also reacts to the forces generated by other tube feet, eventually establishing a shared rhythm. This decision-making method of the starfish is what is known as edge computing—relieving the computational burden on the core by distributing some data to other parts of the computer for processing, thus achieving higher efficiency. Therefore, the starfish represents an edge computing structure capable of achieving edge-cloud collaboration. By studying the motion model of starfish, it can be used to guide multi-agent system-based controllers, thereby achieving microgrid frequency control through the coordinated action of multiple agents.

[0148] In the load frequency control (LFC) model of an islanded microgrid, under the SFD-LFC framework, the microgrid includes i agents representing different distributed power sources. This structure mimics the distributed neural network structure of a starfish, and has better decision-making ability and anti-interference capability.

[0149] In agent pre-learning, multiple agents can receive each other's policies and states. By designing a reasonable global reward function, multiple agents can train their own optimal control allocation strategies. In online applications, these agents can make independent decisions by independently collecting the state of the microgrid without needing to communicate with each other. This is analogous to the coordinated movement of starfish pins; each pin only needs to collect its current state and make its own decision, without communicating with other pins. The movement of different pins changes the environment, making the changes observable by other agents. This is similar to how the independent decisions of the generating units in this paper affect the frequency of the microgrid, causing environmental changes that in turn affect the observations and decisions of other agents. This distributed decision-making theory replaces the controller and distributor combination structure in the traditional load frequency control (LFC) structure. Each agent outputs the current generating power command of the generating unit, and i agents participate in the decision-making process, much like the i pins of a starfish. In online applications, the real-time monitoring system for each unit calculates and stores the microgrid's frequency deviation (f) and the output of each unit, which are then fed into the SFD-LFC system. The agent in the DDMR-MADDPG algorithm calculates, based on the current microgrid state, the appropriate power generation command to be issued for each unit and the LFC power generation command for units not included in the statistics. The agent's control interval is 4 seconds. This agent addresses the coordination problem between the LFC controller and distributor by using frequency deviation and power generation cost as multiple objectives.

[0150] In online applications, each agent can independently decide to output its own unit's power generation command by locally collecting and receiving the microgrid's frequency and output status and importing them into the strategy function. Each agent only needs to make a globally optimal strategy based on its own observed status, without needing to know the global status. The units do not need to communicate, which is an advantage that no distributed algorithm or optimal control algorithm has.

[0151] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A fully distributed load frequency control method for islanded microgrids, characterized in that, The method includes the following steps: Step S1, Islanded Microgrid Modeling: Model the islanded microgrid using a Markov decision process, setting each generator set as an agent and initializing the action space and state space; Step S2, Agent pre-learning: The agent is trained in a coordinated manner using the principle of centralized training to obtain a distributed global optimal coordination strategy; Step S3, Load Frequency Control: Each distributed generator unit locally collects the frequency status and output status of the microgrid, and uses a distributed execution strategy to decide and output its own generator power command to perform load frequency control (LFC). This also includes the structure of Distributed Deep Reinforcement Learning-Distributed Decomposition Multi-Agent Deep Deterministic Policy Gradient Algorithm (DDMR-MADDPG): Parallel systems: Each parallel system is injected with load disturbances of different amplitudes and durations. The amplitude of the random disturbances is set according to the course learning strategy and increases as the number of episodes increases. Explorers explore the environment of their respective parallel systems and provide the exploration samples obtained to the leader. Each explorer has only one actor network, and different explorers use different exploration principles to conduct distributed exploration of the environment. Demonstrator: Based on the idea of ​​imitation learning, the demonstrator is equipped with a controller or a distributor. The controllers or distributors in different parallel systems have obtained preset performance through parameter tuning. They interact with the environment and obtain demonstration samples, which are then provided to the leader for learning. Leader: This is the agent that will eventually be installed in the load frequency control LFC coordination strategy in the online application; In each step, the leader randomly takes samples from the common experience pool and updates its own network parameters through the classification experience replay mechanism, and then periodically transmits these network parameters to the explorer for the explorer to update; Public Experience Pool: A categorized experience replay strategy is adopted. Each leader has two independent experience pools. The first public experience pool stores samples from explorers, and the second public experience pool stores samples from demonstrators. The leader selects samples from the first public experience pool with a set variable probability ξ, and selects samples from the second public experience pool with 1-ξ. The probability changes as the number of episodes increases. The explorer employs any of the following strategies for action exploration: The explorer uses a greedy strategy, and the corresponding exploration action expression is: （8） In the formula, It is the first l The actions of an explorer, It is the first l The strategy function of each explorer It is a random action; The explorer employs an OU noise detection strategy, and the corresponding detection action expression is: （9） In the formula, It is the first j The actions of an explorer, It is the first j The strategy function of each explorer This is OU noise; The explorer employs a Gaussian noise detection strategy, and the corresponding exploration action expression is: （10） In the formula, It is the first m The actions of an explorer, It is the first m The strategy function of each explorer It is Gaussian noise.

2. The fully distributed load frequency control method for islanded microgrids according to claim 1, characterized in that, Step S1 specifically includes: 1) Construct a multi-agent reinforcement learning stochastic game framework based on Markov decision processes. ,in, This represents the state of an isolated microgrid. For the first The actions of an intelligent agent For the first Rewards for each intelligent agent The number of agents. Let be the state transition probability. Discount factor; 2) Determine the action space and state space; 3) Determine the objective function and the comprehensive reward function.

3. The fully distributed load frequency control method for islanded microgrids according to claim 2, characterized in that, The action space and state space are specifically as follows: Action space : will the first Power output command for each generator set As the first Actions of an agent ; state space : , where Δ f For the frequency deviation of islanded microgrids, This is the integral of time over frequency deviation. For the first Increased power output of each generator set yes Time generator set i The size of the power generation command.

4. The fully distributed load frequency control method for islanded microgrids according to claim 3, characterized in that, The determination of the objective function and the comprehensive reward function are specifically as follows: Objective function: The objective function is to minimize the sum of the cumulative total frequency deviation and the cumulative total power generation cost. The expression is as follows: （1） In the formula, , , These are the first two numbers after considering power disturbances. i The dynamic coefficients of the power generation cost of each generator set at various times. It is the first i Increased power output of each generator set It's the cost of electricity generation. T It is the total operating cycle. The number of generator sets; The comprehensive reward function considers the absolute value of the frequency deviation and the total power generation cost, and incorporates a control penalty term to guide the agent towards the optimal frequency. The expression for the comprehensive reward function is: （2） （3） In the formula, r For the reward function value, For frequency deviation, For the first i The power generation cost of a single generator set , These are the corresponding weight coefficients. B For the penalty function, To set the frequency deviation threshold.

5. A fully distributed load frequency control method for islanded microgrids according to claim 3, characterized in that, The state transition of the isolated microgrid is performed using a joint strategy, expressed as follows: （4） In the formula, For the first i The policy function of each agent A i For the first i The actions of an intelligent agent S This represents the state of an isolated microgrid. The number of agents; The reward value expression for each agent is: （5） In the formula, It is the first i The agent in the th... t Rewards during the walk It is the first i The agent in the th... t The movement during walking; , These are the agent's state value and action value, respectively. The Bellman equations for the state-value function and the state-action-value function in a multi-agent system are as follows: （6） （7） In the formula, It is the first i An agent is in state The following action, It is the first i The mathematical expectation of an agent For islanded microgrids t The state at +1 step For islanded microgrids t The combined action at +1 step, For the first i The Q-value function of an agent This is the discount factor.

6. The fully distributed load frequency control method for islanded microgrids according to claim 5, characterized in that, In step S2, the agent is trained in a coordinated manner using the principle of centralized training. Specifically, the agent is trained in a centralized and coordinated manner using the Distributed Deep Reinforcement Learning-Distributed Decomposition Multi-role Multi-agent Deep Deterministic Policy Gradient Algorithm DDMR-MADDPG.

7. A fully distributed load frequency control method for islanded microgrids according to claim 6, characterized in that, The Distributed Deep Reinforcement Learning-Distributed Decomposition Multi-Agent Deep Deterministic Policy Gradient Algorithm (DDMR-MADDPG) employs a multi-agent deep reinforcement learning model based on an actor-critic framework. The actor network outputs actions based on the current observation state, and the critic network calculates the Q-value based on the state-action pair and feeds the calculated Q-value back to the actor network to adjust the action policy.

8. A fully distributed load frequency control method for islanded microgrids according to claim 1, characterized in that, The agent pre-learning process specifically includes the following sub-steps: 1) Optimize the demonstrator's controller and allocator, and initialize the parameters of the leader and explorer; 2) Initialize the episodes; 3) Initialize the exploration principle and obtain the initial state; 4) Select a random perturbation for each parallel system, calculate the explorer action for each system, and output the action for each demonstrator based on its principle; 5) Based on the classification experience, the samples are stored in the first public experience pool or the second public experience pool. Each leader draws samples from the first public experience pool by setting a probability ξ, or draws samples from the second public experience pool by 1-ξ. 6) Each leader determines the target value for each sample in the mini-batch, calculates the loss function, and updates its own parameters according to the policy gradient; 7) The leader updates the target network and distributes parameters to the explorers; 8) Determine if the current time has reached the set maximum step size. If not, increment the step size by 1 and go to step 4; otherwise, go to step 9. 9) Determine if the current episode count has reached the set number of episodes. If not, increment the episode count by 1 and proceed to step 3; otherwise, proceed to step 10. 10) Output the training results.

Images