A microgrid frequency recovery and power distribution control method

Abstract

A micro-grid frequency recovery and power distribution control method, comprising the following steps: combining game theory with reinforcement learning to obtain a Nash Q reinforcement learning control method; when the system detects that the micro-grid frequency deviates, introducing droop control to compensate power to recover the frequency; regarding each distributed power supply as an intelligent agent in the system; using the Nash Q reinforcement learning to take the micro-grid frequency recovery and power distribution as learning goals; defining rewards and punishments according to the frequency deviation of each distributed power supply in the droop control; and through the information of the opponent intelligent agent, performing matrix game to find the Nash equilibrium solution of power distribution. The present application realizes the requirements of fast recovery of micro-grid frequency and accurate distribution of power.

Description

Technical Field

[0001] This invention belongs to the field of microgrid secondary control, and specifically relates to a microgrid frequency recovery and power distribution control method based on Nash Q learning. Background Technology

[0002] With the advancement of power grid technology, microgrids are gradually becoming complex and independent systems. They can be connected to the main power grid for coordinated operation or operate as independent systems. When a microgrid is connected to the main grid, its dynamic characteristics are mainly determined by the main grid. When operating in islanded mode, due to the lack of main grid support and the complexity of the control structure and the diversity of control objectives, some scholars have proposed hierarchical control theory to systematically regulate and control microgrids. Typically, the primary control in hierarchical control of microgrids is droop control to achieve autonomous power distribution among distributed power sources. However, droop control has a differential regulation; when the load changes, the system cannot guarantee that the frequency remains stable at the rated value. Therefore, a second-level control layer is needed to reduce or eliminate this deviation.

[0003] Traditional secondary control mostly employs a centralized approach. While this avoids the problems of decentralized frequency regulation, it suffers from poor structural scalability, reliance on a central controller, and the need for complex communication networks. With the increasing attention given to the distributed nature of Multi-Agent Systems (MAS), distributed secondary optimization control has proven to be a feasible solution for recovering droop bias. Regarding the frequency recovery problem, some literature proposes using a controller with an R(λ) learning algorithm based on a flat-reward model to optimize the power output of different units within the microgrid and the system frequency. Furthermore, other literature proposes a control method for microgrid frequency regulation using a reinforcement learning Q-learning algorithm, which effectively stabilizes the microgrid frequency at its rated value by compensating for the power deficit in secondary frequency regulation.

[0004] While the above studies were able to effectively recover the frequency deviation, they disrupted the original power distribution balance of the droop control, thus requiring further power distribution recovery. Summary of the Invention

[0005] The purpose of this invention is to solve the problem of rapid recovery of the original power distribution under droop control in microgrids when the secondary control frequency is restored. A microgrid frequency restoration and power distribution control method based on Nash Q learning is proposed.

[0006] This invention is achieved through the following technical solutions.

[0007] The microgrid frequency recovery and power distribution control method of the present invention is characterized by comprising the following steps:

[0008] (1) When the microgrid frequency deviates or the power distribution is unbalanced, a droop control method is introduced to perform power compensation in order to restore the frequency.

[0009] (2) The reinforcement learning Q-learning method is adopted to adaptively guide the learning of the target in the microgrid environment, generate a set of optimal control strategies, and realize the rapid recovery of the microgrid frequency.

[0010] (3) Based on reinforcement learning, game theory is introduced to optimize the control strategy, so as to achieve Nash equilibrium of the joint behavior of multiple agents and realize the optimal control of power distribution among agents.

[0011] Furthermore, the method for adjusting the microgrid frequency deviation or power distribution imbalance described in step (1) by introducing droop control is as follows:

[0012] (1-1) When the microgrid frequency deviates, droop control is introduced for adjustment. Under the premise of an islanded microgrid, the active power-frequency droop control equation for the parallel inverter is:

[0013]

[0014] In the formula, f i Let be the output frequency of the i-th inverter; P is the rated frequency of the i-th inverter; i Let be the output power of the i-th inverter. Let m be the rated power of the i-th inverter. i Let be the droop coefficient of the i-th inverter. As can be seen from equation (1), when a frequency deviation occurs, power compensation can be performed to eliminate the frequency deviation.

[0015] (1-2) When the power distribution in a microgrid is unbalanced, since the inverters are connected in parallel, the output frequencies of each inverter are also equal. By setting appropriate droop coefficients and rated power, the product of the droop coefficients and rated power of each parallel inverter is equal. Based on this, we have:

[0016]

[0017] In the formula, P i Let m be the output power of the i-th inverter. i Let C be the droop coefficient of the i-th inverter, and C be a constant. As can be seen from equation (2), the output power of the inverter is inversely proportional to the droop coefficient. Therefore, the output power can be proportionally distributed by adjusting the droop coefficient.

[0018] Furthermore, the method described in step (2) for using reinforcement learning Q-learning to adaptively guide learning of the target in the microgrid environment, generating an optimal control strategy, and achieving rapid frequency recovery of the microgrid is as follows:

[0019] (2-1) Use an action-value function to record the state and action, and the value function Q π The method for calculating (s,α) is as follows:

[0020]

[0021] In the formula, Q π (s,α) is the expected value of the reward obtained by taking action α in state s under policy π; r(s,α) is the immediate reward obtained by taking action α in state s. γ is the value function corresponding to the next state s' in strategy π; γ is the decay rate, used to calculate the cumulative reward from state s' to the end of the round.

[0022] (2-2) Update the value function table in each learning cycle, and form an optimal control strategy through iterative updates. The iteration principle is as follows:

[0023]

[0024] In the formula, s t α t Let Q represent the state and action in the t-th iteration, respectively; t (s t ,α t ) represents the value of the t-th iteration; r t (s t ,α t ) represents state s t Take action α t The reward; s t Take action α in the state t Reaching the next state s t+1 The largest possible Q value; α t Let α be the learning rate. t A smaller value indicates greater emphasis on previous training results; γ is the decay rate, and a larger γ indicates greater emphasis on the role of the next step.

[0025] (2-3) A greedy strategy is used to select actions. In the current state, the agent will select the action with the highest Q value to execute. The action selection satisfies the following principles:

[0026]

[0027] In the formula, the function argmax(f(x)) represents the variable x corresponding to the maximum value of f(x). Let α be the action value function for taking action α in state s under policy π after the k-th iteration of learning. Regardless of the agent's state, the optimal action policy is to select the action with the maximum Q value based on the learned g×h order Q matrix according to the current state.

[0028] Furthermore, the method described in step (3) for optimizing the control strategy by introducing game theory based on reinforcement learning to achieve Nash equilibrium of the joint behavior of multiple agents and realize optimal control of power distribution among agents is as follows:

[0029] (3-1) Make all agents adopt the Nash equilibrium strategy. The Nash Q function of agent i is defined as the joint behavior for state Si. The sum of the current reward and future reward of agent i is:

[0030]

[0031] In the formula, (π1,…,π) n (This is a joint strategy) For agent i to perform joint behavior (α1,…,α) in state s n The rewards obtained, For state s through joint behavior (α1,…,α) n The probability of transitioning to state s', V i (s,π1,…,π n Let be the state-value function of agent i in state s' under the joint policy. The total discounted reward for all other agents when performing the Nash equilibrium policy under the current state.

[0032] (3-2) The Nash equilibrium strategy that maximizes reward is obtained through iterative updates. The iterative updates of the Nash Q-learning algorithm satisfy:

[0033]

[0034] In the formula, It is the Nash equilibrium point of agent i in the new state. Is agent i in state s? t and joint behavior (α1,…,α) n Immediate reward under α t γ is the learning rate, and γ is the decay rate.

[0035] (3-3) The Lemke-Howson algorithm is used for each action selection and Nash Q value calculation. When the frequency deviation and power allocation meet the requirements, the iteration stops and the optimal control strategy is generated.

[0036] Features and beneficial effects of the present invention:

[0037] (1) By adopting hierarchical control, the frequency can be kept stable at the rated value when the microgrid load changes.

[0038] (2) The use of distributed secondary optimization control effectively avoids the problems of poor scalability of centralized structure, reliance on central controller, and complex communication.

[0039] (3) The combination of multi-agent and reinforcement learning enables each agent to autonomously adjust according to the environment and obtain the best control strategy.

[0040] (4) The introduction of game theory and its application to the secondary control of microgrids effectively compensates for the shortcomings of traditional reinforcement learning in multi-agent environments.

[0041] (5) After reinforcement learning training, the microgrid can maintain frequency stability and power distribution balance when the load changes. Detailed Implementation

[0042] The present invention will be further described in detail through the following specific embodiments.

[0043] This invention addresses the issue of microgrid frequency recovery control while maintaining balanced power distribution when frequency deviation or power imbalance occurs. It utilizes reinforcement learning and game theory to train multiple agents to reach Nash equilibrium, thereby achieving frequency recovery control.

[0044] The frequency recovery method implemented in this invention includes the following steps:

[0045] 1. Set the input environmental variable as frequency deviation ∆f, which can be divided into a series of discrete intervals corresponding to the environmental state set S: Here, the environmental state set is divided into: {(-∞, -0.5),(-0.5, -0.3),(-0.3, -0.1),(-0.1, -0.005),(-0.005, 0.005),(0.005, 0.1),(0.1, 0.3),(0.3, 0.5),(0.5, +∞)}.

[0046] 2. Set the action variable as the adjustment power ∆P, which can be divided into a series of discrete numbers. Set the action set A: {(α1,…,α...} nThe expression is {-0.2, -0.16, -0.12, -0.06, -0.02, -0.01, -0.005, -0.001, 0.001, 0.005, 0.01, 0.02, 0.06, 0.12, 0.16, 0.2}.

[0047] 3. Based on the control objective and the state set, the reward function R is set as follows:

[0048]

[0049] 4. The agent in state ∆f t Next, select action α according to the greedy strategy. t The maximum maxQ(s) is obtained by observing the next state and receiving real-time feedback. t+1 The next step will be to execute maxQ(s) value. t+1 The action corresponding to ) is updated according to formula (4) to update the Q value.

[0050] 5. Update the state, return to step 4, and continue the loop until ∆f∈(50±0.005).

[0051] The power allocation method implemented in this invention includes the following steps:

[0052] 1. Set the input state variable as the output power P of each agent. i The output action variable is the compensation power ∆P i ∆P i It can be divided into {600, 400, 200, 100, 50, 25, 0, -25, -50, -100, -200, -400, -600}.

[0053] 2. Two reward functions are defined based on the control objective and the input state, representing the reward given for taking an action in response to the state:

[0054]

[0055] In the formula, P i The output active power of each inverter; This represents the rated active power of each inverter. A penalty is imposed when the output power exceeds the rated power or falls below zero.

[0056]

[0057] In the formula, m i P is the droop factor of the i-th inverter; iLet be the output power of the i-th inverter. The closer the product of the droop factor and the output power of each inverter is to each other, the smaller the penalty; conversely, the penalty increases as the product of the droop factor and the output power of each inverter approaches the target value. A reward is given when the ratio of the product of the droop factor and the output power of each inverter reaches within 0.005 above or below the target value 1.

[0058] 3. The agent, according to the search method, is in state P i The following selects action α from the set of compensated power actions. i Observe the next state P i+1 To receive rewards and observe other agents in P i Actions performed in a given state and the rewards obtained therefrom.

[0059] 4. Update the Q value, where the Q value satisfies:

[0060]

[0061] 5. Update status P i The Nash Q-value and strategy are determined. The Lemke-Howson algorithm is used to solve for the Nash Q-value, which is suitable for two-player, two-matrix games. Return to step 3 and proceed to the next state iteration until the learning ends when m1P1 / m2P2 = (1 ± 0.005).

Claims

1. A microgrid frequency recovery and power distribution control method, characterized in that, Includes the following steps: (1) When the microgrid frequency deviates or the power distribution is unbalanced, a droop control method is introduced to perform power compensation in order to restore the frequency; (2) The reinforcement learning Q-learning method is adopted to adaptively guide the learning of the target in the microgrid environment, generate a set of optimal control strategies, and realize the rapid recovery of the microgrid frequency; (3) Based on reinforcement learning, game theory is introduced to optimize the control strategy, so as to achieve Nash equilibrium of the joint behavior of multiple agents and realize the optimal control of power distribution among agents. When the microgrid frequency deviates or the power distribution is unbalanced, the method of introducing droop control for adjustment in step (1) is as follows: (1-1) When the microgrid frequency deviates, droop control is introduced for adjustment; under the premise of an islanded microgrid, the active power-frequency droop control equation for the parallel inverter is: （1） In the formula, f i Let be the output frequency of the i-th inverter; P is the rated frequency of the i-th inverter; i Let be the output power of the i-th inverter. Let m be the rated power of the i-th inverter. i Let be the droop coefficient of the i-th inverter; as shown in equation (1), when the frequency deviates, power compensation can be performed to eliminate the frequency deviation. (1-2) When the power distribution in a microgrid is unbalanced, since the inverters are connected in parallel, the output frequencies of each inverter are also equal. By setting appropriate droop coefficients and rated power, the product of the droop coefficients and rated power of each parallel inverter is equal. Based on this, we have: （2） In the formula, P i Let m be the output power of the i-th inverter. i Let C be the droop coefficient of the i-th inverter, and C is a constant. As can be seen from equation (2), the output power of the inverter is inversely proportional to the droop coefficient. Therefore, the output power can be proportionally distributed by adjusting the droop coefficient. The method described in step (2) for using reinforcement learning Q-learning to adaptively guide learning of the target in the microgrid environment, generating an optimal control strategy, and achieving rapid frequency recovery of the microgrid is as follows: (2-1) Use an action-value function to record the state and action, and the value function Q π The method for calculating (s,α) is as follows: （3） In the formula, Q π (s,α) represents the expected reward for taking action a in state s under policy π. It is the immediate reward obtained by taking action a in state s; γ is the value function corresponding to the next state s' in strategy π; γ is the decay rate, used to calculate the cumulative reward from state s' to the end of the round; (2-2) Update the value function table in each learning cycle, and form an optimal control strategy through iterative updates. The iteration principle is as follows: （4） In the formula, s t α t Let these represent the state and action in the t-th iteration, respectively. This represents the value of the t-th iteration; Represents state s t Take action α t The reward; s t Take action α in the state t Reaching the next state s t+1 The largest possible Q value; α t Let α be the learning rate. t A smaller value indicates that previous training results are given more importance; γ is the decay rate, and a larger γ indicates that the role of the next step is given more importance. (2-3) A greedy strategy is used to select actions. In the current state, the agent will select the action with the highest Q value to execute. The action selection satisfies the following principles: （5） In the formula, the function argmax(f(x)) represents the variable x corresponding to the maximum value of f(x). Let α be the action value function for taking action α in state s under policy π after the k-th iteration of learning. Regardless of the state of the agent, the optimal action policy is to select the action corresponding to the maximum Q value based on the learned g×h order Q matrix according to the current state. The method described in step (3), which involves introducing game theory to optimize the control strategy based on reinforcement learning, to achieve Nash equilibrium of the joint behavior of multiple agents and realize optimal control of power distribution among agents, is as follows: (3-1) Make all agents adopt the Nash equilibrium strategy. The Nash Q function of agent i is defined as the joint behavior for state Si. The sum of the current reward and future reward of agent i is: （6） In the formula, (π1,…,π) n (This is a joint strategy) For agent i to perform joint behavior (α1,…,α) in state s n The rewards obtained, For state s, through joint behavior (α1,…,α) n The probability of transitioning to state s', V i (s,π1,…,π n Let be the state-value function of agent i in state s' under the joint policy. The total discounted reward for all other agents when executing the Nash equilibrium policy under the current state; (3-2) The Nash equilibrium strategy that maximizes reward is obtained through iterative updates. The iterative updates of the Nash Q-learning algorithm satisfy: （7） In the formula, It is the Nash equilibrium point of agent i in the new state. Is agent i in state s? t and joint behavior (α1,…,α) n Immediate reward under α t γ is the learning rate, and γ is the decay rate; (3-3) The Lemke-Howson algorithm is used for each action selection and Nash Q value calculation. When the frequency deviation and power allocation meet the requirements, the iteration stops and the optimal control strategy is generated.

Images