Main and distribution network cooperative load frequency control method based on multi-agent reinforcement learning

Abstract

The invention discloses a main and distribution network cooperative load frequency control method based on multi-agent reinforcement learning, and is suitable for power system frequency adjustment. According to the method, a photovoltaic generator and energy storage battery equipment at a distribution network end are controlled through a maximum entropy heterogeneous multi-agent reinforcement learning algorithm, active power is injected into a main power grid through a distribution network access node of the main power grid, and dynamic adjustment of the frequency of the main power grid is achieved. The intelligent agent continuously interacts with the dynamic environment to continuously optimize the control strategy of the intelligent agent so as to achieve the purpose of minimizing the frequency deviation of the system. During actual operation, the intelligent agent makes a decision autonomously according to the system state monitored in real time, cooperative control over the distributed photovoltaic generator and the energy storage battery at the distribution network end is achieved when the main network breaks down, and the stability and reliability of the power grid frequency in the complex environment are guaranteed. The method provided by the invention has good adaptivity and expandability, can effectively deal with challenges such as load change and line fault in a power grid, and improves the adjustment capability of new energy power generation equipment.

Description

Technical Field

[0001] The present invention relates to the technical field of power system frequency control, and in particular to a main distribution network collaborative load frequency control method based on multi-agent reinforcement learning. Background Art

[0002] Load frequency control is a key technology used in modern power systems to stabilize frequency fluctuations and manage power exchange on interconnection lines between different regions. With the large-scale integration of distributed energy resources and the reduction of system inertia, modern power systems face severe challenges such as increased complexity, uncertainty, and volatility. Traditional load frequency control methods usually use proportional-integral controllers to suppress frequency oscillations in power systems, which makes it difficult to achieve adaptive control in complex and time-varying systems. To address the above problems, load frequency control methods based on deep reinforcement learning have attracted the attention of researchers due to their model-free nature and powerful nonlinear expression capabilities. The "model-free" feature of deep reinforcement learning algorithms has significant advantages in solving complex, nonlinear, and rapidly changing problems in power grids. Deep reinforcement learning-based controllers do not require explicit modeling and learn the dynamic characteristics of the system through continuous trial and error.

[0003] However, while numerous studies have used deep reinforcement learning methods to address load frequency control, few have fully considered the heterogeneous characteristics of renewable energy resources and energy storage devices, as well as the coordination issues of the main distribution network, during the coordinated optimization process. Different types of power generation equipment (such as photovoltaic generators and energy storage batteries) are uniformly represented as a neural network, whose output is a unified regional control error signal rather than a specific action for each device. This approach makes the controller vulnerable to attack, fails to fully utilize the regulation capabilities of various resources, and ignores the impact of meteorological factors on renewable energy power generation equipment. Summary of the Invention

[0004] In order to solve the problems existing in the above-mentioned prior art, the purpose of the present invention is to propose a main distribution network collaborative load frequency control method based on multi-agent reinforcement learning, which can effectively improve the performance of load frequency control, enhance the ability of distributed new energy equipment to participate in system frequency regulation, and provide an effective solution for the optimized operation of the power system.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The main distribution network coordinated load frequency control method based on multi-agent reinforcement learning controls the heterogeneous distributed renewable energy power generation equipment in the distribution network through a heterogeneous multi-agent reinforcement learning algorithm. Active power is injected into the main grid through the distribution network access node of the main grid to achieve frequency regulation of the main grid. The heterogeneous distributed renewable energy power generation equipment includes photovoltaic generators and energy storage batteries. The specific steps are as follows:

[0007] Modeling the state space and action space of a multi-agent reinforcement learning environment: Each photovoltaic generator or energy storage battery is modeled as an agent, and the load frequency control problem is optimized into a multi-agent reinforcement learning model. The observation space and action space of each agent are defined. The observation space of each agent includes the active output and speed of the main grid synchronous generator, the load size of the load node, the frequency of the main grid distribution network access node, the maximum and actual output of each distribution network photovoltaic generator, and the actual output of the distribution network energy storage battery; the action space includes the charge and discharge status of the distribution network energy storage battery and the output size of the distribution network photovoltaic generator;

[0008] Design a reward function based on the control objective: The overall goal of load frequency control is to reduce the frequency deviation of the main grid system and maintain the stability of the main grid system frequency by coordinating the output of distribution network photovoltaic generators and energy storage batteries. The reward function is designed based on the load frequency control objective, comprehensively considering frequency deviation, frequency limit violations, equipment utilization, and smoothing the penalty for photovoltaic generator output. The hierarchical analysis method is used to optimize the configuration of the weights of the reward function.

[0009] Environment initialization and experience collection: Initialize the agent strategy network, coordinate the control of photovoltaic generators and energy storage batteries, adjust their active power output, and achieve the goal of reducing the frequency deviation of the distribution network access node and maintaining the frequency stability of the main grid system under the conditions of variable photovoltaic generator irradiance and main grid failure. At the same time, the interaction data between the agent and the environment is recorded and stored in the experience pool.

[0010] The sequence selector outputs the order in which the agents are updated: Each time the agent is updated, data of the same batch size is sampled from the experience pool. The difference in frequency change of the main power grid system before and after the agent's action is used as the input of the sequence selector based on the deep neural network, and the importance of the agent is dynamically output;

[0011] Agent control strategy update: Sort the importance of the agents output by the sequence selector, and prioritize updating the control strategies of agents with higher importance to improve the performance and accuracy of the control algorithm. Update the agent control strategies sequentially based on the interaction data sampled from the experience pool and the update sequence. Repeat the interaction and update iterative process to continuously optimize the agent control strategy.

[0012] Sequence selector parameter update: Sampling feedback data from the agent and the environment at different stages of training to update the sequence selector neural network parameters to improve the sequence selector output accuracy;

[0013] Real-time control and decision-making: In actual operation, each intelligent agent makes autonomous decisions based on its own local observations, stabilizing the main grid frequency and reducing the frequency deviation of the main grid system when facing fluctuations in photovoltaic generator output and main grid failures caused by meteorological factors.

[0014] Preferably, the heterogeneous multi-agent reinforcement learning algorithm adopts maximum entropy heterogeneous multi-agent reinforcement learning, and through maximum entropy regularization, encourages multiple agents to conduct more exploration during the training process, thereby avoiding the control strategy of heterogeneous distributed new energy power generation equipment from falling into a local optimal solution.

[0015] Preferably, the sequence selector uses a deep neural network to approximate the importance of the intelligent agent, uses the frequency deviation of the main power grid system before and after the intelligent agent action as the input of the neural network, and outputs the importance of each device for dynamically adjusting the update order of the intelligent agent.

[0016] Preferably, the output of the photovoltaic generator is affected by irradiance factors, and the active power output is dynamically adjusted below the maximum output power according to frequency regulation requirements to maintain the stability of the main power grid system frequency.

[0017] Preferably, the energy storage battery adjusts its charge and discharge state to compensate for the output of the photovoltaic generator while ensuring the priority output of the photovoltaic generator, thereby maintaining the stability of the frequency of the main power grid system.

[0018] This paper proposes a main distribution network coordinated load frequency control method based on multi-agent reinforcement learning, which has the following beneficial effects:

[0019] Modeling the load frequency control problem as a heterogeneous multi-agent reinforcement learning problem can effectively describe the dynamic characteristics of various heterogeneous devices in the coordinated regulation of the main and distribution networks, thereby improving the control performance.

[0020] Using a deep neural network, the changes in observed quantities before and after an agent's action are used as network input and output as agent importance. When updating the controller, agents with high control importance are prioritized, enhancing the agents' ability to learn the dynamic characteristics of the power grid system.

[0021] During the training process, the performance indicators of the multi-agent reinforcement learning algorithm can balance the exploration and utilization processes through maximum entropy regularization, quickly converge to the optimal solution, and avoid falling into the local optimal solution.

[0022] This method can effectively improve the ability of distributed photovoltaic generators and energy storage batteries at the distribution network end to participate in power system frequency regulation when irradiance is variable and the main grid encounters common faults, providing an effective solution for optimizing power system operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the workflow diagram of the main distribution network collaborative load frequency control method based on multi-agent reinforcement learning. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] like Figure 1 As shown, the present invention provides a method for coordinated load frequency control of the main distribution network based on multi-agent reinforcement learning. This method uses a maximum entropy heterogeneous multi-agent reinforcement learning algorithm to control heterogeneous distributed photovoltaic generators and energy storage batteries in the distribution network, achieving frequency regulation of the main grid system under conditions of variable irradiance and common main grid faults. The following are the specific implementation steps:

[0026] Step S1: Modeling the state space and action space of heterogeneous multi-agent reinforcement learning environment

[0027] When modeling the load frequency control problem as a multi-agent reinforcement learning model, the first consideration is the observations required by each agent. These observations include the active power output of the main grid synchronous generators, the synchronous generator speed, the active power consumption of each load node in the main grid, the main grid system frequency, the maximum and actual output active power of the distribution-side photovoltaic generators, and the active power output of the distribution-side energy storage batteries. Each state variable uses the per-unit value of its corresponding physical quantity or is normalized to a range of 0 to 1 or -1 to 1. All states are concatenated to form the environment state space, as shown below:

[0028]

[0029] states=(state1,state2,state3,state4,state5,state6)

[0030] Among them, Concat means concatenating the state vectors; and Respectively represent the G i The active output and speed of the main grid-side synchronous generator at time t, N gen Indicates the number of synchronous generators in the main grid side system; Indicates the L i Active power consumption of load nodes on the main grid side at time t, N load Indicates the number of system load nodes on the main grid side; f t gsp represents the frequency of the distribution network access node of the main power grid at time t; They represent the PV on the distribution network side at time t i The actual output and maximum output state of each photovoltaic generator, N pv Indicates the number of distributed photovoltaic generators on the distribution network side; Indicates the BESS on the distribution network side i The discharge power of the energy storage battery, N bess Indicates the number of energy storage batteries on the distribution network side.

[0031] For the optimal load frequency control problem, the impact of the output of distributed renewable energy devices on the main power grid is considered. Each photovoltaic generator and energy storage battery is defined as a minimum power dispatch unit. Therefore, the joint action space is defined as:

[0032]

[0033] where i and j represent the number of photovoltaic generators (PV) and battery energy storage systems (BESS) on the distribution grid, respectively. For the agent controlling the PV generator, the action is defined as a percentage of its rated maximum power, taking a continuous value between 0 and 1. The actual output of the PV generator is determined by the following formula:

[0034]

[0035] in, represents the actual output of photovoltaic generator i, represents the action of the agent controlling the i-th photovoltaic generator. Indicates the actual maximum output capacity of photovoltaic generator i, irradiance t Represents the irradiance at time t. It can be seen that the maximum output capacity of the photovoltaic generator is affected by the irradiance. represents the output power limit of photovoltaic generator i, represents the maximum rated power of the i-th photovoltaic generator. To adjust the output of the photovoltaic generator to achieve the desired control target. For the intelligent agent related to the energy storage battery, the range of its output action is [-1, 1], and the intelligent agent action represents the multiplication factor of the maximum output capacity of its energy storage battery. The actual output of the jth energy storage battery can be expressed as: in represents the actual output of energy storage battery j, represents the maximum output of energy storage battery j, represents the action of the agent controlling the j-th energy storage battery.

[0036] Step S2: Reward function design

[0037] The control objective of this method is to reduce the frequency deviation of the main power grid system, avoid the main power grid system frequency exceeding the limit, balance the output of distributed new energy equipment in the distribution network, and realize the coordinated control of photovoltaic and storage.

[0038] This invention utilizes a maximum entropy heterogeneous multi-agent reinforcement learning algorithm. Each distributed power generation device, including the distributed photovoltaic generators and energy storage batteries on the distribution network side, is controlled by an independent agent. Each agent aims to optimize its own control strategy, achieving collaboration while maximizing its cumulative control benefits. Using an architecture that combines centralized training with distributed execution, each agent, after training, is able to make independent decisions based on its own observations.

[0039] The reward function designed based on the overall goal of load frequency control is as follows:

[0040] r=-(η f r f +η fc r fc +η b r b +η u r u +η d r d +η c r c )

[0041] The total reward function r of the power system is composed of two types of objectives: First, from the perspective of power grid operation, r f is the stable frequency reward function term, which is used to quantify the degree of main grid frequency deviation, r fc The main grid frequency over-limit reward item is used to constrain whether the main grid frequency exceeds the upper and lower limits to ensure the safety of the grid; secondly, from the perspective of equipment control, r c is the photovoltaic generator output power smoothing term, which represents the control cost caused by smoothing the distributed energy output power on the distribution network side, r u The balanced equipment utilization item is used to measure the usage balance between various distribution network-side devices to avoid equipment overload or uneven utilization.

[0042] In addition, in order to further improve the efficiency and safety of energy scheduling, two penalty terms related to energy storage batteries are designed: r d The penalty term for energy storage battery discharge is used to suppress battery discharge at unnecessary times and encourage efficient use of energy; bIt is a charging restriction item for energy storage batteries, which is used to limit the charging behavior when the frequency of the main grid system is lower than the target value, so as to prevent unreasonable charging operations when the main grid system power is insufficient. Specifically, when the frequency of the main grid system is lower than the set threshold, the intelligent control strategy gives priority to adjusting the output of the photovoltaic generator on the distribution network side to prevent the energy storage battery from absorbing the already limited active power from the main grid. On the contrary, when the frequency of the main grid system rises above the target value due to excess power generation of the photovoltaic generator on the distribution network side, the energy storage battery is stimulated to charge to enhance the buffering capacity of the entire power system against frequency fluctuations. This coordination mechanism helps to achieve the coordinated operation between photovoltaic generators and energy storage batteries, thereby enhancing the stability and regulation efficiency of grid operation. Where η f ,η fc ,η b ,η u ,η d ,η c Respectively represent the hyperparameters corresponding to each reward function sub-item, which are used to represent the weight of each sub-item in the overall reward function r. f +η fc +η b +η u +η d +η c =1. In actual use, users can adjust the hyperparameters of each sub-item of the reward function according to their preferences. The specific meanings of each sub-item of the overall reward function r are shown in the following formula:

[0043]

[0044] In the present invention, f t gsp represents the frequency of the grid supply point (GSP) of the main power grid, f set The rated target frequency of the main grid. The upper and lower limits of frequency fluctuation are and f are set as constraint thresholds to calculate the penalty term for exceeding the safety range. max(a,b) means taking the maximum value of variables a and b. The symbol {·} is used to distinguish different types of device sets, where P t {·},i represents the active power output of the i-th device (such as photovoltaic generator PV or energy storage battery BESS) at time t, P t {·},average Represents the average active power output of all photovoltaic generators or energy storage batteries at time t. The absolute value of the difference between the two represents the difference between the output of the i-th device and the average output of the overall type of equipment, which is used to encourage the balanced output of photovoltaic generators and energy storage batteries. t j represents the active output of energy storage battery j at time t, so rd Indicates the penalty for the energy storage battery that has discharged. c In the expression of represents the active power output of the i-th photovoltaic generator at time t, and It represents its output at the previous moment, and the absolute value of the difference between the two is taken as the reward function term to reduce the volatility of the photovoltaic generator output.

[0045] Step S3: Environment initialization and experience collection

[0046] After completing the environment definition, the policy network and sequence selector network are initialized according to the environment state space and the agent action space. The agent is controlled to execute its strategy or use expert experience, collect interaction data with the environment, and store the data in the experience pool. Each interaction data is stored in the form of (s, a, r, s'), where s is the current state of the environment, a is the joint action performed by the agent, r is the environment feedback reward, and s' is the state of the environment after the action is executed.

[0047] Step S4: Determine the agent update order

[0048] After the experience pool has stored a certain amount of data, a batch of data is sampled from the experience pool to update the agent. The change in the environment state before and after the agent's action, |s'-s|, is used as the input of the sequential selector network, which outputs the importance of different agents.

[0049] Step S5: Update the Sequential Agent Policy Network

[0050] The importance of the agents output by the sequence selector is ranked, prioritizing the update of control strategies for agents with higher importance to improve the performance and accuracy of the control algorithm. The control strategies of the agents are updated sequentially based on the interaction data sampled from the experience pool and the update sequence. This iterative process of interaction and update is repeated to continuously optimize the control strategies of the agents.

[0051] Step S6: Update sequence selector parameters

[0052] Periodically evaluate the comprehensive performance of the agent strategy, and update the sequence selector neural network parameters based on the evaluation cumulative reward r to improve the sequence selector output accuracy, improve algorithm performance, and accelerate convergence.

[0053] Step S7: Real-time control and decision-making

[0054] In actual operation, each agent makes autonomous decisions based on its own sensor-based observations. It dynamically adjusts the output of each device in the face of fluctuations in photovoltaic generator output caused by meteorological factors and routine main grid failures, stabilizing the main grid frequency and reducing system frequency deviations. At the same time, information about the interaction between the agent and the environment is stored in an experience pool for continuous iterative optimization of the agent, thereby continuously improving algorithm performance.

[0055] The present invention is applicable to optimal load frequency control in power systems. This method controls distributed photovoltaic generators and energy storage batteries in the distribution network through the maximum entropy heterogeneous multi-agent reinforcement learning algorithm, and realizes the regulation of the main power grid system frequency under the conditions of variable irradiance and routine faults in the main grid. The multi-agent continuously interacts with the environment and obtains feedback, continuously optimizes its control strategy, and reduces the frequency deviation of the main power grid system, avoids the system frequency exceeding the limit, balances the output of distributed new energy equipment, and realizes the purpose of photovoltaic storage coordinated frequency regulation. In actual operation, each agent makes autonomous decisions based on its own observations, realizes multi-device collaboration, and ensures the stability and reliability of the grid frequency in complex power grid environments such as variable irradiance and routine line faults. The method of the present invention has flexible adaptability and scalability, and can effectively deal with frequency control problems caused by photovoltaic generator output fluctuations, load changes, line breaks and other faults caused by irradiance changes, improve the stability and safety of system operation, and enhance the frequency regulation capability of new energy power generation equipment.

Claims

1. A main distribution network coordinated load frequency control method based on multi-agent reinforcement learning, characterized in that: The method uses a heterogeneous multi-agent reinforcement learning algorithm to control heterogeneous distributed new energy power generation equipment in a distribution network, injects active power into the main grid through the distribution network access node of the main grid, and realizes frequency regulation of the main grid. The heterogeneous distributed new energy power generation equipment includes a photovoltaic generator and an energy storage battery. The specific steps are as follows: Modeling the state space and action space of a multi-agent reinforcement learning environment: Each photovoltaic generator or energy storage battery is modeled as an agent, and the load frequency control problem is optimized into a multi-agent reinforcement learning model. The observation space and action space of each agent are defined. The observation space of each agent includes the active output and speed of the main grid synchronous generator, the load size of the load node, the frequency of the main grid distribution network access node, the maximum and actual output of each distribution network photovoltaic generator, and the actual output of the distribution network energy storage battery; the action space includes the charge and discharge status of the distribution network energy storage battery and the output size of the distribution network photovoltaic generator; Design a reward function based on the control objective: The overall goal of load frequency control is to reduce the frequency deviation of the main grid system and maintain the stability of the main grid system frequency by coordinating the output of distribution network photovoltaic generators and energy storage batteries. The reward function is designed based on the load frequency control objective, comprehensively considering frequency deviation, frequency limit violations, equipment utilization, and smoothing the penalty for photovoltaic generator output. The hierarchical analysis method is used to optimize the configuration of the weights of the reward function. Environment initialization and experience collection: Initialize the agent strategy network, coordinate the control of photovoltaic generators and energy storage batteries, adjust their active power output, and achieve the goal of reducing the frequency deviation of the distribution network access node and maintaining the frequency stability of the main grid system under the conditions of variable photovoltaic generator irradiance and main grid failure. At the same time, the interaction data between the agent and the environment is recorded and stored in the experience pool. The sequence selector outputs the order in which the agents are updated: Each time the agent is updated, data of the same batch size is sampled from the experience pool. The difference in frequency change of the main power grid system before and after the agent's action is used as the input of the sequence selector based on the deep neural network, and the importance of the agent is dynamically output; Agent control strategy update: Sort the importance of the agents output by the sequence selector, and prioritize updating the control strategies of agents with higher importance to improve the performance and accuracy of the control algorithm. Update the agent control strategies sequentially based on the interaction data sampled from the experience pool and the update sequence. Repeat the interaction and update iterative process to continuously optimize the agent control strategy. Sequence selector parameter update: Sampling feedback data from the agent and the environment at different stages of training to update the sequence selector neural network parameters to improve the sequence selector output accuracy; Real-time control and decision-making: In actual operation, each intelligent agent makes autonomous decisions based on its own local observations, stabilizing the main grid frequency and reducing the frequency deviation of the main grid system when facing fluctuations in photovoltaic generator output and main grid failures caused by meteorological factors.

2. The main distribution network coordinated load frequency control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The heterogeneous multi-agent reinforcement learning algorithm adopts maximum entropy heterogeneous multi-agent reinforcement learning. Through maximum entropy regularization, it encourages multi-agents to conduct more exploration during the training process, thereby preventing the control strategy of heterogeneous distributed new energy power generation equipment from falling into a local optimal solution.

3. The main distribution network coordinated load frequency control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The sequence selector uses a deep neural network to approximate the importance of the intelligent agent, uses the frequency deviation of the main power grid system before and after the intelligent agent action as the input of the neural network, and outputs the importance of each device for dynamically adjusting the update order of the intelligent agent.

4. The main distribution network coordinated load frequency control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The output of the photovoltaic generator is affected by the irradiance factor, and the active power output is dynamically adjusted below the maximum output power according to the frequency regulation demand to maintain the stability of the main power grid system frequency.

5. The main distribution network coordinated load frequency control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The energy storage battery adjusts its charge and discharge state to compensate for the output of the photovoltaic generator while ensuring the priority output of the photovoltaic generator, thereby maintaining the stability of the frequency of the main power grid system.

Images