Multi-microgrid distributed control method and device based on multi-agent reinforcement learning

Abstract

The invention relates to a multi-microgrid distributed control method and device based on multi-agent reinforcement learning. The method comprises the following steps: modeling a power network system, and modeling a multi-microgrid forming process into a distributed partially observable Markov decision process; deploying a multi-agent reinforcement learning algorithm according to a partial observable Markov decision process, performing environment interaction sampling information between each agent and the power network system, and storing the sampling information into respective experience playback pools; based on a DQN algorithm, training a strategy network corresponding to each agent to learn an optimal behavior by sampling experience samples in an experience replay pool according to an action mask technology and an experience screening technology; and utilizing the trained strategy network model to respectively control the switch action of each region of the power system to form a micro-grid, and recovering load power supply. By adopting the method, the toughness of the power system can be effectively improved, and rapid formation and recovery of the multi-microgrid can be realized.

Description

Technical Field

[0001] The present application relates to the field of smart grid optimization technology, and in particular to a multi-microgrid distributed control method and device based on multi-agent reinforcement learning. Background Art

[0002] The traditional power system adopts a top-down centralized energy production model, but with the decentralization and digitalization of energy production, the power system is gradually shifting to a distributed and diversified microgrid model. By integrating distributed energy resources, microgrids can provide power supply to critical infrastructure after a disaster, thereby enhancing the resilience of the power system. Especially in extreme disaster events, it has become an effective strategy to restore critical loads by forming multiple microgrids. The problem of forming multiple microgrids faces many challenges. First, the decision space is too large, resulting in high computational complexity, especially in large-scale power systems, where the number of decision variables grows exponentially. Second, most of the existing research methods are based on deterministic scenarios, while actual disaster scenarios are often uncertain and dynamic, making it difficult to effectively deal with them through traditional optimization methods. In addition, traditional centralized control methods rely on global information, which not only poses privacy leakage and network security risks, but also makes it difficult to operate efficiently in large-scale systems.

[0003] Existing technologies have problems such as high computational complexity, poor adaptability, and low training efficiency when dealing with the formation of multiple microgrids. Therefore, a new method is urgently needed to effectively improve the resilience of the power system and realize the rapid formation and recovery of multiple microgrids under the condition of incomplete measurement and control information. Summary of the invention

[0004] Based on this, it is necessary to provide a multi-microgrid distributed control method and device based on multi-agent reinforcement learning, which can effectively improve the resilience of the power system and realize the rapid formation and recovery of multiple microgrids in response to the above-mentioned technical problems.

[0005] A multi-microgrid distributed control method based on multi-agent reinforcement learning, the method comprising: Modeling the power network system, the multi-microgrid formation process is modeled as a distributed partially observable Markov decision process; The multi-agent reinforcement learning algorithm is deployed based on the partially observable Markov decision process. Each agent interacts with the power grid system through the connection-only mechanism and the self-organizing network mechanism to sample information and then stores it in its own experience replay pool. Based on the DQN algorithm, each agent samples the experience samples in the experience replay pool and trains its corresponding strategy network to learn the optimal behavior based on the action masking technology and experience screening technology. The trained strategy network model is used to control the switch actions in each area of ​​the power system to form a microgrid and restore power supply to the load.

[0006] A multi-microgrid distributed control device based on multi-agent reinforcement learning, the device comprising: The system modeling module is used to model the power network system and model the multi-microgrid formation process as a distributed partially observable Markov decision process; The model training module is used to deploy a multi-agent reinforcement learning algorithm based on a partially observable Markov decision process. Each agent interacts with the power grid system through a connection-only mechanism and a self-organizing network mechanism to sample information and store it in its own experience replay pool. Based on the DQN algorithm, each agent samples experience samples from the experience replay pool and trains its corresponding strategy network to learn the optimal behavior based on action masking technology and experience screening technology. The multi-microgrid distributed control module is used to use the trained strategy network model to control the switching actions in each area of ​​the power system to form a microgrid and restore power supply to the load.

[0007] The above-mentioned multi-microgrid distributed control method and device based on multi-agent reinforcement learning first models the problem as a multi-agent partially observable Markov decision process. Each agent only relies on local observation information for decision-making, without the need for global data concentration, which solves the problem of communication interruption and incomplete measurement in disasters, improves the system's adaptability to dynamic uncertainty scenarios, and avoids the privacy leakage and single point failure risks of centralized control. The connection-only mechanism is used to limit the communication range, reduce the amount of data transmission and link dependence, reduce privacy risks and improve anti-destruction capabilities; the self-organizing network mechanism supports dynamic reconstruction of communication topology, automatically maintains local coordination when disasters cause link failures, ensures that control instructions are not interrupted, and enhances disaster recovery resilience. Finally, according to the action masking technology, the physical constraints of the power system are embedded, invalid actions are shielded, and the decision space is compressed from the exponential level to the polynomial level, greatly reducing the computational complexity; the experience screening technology prioritizes the retention of high-value disaster recovery samples, alleviates the problem of scarcity of disaster data, improves the learning efficiency of agents for extreme scenarios, and achieves minute-level rapid recovery. This application supports regional parallel decision-making through a multi-agent distributed architecture to avoid centralized computing delays and is suitable for distribution networks with hundreds of nodes; the DQN-based policy network captures nonlinear relationships through end-to-end learning, realizes joint optimization of topology optimization and energy scheduling, and improves the recovery rate of key loads and the utilization rate of renewable energy. Each agent operates independently and collaborates through wired communication. Even if some areas are isolated from the main grid, they can still autonomously optimize strategies based on local experience to ensure the continuous and stable operation of isolated microgrids, and have strong robustness against extreme events such as multiple faults and network attacks. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1A schematic diagram of a flow chart of a multi-microgrid distributed control method based on multi-agent reinforcement learning in one embodiment; Figure 2 A schematic diagram of the interaction between multiple intelligent agents and a power grid environment in one embodiment; Figure 3 A schematic diagram of a specific decision-making process of a single agent in one embodiment; Figure 4 A schematic diagram of an action masking technique in another embodiment; Figure 5 A schematic diagram of an experience screening process in an embodiment; Figure 6 A schematic diagram of a solution result for forming a multi-microgrid in one embodiment; Figure 7 It is a structural block diagram of a multi-microgrid distributed control device based on multi-agent reinforcement learning in one embodiment. DETAILED DESCRIPTION

[0009] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0010] In one embodiment, Figure 1 As shown, a multi-microgrid distributed control method based on multi-agent reinforcement learning is provided, comprising the following steps: Step 102 , modeling the power network system, and modeling the multi-microgrid formation process as a distributed partially observable Markov decision process.

[0011] The power network system model includes busbar nodes and Feeder lines, represented by ,in represents the busbar, Represents the feeder line; at each node There is a consumption of active power and reactive power The load has priority and is expressed in terms of weight. With the help of information and communication technology, each line and load in the system is equipped with a controllable switch that can be connected and disconnected through remote control. When the large power grid loses service due to extreme natural disasters, multiple microgrids are formed by distributed generators installed at certain nodes to power critical loads. Each microgrid is powered by a distributed generator.

[0012] The multi-microgrid formation process is modeled as a distributed partially observable Markov decision process, which contains the state information of the environment. , the agent’s observation information , the action space of the agent , the reward function , state transition process .

[0013] Specifically, Contains the characteristic data of all nodes and lines in the power grid system, and each agent observes yes A subset of ; Assume a total of Agent, Observation of an agent , and are the active power and reactive power output by distributed generators, and Represent the active and reactive power consumed by the load, respectively. It is a 0-1 type variable representing the connection status of the load, 0 represents disconnection and 1 represents connection.

[0014] The action space of the agent It contains switches of all loads within the observation range of the agent. In the initial state, it is assumed that all switches in the system are in the disconnected state. The agent selects a load switch to be connected, and the already connected switch is not allowed to be converted to the disconnected state again. This is the connection-only mechanism. When a load is converted to the connected state, all lines between the node where the load is located and the node where the distributed generator is located are found through the graph search method and switched to the connected state. This is the self-organizing network mechanism. Similarly, the line switch also follows the connection-only mechanism.

[0015] In the initial state, , and are set to 0. When a load is selected and connected, its device parameters are extracted from the corresponding parameters of the test case system and assigned to and , is assigned a value of 1. Due to the existence of the connection-only mechanism, these data remain unchanged after being changed in a training round until the next round of training begins.

[0016] This application adopts a distributed multi-agent framework. The environmental rewards obtained by each agent include not only individual rewards, but also global rewards that reflect global performance. The purpose of forming a microgrid is to maximize the power supply load of the priority load, so the individual reward is related to the load change of the current microgrid. The formula is shown in Formula 1.

[0017] (1); in is the priority weight of the relevant load, when When , it means that a new load added to the microgrid has been powered. At this time, the environmental reward is set to itself; when When , it means that the agent has chosen an action with no benefit. At this time, the environment is set to a negative value; therefore, the individual reward function of the agent can be expressed by Formula 2.

[0018] (2)

[0019] In particular, the operation of the power grid system needs to meet many operating constraints to ensure the stable and safe operation of the system. These constraints are shown in equations (3)-(7).

[0020] (3)

[0021] (4)

[0022] (5)

[0023] (6)

[0024] (7)

[0025] Constraints 3 and 4 are the upper and lower bounds of the generator output power, and constraint 5 ensures the power balance between the generator and the load in the system. is the line loss; constraint 6 is the line transmission capacity constraint; constraint 7 is the operating voltage of each busbar within the safe range. When the above constraints are not met, the individual reward Also a negative value.

[0026] For the global reward, if the actions of all agents lead to an increase in the accumulated weighted load of the entire power network, all agents receive a positive value as the global reward. The formula is shown in Equation 8.

[0027] (8)

[0028] The goal of the reward function is to maximize the expected total reward obtained by each agent, as shown in Equation 9.

[0029] (9)

[0030] Step 104, deploy a multi-agent reinforcement learning algorithm based on a partially observable Markov decision process, and each agent interacts with the power grid system through a connection-only mechanism and a self-organizing network mechanism to sample information and then stores it in its own experience replay pool; based on the DQN algorithm, each agent samples experience samples from the experience replay pool and trains its corresponding strategy network to learn the optimal behavior based on the action masking technology and the experience screening technology.

[0031] Deploy multi-agent reinforcement learning methods from a distributed computing perspective, such as Figure 3 As shown, each agent obtains its observation range information, and then Input its policy network output the action selected in the current state , that is, each agent selects the load to be powered and turns on the switch. These individual actions are aggregated into a joint action and input into the power system to change the state of the system. Then, each agent receives corresponding environmental feedback information, including reward value and the next moment state In this process, the agent’s current state, actions taken, environmental rewards, and state at the next moment are saved in their respective experience pools in the form of tuples for subsequent training.

[0032] In terms of the design of the policy network, the hidden layer of the policy network consists of an adaptive attention mechanism, a graph convolutional network, and a fully connected layer; in the input data of the policy network, is the node feature data, is the edge information; since the agent only changes the connection status of one load at a time, the part of the feature data that changes is very small, which may make it difficult for the policy network to identify valuable features. Therefore, an adaptive attention mechanism is introduced to first The data is input into several fully connected layers, and then the normalized feature data is obtained through the softmax layer, which is then weighted with the original node feature data to calculate the new node feature data and the feature data after the adaptive attention mechanism. Together they serve as the input of the graph convolutional network. Then they are mapped to action value distributions through the fully connected layer. Then, the agent’s actions The greedy policy selects the next action to be performed. The activation function used in the policy network is ReLU.

[0033] This application uses the DQN algorithm architecture to train each agent to learn a strategy. The DQN framework has a dual-strategy network structure. In addition to the Q network used to generate action values ​​at the current moment, there is also a target Q network used to generate action values ​​at the next moment. The loss function of the Q network is As shown in formula 10.

[0034] (10)

[0035] (11)

[0036] in, and Represent the parameters of the Q network and the target Q network respectively, is the discount factor, For the experience pool; Indicates that the agent is observing Next action The action value of To take action Transfer to the next state The reward value generated.

[0037] The action masking technology and experience screening technology are used to optimize the training process. Since selecting repeated actions will lead to invalid benefits, when the agent selects actions within a round of training, it is necessary to mask the actions that have been selected. Figure 4 As shown in Figure 1, the action masking technique is to multiply the action value distribution output by the Q network by a mask sequence consisting of 0 and 1, where the length of the mask sequence is equal to the number of actions output by the Q network. Set the action positions to be discarded in the mask to 0, and the action positions to be retained to 1.

[0038] The experience screening mechanism is to control the ratio of positive return samples to negative return samples in the experience pool of the agent to be 1:1. Figure 5 As shown, when the number of positive return samples in the experience pool is greater than the number of negative return samples, only negative return samples are allowed to enter the experience pool; when the number of negative return samples is greater than the number of positive return samples, only positive return samples are allowed to enter the experience pool.

[0039] Step 106, using the trained strategy network model to control the switch actions in each area of ​​the power system to form a microgrid and restore power supply to the load.

[0040] In the known power system scenario, multiple regions are divided as observation inputs for the intelligent agents in the region. Multiple intelligent agents interact with the power system environment to obtain observation and feedback information, save this information in the experience replay buffer pool, and then sample the sample data in the experience pool by random small batch sampling to train the Q network. The target Q network adopts the soft update method, and each update assigns some parameters in the Q network to the target Q network. The soft update formula is shown in Equation 12.

[0041] (12)

[0042] in, is the coefficient, a number between 0 and 1.

[0043] During the training process, the training is stopped after the average reward value reaches a relatively stable state, and the trained Q network model is saved. The power system parameters of the training scenario are input into the Q network model to obtain the decision action, and finally the networking scheme of multiple microgrids is solved.

[0044] The above multi-microgrid distributed control method based on multi-agent reinforcement learning first models the problem as a multi-agent partially observable Markov decision process. Each agent only relies on local observation information for decision-making, without the need for global data concentration, which solves the problem of communication interruption and incomplete measurement in disasters, improves the system's adaptability to dynamic uncertainty scenarios, and avoids the privacy leakage and single point failure risks of centralized control. The connection-only mechanism is used to limit the communication range, reduce the amount of data transmission and link dependence, reduce privacy risks and improve anti-destruction capabilities; the self-organizing network mechanism supports dynamic reconstruction of communication topology, automatically maintains local coordination when disasters cause link failures, ensures that control instructions are not interrupted, and enhances disaster recovery resilience. Finally, according to the action masking technology, the physical constraints of the power system are embedded, invalid actions are shielded, and the decision space is compressed from the exponential level to the polynomial level, greatly reducing the computational complexity; the experience screening technology prioritizes the retention of high-value disaster recovery samples, alleviates the problem of scarcity of disaster data, improves the learning efficiency of agents for extreme scenarios, and achieves minute-level rapid recovery. This application supports regional parallel decision-making through a multi-agent distributed architecture to avoid centralized computing delays and is suitable for distribution networks with hundreds of nodes; the DQN-based policy network captures nonlinear relationships through end-to-end learning, realizes joint optimization of topology optimization and energy scheduling, and improves the recovery rate of key loads and the utilization rate of renewable energy. Each agent operates independently and collaborates through wired communication. Even if some areas are isolated from the main grid, they can still autonomously optimize strategies based on local experience to ensure the continuous and stable operation of isolated microgrids, and have strong robustness against extreme events such as multiple faults and network attacks.

[0045] In one embodiment, modeling a power network system includes: The power network system model includes busbar nodes and Feeder lines, represented by ,in represents the busbar, Indicates a feeder line; At each node There is a consumption of active power and reactive power Load; Each line and load in the power system is equipped with a controllable switch to connect and disconnect through remote control. When the large power grid loses service due to extreme natural disasters, multiple microgrids are formed by distributed generators installed at certain nodes to supply power to critical loads. Each microgrid is powered by a distributed generator.

[0046] In one embodiment, the partially observable Markov decision process includes the state information of the environment, the observation information of the agent, the action space of the agent, the reward function and the state transition process; the state information of the environment includes the characteristic data of all nodes and lines in the power grid system, and the observation information of each agent Is the status information of the environment A subset of; observation information of the agent , and are the active power and reactive power output by distributed generators, and Represent the active and reactive power consumed by the load, respectively. For edge information, It is a 0-1 type variable representing the connection status of the load, 0 represents disconnection, 1 represents connection, For the The action space of the agent contains the switches of all loads within the observation range of the agent.

[0047] In one embodiment, the reward function includes an individual reward function and a global reward; the model of the reward function is: .

[0048] Constraints: ; in, is the line loss, For individual rewards, represents the global reward, and are the active power and reactive power output by distributed generators, and They represent the upper and lower limits of the active power output of the distributed generators, represents the busbar, represents the busbar node, represents the busbar node set, Indicates the feeder line, represents the set of feeder lines, Indicates the maximum capacity of the feeder line, j represents the imaginary unit, Represents the reactive power transmitted on the feeder line, Indicates busbar node The active power consumed by the connected load at It represents the active power transmitted on the feeder line. Indicates busbar node The voltage amplitude at .

[0049] In one of the embodiments, the connection-only mechanism means that the intelligent agent selects a load switch to be switched to a connected state, and the already connected switch is not allowed to be switched to a disconnected state again; the self-organizing network mechanism means that when a load is switched to a connected state, a graph search method is used to find all the lines between the node where the load is located and the node where the distributed generator is located, and then switch to a connected state.

[0050] In one embodiment, the hidden layer of the policy network is composed of an adaptive attention mechanism, a graph convolutional network, and a fully connected layer; the policy network has a dual-strategy network structure, including a Q network for generating action values ​​at the current moment and a target Q network for generating action values ​​at the next moment.

[0051] In one embodiment, the loss function of the Q network is: ; in, and Represent the parameters of the Q network and the target Q network respectively, is the discount factor, For the experience pool; Indicates that the agent is observing Next action The action value of To take action Transfer to the next state The reward value generated.

[0052] In one embodiment, the target Q network is updated as follows: ; in, is the coefficient, a number between 0 and 1.

[0053] In one of the embodiments, the action masking technology represents multiplying the action value distribution output by the Q network by a mask sequence consisting of 0 and 1, where the length of the mask sequence is equal to the number of actions output by the Q network, wherein the action positions to be discarded in the mask are set to 0, and the action positions to be retained are set to 1; the experience screening mechanism represents controlling the ratio of positive gain samples to negative gain samples in the experience pool of the intelligent agent to be 1:1, and when the number of positive gain samples in the experience pool is greater than the number of negative gain samples, only negative gain samples are allowed to enter the experience pool; when the number of negative gain samples is greater than the number of positive gain samples, only positive gain samples are allowed to enter the experience pool.

[0054] In a specific embodiment, it is assumed that after an extreme disaster causes the public power grid to lose service, all loads in the power grid lose power supply, and only distributed generators installed at certain nodes can be relied upon to supply power to the loads. Figure 6 As shown, Figure 2 The grid environment in the example is finally divided into three areas, and the distributed generators in the three microgrids supply power to the loads in the area. The total rated power utilization of the distributed generators is 90.3%, and 158.025MW of load is restored.

[0055] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0056] In one embodiment, Figure 7 As shown, a multi-microgrid distributed control device based on multi-agent reinforcement learning is provided, including: a system modeling module 702, a model training module 704 and a multi-microgrid distributed control module 706, wherein: The system modeling module 702 is used to model the power network system and model the multi-microgrid formation process as a distributed partially observable Markov decision process; The model training module 704 is used to deploy a multi-agent reinforcement learning algorithm according to the partially observable Markov decision process, and each agent interacts with the power grid system through a connection-only mechanism and a self-organizing network mechanism to sample information and store it in its own experience replay pool; based on the DQN algorithm, each agent samples experience samples in the experience replay pool and trains its corresponding strategy network to learn the optimal behavior according to the action masking technology and the experience screening technology; The multi-microgrid distributed control module 706 is used to use the trained strategy network model to control the switch actions in each area of ​​the power system to form a microgrid and restore power supply to the load.

[0057] For the specific definition of the multi-microgrid distributed control device based on multi-agent reinforcement learning, please refer to the definition of the multi-microgrid distributed control method based on multi-agent reinforcement learning above, which will not be repeated here. Each module in the above-mentioned multi-microgrid distributed control device based on multi-agent reinforcement learning can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0058] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0059] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-microgrid distributed control method based on multi-agent reinforcement learning, characterized in that: The method comprises: Modeling the power network system, the multi-microgrid formation process is modeled as a distributed partially observable Markov decision process; Deploy a multi-agent reinforcement learning algorithm based on the partially observable Markov decision process, and each agent interacts with the power grid system through a connection-only mechanism and a self-organizing network mechanism to sample information and store it in its own experience replay pool; based on the DQN algorithm, each agent samples experience samples from the experience replay pool and trains its corresponding strategy network to learn the optimal behavior based on action masking technology and experience screening technology; The trained strategy network model is used to control the switch actions in each area of ​​the power system to form a microgrid and restore power supply to the load.

2. The method according to claim 1, characterized in that Modeling of power network systems, including: The power network system model includes busbar nodes and Feeder lines, represented by ,in represents the busbar, Indicates a feeder line; At each node There is a consumption of active power and reactive power Load; Each line and load in the power system is equipped with a controllable switch to connect and disconnect through remote control. When the large power grid loses service due to extreme natural disasters, multiple microgrids are formed by distributed generators installed at certain nodes to supply power to critical loads. Each microgrid is powered by a distributed generator.

3. The method according to claim 1, characterized in that The partially observable Markov decision process includes the state information of the environment, the observation information of the agent, the action space of the agent, the reward function and the state transition process; the state information of the environment contains the characteristic data of all nodes and lines in the power grid system, and the observation information of each agent Is the status information of the environment A subset of; the observation information of the agent , and are the active power and reactive power output by distributed generators, and Represent the active and reactive power consumed by the load, respectively. For edge information, It is a 0-1 type variable representing the connection status of the load, 0 represents disconnection, 1 represents connection, For the An agent; the action space of the agent includes the switches of all loads within the observation range of the agent.

4. The method according to claim 3, characterized in that The reward function includes an individual reward function and a global reward; the model of the reward function is: Constraints: in, is the line loss, For individual rewards, represents the global reward, and are the active power and reactive power output by distributed generators, and They represent the upper and lower limits of the active power output of the distributed generators, represents the busbar, represents the busbar node, represents the busbar node set, Indicates the feeder line, represents the set of feeder lines, Indicates the maximum capacity of the feeder line, j represents the imaginary unit, Represents the reactive power transmitted on the feeder line, Indicates busbar node The active power consumed by the connected load at It represents the active power transmitted on the feeder line. Indicates busbar node The voltage amplitude at .

5. The method according to claim 1, characterized in that The connection-only mechanism means that the intelligent agent selects a load switch to be switched to the connected state, and the already connected switch is not allowed to be switched to the disconnected state again; the self-organizing network mechanism means that when a load is switched to the connected state, all the lines between the node where the load is located and the node where the distributed generator is located are found through a graph search method, and then switched to the connected state.

6. The method according to claim 1, characterized in that The hidden layer of the policy network consists of an adaptive attention mechanism, a graph convolutional network and a fully connected layer; the policy network has a dual-strategy network structure, including a Q network for generating action values ​​at the current moment and a target Q network for generating action values ​​at the next moment.

7. The method according to claim 6, characterized in that The loss function of the Q network is: in, and Represent the parameters of the Q network and the target Q network respectively, is the discount factor, For the experience pool; Indicates that the agent is observing Next action The action value of To take action Transfer to the next state The reward value generated.

8. The method according to claim 7, characterized in that The target Q network is updated as follows: in, is the coefficient, a number between 0 and 1.

9. The method according to claim 6, characterized in that The action masking technology means multiplying the action value distribution output by the Q network by a mask sequence consisting of 0 and 1, where the length of the mask sequence is equal to the number of actions output by the Q network, wherein the action positions to be discarded in the mask are set to 0, and the action positions to be retained are set to 1; the experience screening mechanism means that the ratio of positive gain samples to negative gain samples in the experience pool of the control agent is 1:1, and when the number of positive gain samples in the experience pool is greater than the number of negative gain samples, only negative gain samples are allowed to enter the experience pool; when the number of negative gain samples is greater than the number of positive gain samples, only positive gain samples are allowed to enter the experience pool.

10. A multi-microgrid distributed control device based on multi-agent reinforcement learning, characterized in that: The device comprises: The system modeling module is used to model the power network system and model the multi-microgrid formation process as a distributed partially observable Markov decision process; A model training module is used to deploy a multi-agent reinforcement learning algorithm according to the partially observable Markov decision process, wherein each agent interacts with the power grid system through a connection-only mechanism and a self-organizing network mechanism to sample information and then stores it in its own experience replay pool; based on the DQN algorithm, each agent samples experience samples in the experience replay pool and trains its corresponding strategy network to learn the optimal behavior according to the action masking technology and the experience screening technology; The multi-microgrid distributed control module is used to use the trained strategy network model to control the switching actions in each area of ​​the power system to form a microgrid and restore power supply to the load.

Images