Power distribution network fault recovery method and system based on multi-agent reinforcement learning

Abstract

The invention belongs to the technical field of power distribution network fault recovery, and provides a power distribution network fault recovery method and system based on multi-agent reinforcement learning, and the method comprises the steps: taking a recovered load as a target function based on the load priority of a power distribution network, and taking the operation limitation of each node line in the power distribution network as a constraint condition; constructing a power distribution network fault recovery problem model; based on the power distribution network fault recovery problem model, constructing a multi-agent reinforcement learning agent set, and converting the power distribution network fault recovery problem into a Markov decision process; and a multi-agent reinforcement learning framework is constructed based on the agent set of multi-agent reinforcement learning, multi-agent training is carried out, each agent generates a decision according to respective observation information, and a fault recovery strategy is obtained. The objective function and the constraint condition constructed based on the load priority of the power distribution network and the node line operation limitation can ensure the accuracy and the feasibility of the recovery strategy. And the strategy generation time is shortened through the multi-agent parallel decision-making process.

Description

Technical Field

[0001] The present invention belongs to the technical field of distribution network fault recovery, and in particular relates to a distribution network fault recovery method and system based on multi-agent reinforcement learning. Background Art

[0002] Modern distribution networks have variable structures and complex operating modes, and the requirements for distribution automation are getting higher and higher. Distribution network fault recovery must focus on shortening power outage time and improving fault handling capabilities, minimizing power outage losses, and improving social and economic benefits. Distribution network fault recovery belongs to mixed integer nonlinear programming problems and is a typical NP-hard problem. In the fault recovery method, the non-fault power-off area can close the contact switch to restore connectivity with the main grid and reconstruct to restore load power supply, or form an island to restore power supply through DGs black start, or a combination of the two. However, traditional fault recovery methods are difficult to achieve rapid generation of distribution network fault recovery strategies in complex scenarios. The frequent occurrence of extreme events increases the risk of major power outages in the power grid. With the increase of extreme weather such as hail, heavy snow and rainstorms, the probability of distribution network failures has greatly increased. Therefore, it is particularly important to establish a reasonable and effective fault recovery plan to improve system stability.

[0003] QMIX is a deep reinforcement learning algorithm designed specifically for collaborative tasks in multi-agent systems (MAS). However, when the action space is large and the agent needs to choose complex actions, QMIX's local Q-value network may find it difficult to effectively learn the appropriate strategy. Although QMIX introduces a dual Q network and experience replay mechanism to alleviate the over-estimation problem, the learning process may still be limited in high-dimensional action spaces. At the same time, the network parameter input dimension of Qmix is ​​usually fixed, which means that if the application scenario of the multi-agent network changes, the network input dimension will change, which may cause the original model to be directly unusable, thereby increasing the cost of training.

[0004] The Transformer architecture mainly consists of two parts: the encoder and the decoder. Even though the Transformer has shown strong versatility, it is still just a general framework and does not have the ability to directly handle specific tasks. It needs to be combined with other algorithms to be used in specific scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a distribution network fault recovery method and system based on multi-agent reinforcement learning to solve the problems of high training cost and low specificity of the prior art.

[0006] To achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a distribution network fault recovery method based on multi-agent reinforcement learning, comprising: Based on the distribution network load priority, the restored load is used as the objective function and the operation restrictions of each node line in the distribution network are used as constraints to construct a distribution network fault restoration problem model. Based on the distribution network fault recovery problem model, a multi-agent reinforcement learning agent set is constructed to transform the distribution network fault recovery problem into a Markov decision process. A multi-agent reinforcement learning framework is constructed based on the agent collection of multi-agent reinforcement learning to train multiple agents. Each agent makes a decision based on its own observation information to obtain a fault recovery strategy.

[0007] Furthermore, the load priority of the distribution network is based on the restored load as the objective function, specifically including: Based on the load priority, it is defined as the objective function of restoring the load in the entire time range. Specifically, the total length of the distribution network fault recovery step is , the node set is The objective function in the scenario is: (1) In the formula, is the key factor of node i load, is the recovery state of the connected node i at time t. If it is recovered, ,otherwise, ;load is the active power demand of the load connected to node i at time t.

[0008] Furthermore, the operation restrictions of each node line in the distribution network are used as constraints, specifically including: after completing the construction of the objective function of the distribution network fault recovery problem, based on the power flow constraints of the power system, the following constraints are established: (2) (3) (4) (5) Where is the power flow constraint of the three-phase balanced distribution network, and is the active power and reactive power of node i at time t; and is the voltage amplitude and phase of node i at time t; and is the reactive power demand and active power demand of the load connected to node i at time t; and are the conductance and susceptance of branch ij; and is the active power output of the generator and PV module connected to node i at time t; Construct a radial topology constraint: (6) In the formula, is the number of microgrids in the distribution network at time t, is the total number of nodes in the distribution network; Construct node voltage and branch current constraints: (7) (8) In the formula and are the node voltage and branch current respectively; and is the maximum and minimum voltage limit of the node, and It is the maximum and minimum current limit on the line; Construct node output constraints: (9) (10) In the formula, and is the actual active output and maximum allowed active output of PV connection node i at time t.

[0009] Furthermore, the agent set for constructing multi-agent reinforcement learning specifically includes: The state space set, observation space set, action space set and the agent's reward mechanism.

[0010] Further, specifically: all devices with action behavior states on the network are divided, an intelligent agent model is set for each device, a switch intelligent agent is set for each line switch on the network, and a PV intelligent agent and a load intelligent agent are set for each PV device and load on each node; Collect all status information in the distribution network scenario and build a global state space set, which includes node voltage, line current, fault line ID, switch status, restored load power, and current step PV node information; The state space that each agent can observe is set to construct the observation space set of each agent; the switch agent can observe the global state information, and the PV agent and the load agent can observe the state information of the local microgrid and the agent interaction information under the microgrid; By mapping the action behaviors of the agent, the action space of the agent is constructed; the switch agent only decides whether to close the switch, and the load agent only decides whether to restore the load, so the action space of the two is 0 or 1; for the PV agent, a discrete action control scheme is adopted to map its output power to the range of [0,1] according to the maximum output power limit, and divide its action space of [0,1] with a granularity of 0.1; (11) In the formula, and For the action of switches and loads, For the action of PV equipment; Setting up the reward mechanism for reinforcement learning; immediate rewards It is set as the sum of the load power restored at each time step under the premise of considering the load priority, and a penalty mechanism is added to make the power grid operate beyond the constraint limit; (12) In the formula, instant reward is the sum of the restored loads at time t considering the load priority, is the system node set, is the weight factor, is the recovery state of the connected node i at time t. If it is recovered, ,otherwise, ;load is the active power demand of the load connected to node i at time t; is the penalty function for the system exceeding the limit. When the system operation status exceeds the limit, the penalty function is obtained by multiplying each constraint by the penalty coefficient and taking the weighted sum. .

[0011] Furthermore, the agent set based on multi-agent reinforcement learning constructs a multi-agent reinforcement learning framework, performs multi-agent training, and each agent generates a decision based on its own observation information to obtain a fault recovery strategy, including: The Qmix algorithm is used to train multiple agents by introducing the attention mechanism, model overloading, and buffer reuse structure. After the training, a fault recovery strategy is generated.

[0012] Further, specific: Build a replay buffer framework, use the replay buffer of the previous task and the current task experience, mix the interaction track of the previous task into the interaction track of the current task, and accelerate the learning of the current task by reusing historical experience; The model reloading framework is built based on the attention transfer mechanism. The attention mechanism enables the agent to aggregate features and obtain a dimensionally unchanged state representation by assigning different weights to the agent's information. (13) The calculation formula of the weight coefficient is as follows: (14) (15) In the formula and To learn the parameters, Will Mapped to key, Will Mapped to query, , calculate the correlation between the vectors of agent i and agent j through the inner product; then use the softmax function to Normalize and get the attention weight ; Through the attention mechanism, the input features are aggregated and mapped to a fixed dimension, and the model parameters are directly transferred between different tasks; Based on the established Markov decision process, the Qmix algorithm framework is set up for multi-agent reinforcement learning training: The loss function of the Qmix algorithm is: (16) In the formula is the current reward function generated based on the target network, The reward function generated for the current training network; (17) In the formula, is the agent’s immediate reward, The reward function for the next stage generated by the target network; Perform gradient descent to implement multi-agent training, and use the previously constructed replay buffer framework and model reload framework to accelerate the training convergence process of the agents; Each agent can make decisions based on its own observation information, and the collection of decisions made by each agent is the fault recovery strategy.

[0013] In a second aspect, the present invention provides a distribution network fault recovery system based on multi-agent reinforcement learning, comprising: A model building module is used to build a distribution network fault recovery problem model based on the distribution network load priority, with the restored load as the objective function and the operation restrictions of each node line in the distribution network as constraints; The problem conversion module is used to build a multi-agent reinforcement learning agent set based on the distribution network fault recovery problem model and convert the distribution network fault recovery problem into a Markov decision process; The training output module is used to build a multi-agent reinforcement learning framework based on the agent set of multi-agent reinforcement learning, conduct multi-agent training, and each agent generates a decision based on its own observation information to obtain a fault recovery strategy.

[0014] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of a distribution network fault recovery method based on multi-agent reinforcement learning.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of a distribution network fault recovery method based on multi-agent reinforcement learning are implemented.

[0016] Compared with the prior art, the present invention has the following technical effects: The present invention adopts the Qmix algorithm to train multiple agents. Qmix is ​​a multi-agent reinforcement learning algorithm based on value function decomposition, which can effectively handle the cooperation problem between multiple agents. Through the Qmix algorithm, each agent can learn its strategy independently, while coordinating each other's actions through the global Q value function to achieve overall optimization. This algorithm structure enables the present invention to quickly respond to faults in the distribution network and generate recovery strategies in time.

[0017] Distribution network fault recovery is highly time-sensitive, requiring the algorithm to provide a solution in a short time. This invention significantly shortens the generation time of fault recovery strategies through the efficient training mechanism of the Qmix algorithm and the parallel decision-making process between intelligent agents, meeting the real-time requirements of distribution network fault recovery.

[0018] This paper introduces the attention mechanism, a technique widely used in deep learning, to improve the model's attention to and processing capabilities of key information. In the distribution network fault recovery problem, the attention mechanism can help the intelligent agent process observation information more effectively and extract features that are useful for decision-making, thereby accelerating the model's learning process.

[0019] In order to further improve the training efficiency, the present invention designs a model reloading and buffer reuse structure. Model reloading allows the agent to share the learned knowledge between different tasks, avoiding repeated training; buffer reuse utilizes historical interaction data and accelerates the learning of the current task by reusing experience. The combination of these two structures significantly reduces the training cost and time of the model.

[0020] The present invention maps the observation information of each intelligent agent into a high-dimensional vector and realizes the aggregate output of fixed dimensions through the attention mechanism. This processing method not only improves the model's ability to process complex information, but also enables the model to have the ability to migrate under different scenario task requirements.

[0021] Since the model can aggregate and process high-dimensional vector information, and the attention mechanism enables the model to flexibly adjust the degree of attention to different features, the present invention can adapt more quickly and provide effective recovery strategies when facing new scenarios or new tasks. This enhanced migration capability further reduces the training cost and time of the model and improves its practical value.

[0022] In summary, the present invention successfully solves the problems of poor real-time performance, high training cost, and weak migration capability in traditional distribution network fault recovery methods by combining the attention mechanism and the Qmix multi-agent reinforcement learning algorithm. This method not only realizes efficient and rapid fault recovery strategy generation, but also reduces training cost and time, and improves the migration capability and practical value of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the present invention.

[0024] Figure 2 This is a logic block diagram of the present invention.

[0025] Figure 3 It is a multi-agent reinforcement learning algorithm framework. DETAILED DESCRIPTION

[0026] The present invention is further described below in conjunction with the accompanying drawings: Example 1, please refer to Figure 1 , a distribution network fault recovery method based on multi-agent reinforcement learning, comprising: Based on the distribution network load priority, the restored load is used as the objective function and the operation restrictions of each node line in the distribution network are used as constraints to construct a distribution network fault restoration problem model. Based on the distribution network fault recovery problem model, a multi-agent reinforcement learning agent set is constructed to transform the distribution network fault recovery problem into a Markov decision process. A multi-agent reinforcement learning framework is constructed based on the agent collection of multi-agent reinforcement learning to train multiple agents. Each agent makes a decision based on its own observation information to obtain a fault recovery strategy.

[0027] Through reinforcement learning algorithms and multi-agent collaboration, fault recovery strategies can be quickly generated to improve recovery efficiency.

[0028] The objective function and constraints constructed based on the distribution network load priority and node line operation restrictions can ensure the accuracy and feasibility of the restoration strategy.

[0029] The parallel decision-making process of multiple agents shortens the strategy generation time and meets the real-time requirements of distribution network fault recovery.

[0030] Using node line operation restrictions as constraints can ensure the safety of the recovery strategy in actual operation.

[0031] Embodiment 2, the present invention provides a distribution network fault recovery method based on multi-agent reinforcement learning, specifically comprising: Step 1: Construction of distribution network fault recovery problem model; first, Figure 2 As shown in the left part, under the premise of considering the load priority, the load restored in the entire time range is used as the objective function. Then, considering the operation restrictions of each node line in the network as constraints, the distribution network fault recovery problem is converted into a mixed integer nonlinear programming problem, and the construction of the distribution network fault recovery problem model is completed.

[0032] like Figure 2 As shown, the step 1 is specifically as follows: Step 1.1: Consider the load priority, which is defined as the sum of the load restoration objective functions in the entire time range. Specifically, let the total length of a distribution network fault restoration step be , the system node set is The objective function in the scenario is: (1) In the formula, is the key factor of node i load, is the recovery state of the connected node i at time t. If it is recovered, ,otherwise, .load is the active power demand of the load connected to node i at time t.

[0033] Step 1.2: After completing the construction of the objective function for the distribution network fault recovery problem, consider the power flow constraints of the power system and establish the following constraints: (2) (3) (4) (5) Where is the power flow constraint of the three-phase balanced distribution network, and are the active power and reactive power of node i at time t. and is the voltage amplitude and phase of node i at time t. and is the reactive power demand and active power demand of the load connected to node i at time t. and are the conductance and susceptance of branch ij. and is the active power output of the generator and PV panels connected to node i at time t.

[0034] Step 1.3: Construct radial topology constraints: (6) In the formula, is the number of microgrids in the distribution network at time t, is the total number of nodes in the distribution network. Step 1.4: Construct node voltage and branch current constraints: (7) (8) In the formula and are the node voltage and branch current respectively. and is the maximum and minimum voltage limit of the node, and It is the maximum and minimum current limit on the line.

[0035] Step 1.5: Construct node output constraints: (9) (10) In the formula, and is the actual active output and maximum allowed active output of PV connection node i at time t.

[0036] Step 2: If Figure 2 As shown in the middle part, based on the established mathematical model, a set of agents for multi-agent reinforcement learning, the environment state space, the agent observation space and the agent reward mechanism are constructed to transform the distribution network fault recovery problem into a Markov decision process.

[0037] like Figure 2 As shown in the middle part, the step 2 is specifically as follows: Step 2.1: Divide all devices with action behavior states on the network and set an agent model for each device. Set a switch agent for each line switch on the network, and set a PV agent and load agent for each PV device and load at each node.

[0038] Step 2.2: Collect all state information in the distribution network scenario and construct a global state space set, which includes node voltage, line current, fault line ID, switch status, restored load power, and current step PV node information.

[0039] Step 2.3: Set the state space that each agent can observe and construct the observation space set of each agent. For the switch agent, the global state information can be observed. For the PV agent and the load agent, the observation information is the state information of the local microgrid and the agent interaction information under the microgrid.

[0040] Step 2.4: Construct the action space of the agent by mapping the action behavior of the agent. The switch agent only decides whether to close the switch, and the load agent only decides whether to restore the load, so the action space of both is 0 or 1. For the PV agent, a discrete action control scheme is adopted to map its output power to the range of [0,1] according to the maximum output power limit, and divide its action space of [0,1] with a granularity of 0.1.

[0041] (11) In the formula, and For the action of switches and loads, The action of the PV device.

[0042] Step 2.5: Set up the reward mechanism for reinforcement learning. The goal of all agents is to restore the load through source-grid load collaborative control, so all agents are in a cooperative relationship and share rewards. The purpose of distribution network fault recovery is to quickly restore the load and meet the system operation constraints during the recovery process. Therefore, the immediate reward It is set to the sum of the load power restored at each time step under the premise of considering the load priority, and a penalty mechanism is added to make the power grid operate beyond the constraint limit.

[0043] (12) In the formula, instant reward is the sum of the restored loads at time t considering the load priority, is the system node set, is the weight factor, is the recovery state of the connected node i at time t. If it is recovered, ,otherwise, .load is the active power demand of the load connected to node i at time t. is the penalty function for the system exceeding the limit. When the system operation status exceeds the limit, the penalty function is obtained by multiplying each constraint by the penalty coefficient and taking the weighted sum. .

[0044] Step 3: If Figure 2 As shown in the right part, a multi-agent reinforcement learning framework is constructed, and the Qmix algorithm is used to train multi-agents. By introducing the attention mechanism and model overloading and buffer reuse structure, the model training is accelerated and the model training cost is reduced in different power grid scenarios.

[0045] like Figure 3 As shown, the step 3 is specifically as follows: Step 3.1: Build a replay buffer framework; use the replay buffer of the previous task and the current task experience to mix the interaction trajectory of the previous task into the interaction trajectory of the current task, and accelerate the learning of the current task by reusing historical experience.

[0046] Step 3.2: Construct a model reloading framework based on the attention transfer mechanism; the attention mechanism enables the agent to aggregate features and obtain a dimensionally invariant state representation by assigning different weights to the agent's information.

[0047] (13) The calculation formula of the weight coefficient is as follows: (14) (15) In the formula and To learn the parameters, Will Mapped to key, Will Mapped to query, , calculate the correlation between the vectors of agent i and agent j through the inner product. Then use the softmax function to Normalize and get the attention weight .

[0048] Through the attention mechanism, the input features are aggregated and mapped to a fixed dimension, so that model parameters can be directly transferred between different tasks.

[0049] Step 3.3: Based on the Markov decision process established in the previous step, set up the Qmix algorithm framework to implement reinforcement learning training for multiple agents.

[0050] The loss function of the Qmix algorithm is: (16) In the formula is the current reward function generated based on the target network, The reward function generated for the current training network.

[0051] (17) In the formula, is the agent’s immediate reward, The reward function for the next stage generated by the target network. Gradient descent is performed to implement multi-agent training, and the previously constructed replay buffer framework and model reloading framework are used to accelerate the training convergence process of the agents.

[0052] Step 3.4: Each agent can make decisions based on its own observation information, and the collection of decisions made by each agent is the fault recovery strategy. After training, each agent is put into use.

[0053] Table 1 Flowchart

[0054] Example 2 proposes a distribution network fault recovery method based on multi-agent reinforcement learning. The core of this method is to decompose the complex distribution network fault recovery problem into a collaboration problem of multiple agents.

[0055] Firstly, considering the load priority, the sum of the restored loads in the whole time range is taken as the objective function, and the operation restrictions of each node line in the network are considered as constraints, and the distribution network fault restoration problem is converted into a mixed integer nonlinear programming problem.

[0056] By establishing power flow constraints, radial topology constraints, node voltage and branch current constraints, and node output constraints, the feasibility and safety of the recovery strategy in actual operation are ensured.

[0057] Multi-agent system construction: All devices with action behavior states on the network are divided, and an intelligent agent model is set for each device, including switch agent, PV agent and load agent.

[0058] Construct a global state space set and an observation space set for each agent to ensure that the agent can obtain the necessary information to make decisions.

[0059] Set the action space and reward mechanism of the agent so that the agent can make decisions based on the observed information, and guide the agent to learn the optimal strategy through the reward mechanism.

[0060] Multi-agent reinforcement learning training: Build a multi-agent reinforcement learning framework and use the Qmix algorithm to train multi-agents.

[0061] The attention mechanism and model reloading and buffer reuse structure are introduced to accelerate model training and reduce model training costs in different power grid scenarios.

[0062] Through the replay buffer framework and model reloading framework, the training convergence process of the intelligent agent is accelerated and the training efficiency is improved.

[0063] Through the collaboration and cooperation of multiple agents, fault recovery strategies can be quickly generated, shortening recovery time and improving recovery efficiency.

[0064] The objective function and constraints constructed based on load priority and node line operation restrictions can ensure the accuracy and feasibility of the recovery strategy and avoid resource waste and safety hazards caused by blind recovery.

[0065] The multi-agent system can be dynamically adjusted according to the actual operation of the distribution network, adapt to the fault recovery needs in different scenarios, and improve the flexibility and adaptability of the system.

[0066] By introducing the attention mechanism and model reloading and buffer reuse structure, the model training can be accelerated and the model training cost can be reduced in different power grid scenarios, thereby improving the training efficiency and economic benefits.

[0067] During the restoration process, the safety of the restoration strategy in actual operation is ensured by considering the power system's flow constraints, node voltages, and branch current constraints, thus avoiding secondary damage to the distribution network.

[0068] In summary, the distribution network fault recovery method based on multi-agent reinforcement learning proposed in Example 2 has significant technical effects in improving recovery efficiency, enhancing strategy accuracy, improving system flexibility, reducing training costs and ensuring system safety.

[0069] In yet another embodiment of the present invention, a distribution network fault recovery system based on multi-agent reinforcement learning is provided, which can be used to implement the above-mentioned distribution network fault recovery method based on multi-agent reinforcement learning. Specifically, the system includes: A model building module is used to build a distribution network fault recovery problem model based on the distribution network load priority, with the restored load as the objective function and the operation restrictions of each node line in the distribution network as constraints; The problem conversion module is used to build a multi-agent reinforcement learning agent set based on the distribution network fault recovery problem model and convert the distribution network fault recovery problem into a Markov decision process; The training output module is used to build a multi-agent reinforcement learning framework based on the agent set of multi-agent reinforcement learning, conduct multi-agent training, and each agent generates a decision based on its own observation information to obtain a fault recovery strategy.

[0070] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0071] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in a computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of a distribution network fault recovery method based on multi-agent reinforcement learning.

[0072] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in a computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the above-mentioned embodiment regarding a distribution network fault recovery method based on multi-agent reinforcement learning.

[0073] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A distribution network fault recovery method based on multi-agent reinforcement learning, characterized in that: include: Based on the load priority of the distribution network, the restored load is used as the objective function and the operation restrictions of each node line in the distribution network are used as constraints to construct a distribution network fault restoration problem model. Based on the distribution network fault recovery problem model, a multi-agent reinforcement learning agent set is constructed to transform the distribution network fault recovery problem into a Markov decision process. A multi-agent reinforcement learning framework is constructed based on the agent collection of multi-agent reinforcement learning to train multiple agents. Each agent makes a decision based on its own observation information to obtain a fault recovery strategy.

2. A distribution network fault recovery method based on multi-agent reinforcement learning according to claim 1, characterized in that: The method based on the distribution network load priority takes the restored load as the objective function, specifically includes: Based on the load priority, it is defined as the objective function of restoring the load in the entire time range. Specifically, the total length of the distribution network fault recovery step is , the node set is The objective function in the scenario is: (1) In the formula, is the key factor of node i load, is the recovery state of the connected node i at time t. If it is recovered, ,otherwise, ;load is the active power demand of the load connected to node i at time t.

3. A distribution network fault recovery method based on multi-agent reinforcement learning according to claim 1, characterized in that: The operation restrictions of each node line in the distribution network are used as constraints, specifically including: after completing the construction of the objective function of the distribution network fault recovery problem, based on the power flow constraints of the power system, the following constraints are established: (2) (3) (4) (5) Where is the power flow constraint of the three-phase balanced distribution network, and is the active power and reactive power of node i at time t; and is the voltage amplitude and phase of node i at time t; and is the reactive power demand and active power demand of the load connected to node i at time t; and are the conductance and susceptance of branch ij; and is the active power output of the generator and PV module connected to node i at time t; Construct a radial topology constraint: (6) In the formula, is the number of microgrids in the distribution network at time t, is the total number of nodes in the distribution network; Construct node voltage and branch current constraints: (7) (8) In the formula and are the node voltage and branch current respectively; and is the maximum and minimum voltage limit of the node, and It is the maximum and minimum current limit on the line; Construct node output constraints: (9) (10) In the formula, and is the actual active output and maximum allowed active output of PV connection node i at time t.

4. A distribution network fault recovery method based on multi-agent reinforcement learning according to claim 1, characterized in that: The agent set for constructing multi-agent reinforcement learning specifically includes: The state space set, observation space set, action space set and the agent's reward mechanism.

5. A distribution network fault recovery method based on multi-agent reinforcement learning according to claim 4, characterized in that: Specifically: all devices with action behavior states on the network are divided, and an intelligent agent model is set for each device. All line switches on the network are set with a switch intelligent agent, and the PV devices and loads on each node are set with a PV intelligent agent and a load intelligent agent. Collect all status information in the distribution network scenario and build a global state space set, which includes node voltage, line current, fault line ID, switch status, restored load power, and current step PV node information; The state space that each agent can observe is set to construct the observation space set of each agent; the switch agent can observe the global state information, and the PV agent and the load agent can observe the state information of the local microgrid and the agent interaction information under the microgrid; By mapping the action behaviors of the agent, the action space of the agent is constructed; the switch agent only decides whether to close the switch, and the load agent only decides whether to restore the load, so the action space of the two is 0 or 1; for the PV agent, a discrete action control scheme is adopted to map its output power to the range of [0,1] according to the maximum output power limit, and divide its action space of [0,1] with a granularity of 0.1; (11) In the formula, and For the action of switches and loads, For the action of PV equipment; Setting up the reward mechanism for reinforcement learning; immediate rewards It is set as the sum of the load power restored at each time step under the premise of considering the load priority, and a penalty mechanism is added to make the power grid operate beyond the constraint limit; (12) In the formula, instant reward is the sum of the restored loads at time t considering load priority, is the system node set, is the weight factor, is the recovery state of the connected node i at time t. If it is recovered, ,otherwise, ; load is the active power demand of the load connected to node i at time t; is the penalty function for the system exceeding the limit. When the system operation status exceeds the limit, the penalty function is obtained by multiplying each constraint by the penalty coefficient and taking the weighted sum. .

6. A distribution network fault recovery method based on multi-agent reinforcement learning according to claim 1, characterized in that: The agent set based on multi-agent reinforcement learning constructs a multi-agent reinforcement learning framework, performs multi-agent training, and each agent generates a decision based on its own observation information to obtain a fault recovery strategy, including: The Qmix algorithm is used to train multiple agents by introducing the attention mechanism, model reloading, and buffer reuse structure. After the training, a fault recovery strategy is generated.

7. A distribution network fault recovery method based on multi-agent reinforcement learning according to claim 6, characterized in that: Specific: Build a replay buffer framework, use the replay buffer of the previous task and the current task experience, mix the interaction track of the previous task into the interaction track of the current task, and accelerate the learning of the current task by reusing historical experience; The model reloading framework is built based on the attention transfer mechanism. The attention mechanism enables the agent to aggregate features and obtain a dimensionally unchanged state representation by assigning different weights to the agent's information. (13) The calculation formula of the weight coefficient is as follows: (14) (15) In the formula and To learn the parameters, Will Mapped to key, Will Mapped to query, , calculate the correlation between the vectors of agent i and agent j through the inner product; then use the softmax function to Normalize and get the attention weight ; Through the attention mechanism, the input features are aggregated and mapped to a fixed dimension, and the model parameters are directly transferred between different tasks; Based on the established Markov decision process, the Qmix algorithm framework is set up for multi-agent reinforcement learning training: The loss function of the Qmix algorithm is: (16) In the formula is the current reward function generated based on the target network, The reward function generated for the current training network; (17) In the formula, is the agent’s immediate reward, The reward function for the next stage generated by the target network; Perform gradient descent to implement multi-agent training, and use the previously constructed replay buffer framework and model reload framework to accelerate the training convergence process of the agents; Each agent can make decisions based on its own observation information, and the collection of decisions made by each agent is the fault recovery strategy.

8. A distribution network fault recovery system based on multi-agent reinforcement learning, characterized in that: include: A model building module is used to build a distribution network fault recovery problem model based on the distribution network load priority, with the restored load as the objective function and the operation restrictions of each node line in the distribution network as constraints; The problem conversion module is used to build a multi-agent reinforcement learning agent set based on the distribution network fault recovery problem model and convert the distribution network fault recovery problem into a Markov decision process; The training output module is used to build a multi-agent reinforcement learning framework based on the agent set of multi-agent reinforcement learning, conduct multi-agent training, and each agent generates a decision based on its own observation information to obtain a fault recovery strategy.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of a distribution network fault recovery method based on multi-agent reinforcement learning as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a distribution network fault recovery method based on multi-agent reinforcement learning as described in any one of claims 1 to 7 are implemented.

Images