Two-stage dynamic reconfiguration operation method for distribution network based on multi-agent reinforcement learning

Abstract

The application discloses a power distribution network two-stage dynamic reconstruction operation method based on multi-agent reinforcement learning, and comprises the following steps: S1, a first stage, a value function decomposition multi-agent deep reinforcement learning model based on transformation is constructed, a sub-agent is allocated to each switch to realize adaptive output of a reconstructed topology; S2, in the exploration process of the agent, the radiativity and connectivity of the power distribution network are considered, and an action meeting the topology reconstruction constraint of the power distribution network is output; and S3, in the state space of the sub-agent in the multi-agent model, the power of each node, the branch current, the time and the output action of the value function decomposition multi-agent deep reinforcement learning model based on transformation in the first stage are considered, and continuous variables in the operation of the power distribution network are optimized. Through the technical scheme, the power grid reconstruction scheme can be determined automatically, and the technical scheme has expandability and practicality.

Description

Technical Field

[0001] This invention relates to the field of dynamic reconfiguration and optimization operation technology of distribution networks, specifically a two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning. Background Technology

[0002] With the rapid development of urban loads and the large-scale integration of new energy sources under the new power system, urban distribution networks are currently facing the problem of uneven load distribution, posing new challenges to the economy and reliability of urban distribution network operation. When building new overhead lines to balance loads is costly or substation land is scarce, distribution network reconfiguration technology is one of the effective solutions. Distribution network reconfiguration can transfer loads and adjust the network structure by changing the on / off states of tie switches and sectionalizing switches, thereby improving the capacity for clean energy absorption, balancing loads, and eliminating line overloads. However, existing distribution network reconfiguration methods, such as mathematical solution methods, require the establishment of explicit models, which are difficult to solve in the increasingly complex operating environment of distribution networks. Heuristic algorithms are prone to getting trapped in local optima. While single-agent reinforcement learning can find the optimal decision scheme through extensive exploration, it requires pre-determining the reconfiguration topology for each output neuron. Therefore, how to safely and economically reconfigure and optimize the operation of distribution networks with the large-scale integration of new energy sources under the new power system has become an urgent research problem. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a two-stage dynamic reconfiguration operation method for power distribution networks based on multi-agent reinforcement learning, characterized by the following steps:

[0004] Step S1, the first stage, constructs a multi-agent deep reinforcement learning model based on value function decomposition with transformation, assigning a sub-agent to each switch to achieve adaptive output for topology reconstruction; the joint state space of the sub-agents in the multi-agent model is constructed using features such as photovoltaic power generation of photovoltaic nodes, load demand of load nodes, power interaction with the upper-level grid, current flowing through each branch, and the current time; the action space of the sub-agents is based on the on / off state of branch switches; the reward function of the value function decomposition model with transformation comprehensively considers the network loss, switch switching loss, and voltage deviation index of the distribution network; the state transition function of the value function decomposition model with transformation considers the randomness of photovoltaic power output, thus constructing the joint action space of the value function decomposition model with transformation.

[0005] Step S2: During the exploration process, the agent considers the radiation and connectivity of the distribution network and outputs actions that satisfy the topology constraints of the distribution network reconfiguration.

[0006] Step S3, the second stage, considers photovoltaic output and curtailment in the action space of the sub-agent; the reward function of the value function decomposition model with transformation considers the distribution network operation loss, curtailment of photovoltaic power and load, and voltage balance. Based on the deep deterministic policy gradient algorithm, the power of each node, branch current, time of day, and the output action of the multi-agent deep reinforcement learning model based on the value function decomposition model with transformation in the state space of the sub-agent in the multi-agent model are considered to optimize the continuous variables in the operation of the distribution network.

[0007] Furthermore, the joint state space of the sub-agents in the multi-agent model is as follows:

[0008] Sub-agent state space:

[0009]

[0010] In the formula: o n,t This represents the observation of the power distribution network environment by a single agent n at time t; This represents the amount of interaction between node i and the upstream power grid at time t; The amount of photovoltaic power generated at node i at time t; t' represents the load demand of node i at time t; t' represents the per-unit value of time t.

[0011] Sub-agent action space:

[0012]

[0013] In the formula: w represents the action space of the sub-agent n; ij Indicates the on / off state of branch ij;

[0014] Joint state space of the value function decomposition model with transformation:

[0015]

[0016] In the formula: Represents the set of observations of all single agents; This represents the set of actions of all single agents.

[0017] Furthermore, the joint action space of the value function decomposition model with transformation is:

[0018] Joint action space of the value function decomposition model with transformation:

[0019]

[0020] In the formula: This represents the set of actions of all single agents.

[0021] Reward function of the value function decomposition model with transformation:

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] In the formula: This represents the economic operation indicators of the distribution network for the value function decomposition model with transformation during the distribution network reconfiguration phase. express Invert; c loss This indicates the active power loss price for distribution network operators; r ij This represents the resistance of branch ij; These represent the single-operation cost of changing the on / off state of feeder tie switches, transformer tie switches, substation tie switches, and branch section switches, respectively. Specifically, a feeder tie switch connects two feeders belonging to the same transformer within the same substation; a transformer tie switch connects two feeders belonging to different transformers within the same substation; a substation tie switch connects two feeders belonging to different substations; and a branch section switch is a switch in a feeder branch that is in a default closed state. This is to suppress large-scale power flow shifts. These flags represent the on / off status of the feeder tie switch, transformer tie switch, substation tie switch, and branch section switch in the reconfiguration area, respectively. 0 indicates open and 1 indicates closed. Voltage offset index; represents V i N and V i,t Here, c represents the rated voltage value of node i and the actual voltage value during time period t, respectively. t l Indicates load balance; R i,t This represents the load rate of node i during time period t; P represents the average load factor of the distribution network during time period t; i,t P represents the active power injected into node i during time period t. i max The maximum allowable active power injected into node i; N represents the number of nodes in the distribution network; This indicates a penalty for exceeding the limits of trendsetting; λ represents the radiality and connectivity penalty; if the agent's output topology does not satisfy the constraints, a penalty is imposed on the agent; t Indicates step count reward; c r represents a positive reward bias, ensuring that the reward value is greater than 0 when the agent performs actions that satisfy the constraints; c represents the maximum penalty;

[0038] State transition function of the value function decomposition model with transformation:

[0039] s t+1 =f(s) t ,a t ,ρ)

[0040]

[0041] In the formula: ρ represents a random variable; N represents a normal distribution; μ N,PV σ represents the mean of a normal distribution; N,PV This represents the standard deviation of the normal distribution of noise.

[0042] Furthermore, the aforementioned radioactivity is:

[0043]

[0044] Where: N B N represents the number of nodes in the distribution network. Sub Indicates the number of substations in the distribution network; E Always This indicates a branch in which no switch remains connected at all times;

[0045] The connectivity is:

[0046] Based on the distribution network topology, construct an n×n connectivity matrix T, where T(i,j) represents whether there is an electrical connection between node i and node j, while also satisfying i < j. If there is an electrical connection between node i and node j, then T(i,j) = 1; otherwise, T(i,j) = 0. The diagonal elements in the matrix are the sum of the row and column of the element. By tracing the final parent node of each node to see if it is a power source node, we can determine whether the node is an isolated point or located in an island.

[0047] Furthermore, the optimization of continuous variables in the operation of the distribution network includes:

[0048] At time t, the deep deterministic policy gradient algorithm in step S3 first determines the reconfiguration topology of the distribution network through the value function decomposition model based on transformation in steps S1 and S2. Then, the output action based on the value function decomposition model based on transformation is used as the state space of the deep deterministic policy gradient model in step S3. After the two-stage optimization at time t is completed, the algorithm transitions to time t+1 for further optimization.

[0049] The two-stage dynamic reconfiguration operation system for distribution networks based on multi-agent reinforcement learning applies the two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning as described in any of claims 1-5, a distribution network reconfiguration model based on a value function decomposition model with transformation, and a distribution network operation optimization model based on a deep deterministic strategy gradient algorithm.

[0050] The reconfiguration-level rapid decision-making model includes: a first-stage distribution network operation status accurate perception module, a reconfiguration scheme decision module, a first-stage experience pool module, a tie switch action module, and a first information interaction module;

[0051] The first-stage distribution network operation status accurate sensing module is used to sense in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, and the current time in the operation of the distribution network.

[0052] The reconfiguration scheme decision module is used to directly optimize the distribution network topology based on the real-time operating status of the distribution network perceived by the system, and adaptively output the reconfigured distribution network topology.

[0053] The first-stage experience pool module is used to store the historical operating status of the distribution network, the historical actions of the model, and the corresponding reward values.

[0054] The communication switch action module is used to remotely control the switch to open and close according to the reconstruction scheme;

[0055] The first information interaction module is used to provide the distribution network reconfiguration results to the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm;

[0056] The distribution network operation optimization model based on the deep deterministic strategy gradient algorithm includes a second information interaction module, a two-stage distribution network operation status accurate perception module, a two-stage experience pool module, a photovoltaic power grid connection decision module, and a load reduction decision module.

[0057] The second information interaction module is used to receive the reconfiguration scheme of the distribution network reconfiguration model based on the value function decomposition model with transformation;

[0058] The second-stage distribution network operation status accurate sensing module is used to sense in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, the current time, and the output action based on the value function decomposition model with transformation during the operation of the distribution network.

[0059] The two-stage experience pool module is used to store the historical operating environment of the two-stage distribution network, the corresponding decision-making actions of the distribution network operation optimization model, and the reward value obtained after the decision.

[0060] The photovoltaic power grid connection decision module is used to determine the photovoltaic power grid connection quantity under the current state based on the distribution network operation optimization model;

[0061] The load reduction decision module is used to determine the load reduction amount under the current state based on the power distribution network operation optimization model.

[0062] The beneficial effects of this invention are as follows: It proposes a two-stage dynamic reconfiguration optimization operation system method for urban power distribution networks based on multi-agent reinforcement learning. This addresses the challenges of mathematical solution methods, which require explicit modeling and are difficult to solve in increasingly complex power distribution network operating environments. Heuristic algorithms are prone to getting trapped in local optima. While single-agent reinforcement learning can find the optimal decision scheme through extensive exploration, it requires pre-determining the reconfiguration topology for each output neuron. This invention introduces a multi-agent reinforcement learning framework based on centralized training and step-by-step execution, constructing a two-stage power distribution network reconfiguration operation optimization model based on multi-agent reinforcement learning. In the first stage, a multi-agent deep reinforcement learning model based on value function decomposition with transformation is used to assign an agent to each switch in the power distribution network. During training, connectivity and radial constraints are added to enable the model to adaptively output reconfiguration schemes that satisfy these constraints. In the second stage, a deep deterministic policy gradient model is used to optimize continuous variables in the power distribution network optimization operation. This deep reinforcement learning model does not require pre-determining the topology for each output neuron and can autonomously determine the reconfiguration scheme, exhibiting scalability and practicality. Attached Figure Description

[0063] Figure 1 A schematic diagram of the two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning;

[0064] Figure 2 This is a schematic diagram illustrating the working principle of a power distribution network reconfiguration model based on a value function decomposition algorithm with transformation.

[0065] Figure 3 This is a schematic diagram of the working principle of a distribution network operation optimization model based on a deep deterministic strategy gradient algorithm.

[0066] Figure 4 The diagram shows the training results of the two-stage dynamic reconfiguration operation optimization of the distribution network. Detailed Implementation

[0067] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0068] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention; that is, the described embodiments are only a part of the embodiments of the invention, and not all of them. The components of the embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0069] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention. It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0070] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0071] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0072] like Figure 1 As shown, the two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning includes the following steps:

[0073] Step S1, the first stage, constructs a multi-agent deep reinforcement learning model based on value function decomposition with transformation, assigning a sub-agent to each switch to achieve adaptive output for topology reconstruction; the joint state space of the sub-agents in the multi-agent model is constructed using features such as photovoltaic power generation of photovoltaic nodes, load demand of load nodes, power interaction with the upper-level grid, current flowing through each branch, and the current time; the action space of the sub-agents is based on the on / off state of branch switches; the reward function of the value function decomposition model with transformation comprehensively considers the network loss, switch switching loss, and voltage deviation index of the distribution network; the state transition function of the value function decomposition model with transformation considers the randomness of photovoltaic power output, thus constructing the joint action space of the value function decomposition model with transformation.

[0074] Step S2: During the exploration process, the agent considers the radiation and connectivity of the distribution network and outputs actions that satisfy the topology constraints of the distribution network reconfiguration.

[0075] Step S3, the second stage, considers photovoltaic output and curtailment in the action space of the sub-agent; the reward function of the value function decomposition model with transformation considers the distribution network operation loss, curtailment of photovoltaic power and load, and voltage balance. Based on the deep deterministic policy gradient algorithm, the power of each node, branch current, time of day, and the output action of the multi-agent deep reinforcement learning model based on the value function decomposition model with transformation in the state space of the sub-agent in the multi-agent model are considered to optimize the continuous variables in the operation of the distribution network.

[0076] The joint state space of the sub-agents in the multi-agent model is as follows:

[0077] Sub-agent state space:

[0078]

[0079] In the formula: o n,t This represents the observation of the power distribution network environment by a single agent n at time t; This represents the amount of interaction between node i and the upstream power grid at time t; The amount of photovoltaic power generated at node i at time t; t' represents the load demand of node i at time t; t' represents the per-unit value of time t.

[0080] Sub-agent action space:

[0081]

[0082] In the formula: w represents the action space of the sub-agent n; ij Indicates the on / off state of branch ij;

[0083] Joint state space of the value function decomposition model with transformation:

[0084]

[0085] In the formula: Represents the set of observations of all single agents; This represents the set of actions of all single agents.

[0086] The joint action space of the value function decomposition model with transformation is:

[0087] Joint action space of the value function decomposition model with transformation:

[0088]

[0089] In the formula: This represents the set of actions of all single agents.

[0090] Reward function of the value function decomposition model with transformation:

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] In the formula: This represents the economic operation indicators of the distribution network for the value function decomposition model with transformation during the distribution network reconfiguration phase. express Invert; c loss This indicates the active power loss price for distribution network operators; r ij This represents the resistance of branch ij; These represent the single-operation cost of changing the on / off state of feeder tie switches, transformer tie switches, substation tie switches, and branch section switches, respectively. Specifically, a feeder tie switch connects two feeders belonging to the same transformer within the same substation; a transformer tie switch connects two feeders belonging to different transformers within the same substation; a substation tie switch connects two feeders belonging to different substations; and a branch section switch is a switch in a feeder branch that is in a default closed state. This is to suppress large-scale power flow shifts. These flags represent the on / off status of the feeder tie switch, transformer tie switch, substation tie switch, and branch section switch in the reconfiguration area, respectively. 0 indicates open and 1 indicates closed. Voltage offset index; represents V i N and V i,t These are the rated voltage value of node i and the actual voltage value during time period t, respectively. Indicates load balance; R i,t This represents the load rate of node i during time period t; P represents the average load factor of the distribution network during time period t; i,t P represents the active power injected into node i during time period t. i max The maximum allowable active power injected into node i; N represents the number of nodes in the distribution network; This indicates a penalty for exceeding the limits of trendsetting; λ represents the radiality and connectivity penalty; if the agent's output topology does not satisfy the constraints, a penalty is imposed on the agent; t Indicates step count reward; c r represents a positive reward bias, ensuring that the reward value is greater than 0 when the agent performs actions that satisfy the constraints; c represents the maximum penalty;

[0107] State transition function of the value function decomposition model with transformation:

[0108] s t+1 =f(s) t ,a t ,ρ)

[0109]

[0110] In the formula: ρ represents a random variable; N represents a normal distribution; μ N,PV σ represents the mean of a normal distribution; N,PV This represents the standard deviation of the normal distribution of noise.

[0111] The radioactivity is as follows:

[0112]

[0113] Where: N B N represents the number of nodes in the distribution network. Sub Indicates the number of substations in the distribution network; E Always This indicates a branch in which no switch remains connected at all times;

[0114] The connectivity is:

[0115] Based on the distribution network topology, construct an n×n connectivity matrix T, where T(i,j) represents whether there is an electrical connection between node i and node j, while also satisfying i < j. If there is an electrical connection between node i and node j, then T(i,j) = 1; otherwise, T(i,j) = 0. The diagonal elements in the matrix are the sum of the row and column of the element. By tracing the final parent node of each node to see if it is a power source node, we can determine whether the node is an isolated point or located in an island.

[0116] The optimization of continuous variables in the operation of the distribution network includes:

[0117] At time t, the deep deterministic policy gradient algorithm in step S3 first determines the reconfiguration topology of the distribution network through the value function decomposition model based on transformation in steps S1 and S2. Then, the output action based on the value function decomposition model based on transformation is used as the state space of the deep deterministic policy gradient model in step S3. After the two-stage optimization at time t is completed, the algorithm transitions to time t+1 for further optimization.

[0118] The two-stage dynamic reconfiguration operation system for distribution networks based on multi-agent reinforcement learning applies the aforementioned two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning, including a distribution network reconfiguration model based on a value function decomposition model with transformation and a distribution network operation optimization model based on a deep deterministic strategy gradient algorithm.

[0119] The reconfiguration-level rapid decision-making model includes: a first-stage distribution network operation status accurate perception module, a reconfiguration scheme decision module, a first-stage experience pool module, a tie switch action module, and a first information interaction module;

[0120] The first-stage distribution network operation status accurate sensing module is used to sense in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, and the current time in the operation of the distribution network.

[0121] The reconfiguration scheme decision module is used to directly optimize the distribution network topology based on the real-time operating status of the distribution network perceived by the system, and adaptively output the reconfigured distribution network topology.

[0122] The first-stage experience pool module is used to store the historical operating status of the distribution network, the historical actions of the model, and the corresponding reward values.

[0123] The communication switch action module is used to remotely control the switch to open and close according to the reconstruction scheme;

[0124] The first information interaction module is used to provide the distribution network reconfiguration results to the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm;

[0125] The distribution network operation optimization model based on the deep deterministic strategy gradient algorithm includes a second information interaction module, a two-stage distribution network operation status accurate perception module, a two-stage experience pool module, a photovoltaic power grid connection decision module, and a load reduction decision module.

[0126] The second information interaction module is used to receive the reconfiguration scheme of the distribution network reconfiguration model based on the value function decomposition model with transformation;

[0127] The second-stage distribution network operation status accurate sensing module is used to sense in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, the current time, and the output action based on the value function decomposition model with transformation during the operation of the distribution network.

[0128] The two-stage experience pool module is used to store the historical operating environment of the two-stage distribution network, the corresponding decision-making actions of the distribution network operation optimization model, and the reward value obtained after the decision.

[0129] The photovoltaic power grid connection decision module is used to determine the photovoltaic power grid connection quantity under the current state based on the distribution network operation optimization model;

[0130] The load reduction decision module is used to determine the load reduction amount under the current state based on the power distribution network operation optimization model.

[0131] Specifically, existing distribution network reconfiguration methods, such as mathematical solutions, require explicit modeling, which is difficult to apply in increasingly complex distribution network operating environments. Heuristic algorithms are prone to getting trapped in local optima, while single-agent reinforcement learning, although capable of finding optimal decision-making schemes through extensive exploration, requires pre-determining the reconfiguration topology for each output neuron. Therefore, in the context of distribution networks with large-scale integration of new energy sources under the new power system, how to safely and economically reconfigure and optimize their operation has become an urgent research problem.

[0132] like Figure 1 As shown, this invention proposes a two-stage dynamic reconfiguration optimization method for urban power distribution networks based on multi-agent reinforcement learning. First, in the first stage, a multi-agent deep reinforcement learning model with value function decomposition and transformation is used to assign a sub-agent to each switch, achieving adaptive output for reconfigured topology. The state space of the sub-agents in the multi-agent model is constructed using features such as photovoltaic power generation at photovoltaic nodes, load demand at load nodes, power interaction with the upstream grid, current flowing through each branch, and the current time. The action space is based on the on / off state of branch switches. The reward function comprehensively considers network losses, switch switching losses, voltage deviation index, etc., while the state transition function considers the randomness brought about by photovoltaic power output.

[0133] In the second stage, based on the deep deterministic policy gradient algorithm, the power of each node, branch current, time, and output actions of the first-stage model are considered in the state space; the action space only considers photovoltaic output and curtailment; the reward function considers distribution network operation losses, curtailment, voltage balance, etc., to optimize continuous variables in distribution network operation.

[0134] Among them, such as Figure 2 As shown, the distribution network reconfiguration model based on the value function decomposition algorithm with transformation consists of a first-stage distribution network operation status accurate perception module, a reconfiguration scheme decision module, a first-stage experience pool module, a tie switch action module, and a first information interaction module. The first-stage distribution network operation status accurate perception module is used to perceive in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, and the current time during the operation of the distribution network. The reconfiguration scheme decision module is used to directly optimize the distribution network topology based on the distribution network operation status perceived by the system in real time, and adaptively output the reconfigured distribution network topology. The first-stage experience pool module is used to store the historical operation status of the first-stage distribution network, the historical actions of the model, and the corresponding reward values. The tie switch action module is used to remotely control the switching to open and close according to the reconfiguration scheme. The first information interaction module is used to provide the distribution network reconfiguration results to the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm.

[0135] Among them, such as Figure 3As shown, the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm consists of a second information interaction module, a two-stage distribution network operation status accurate perception module, a two-stage experience pool module, a photovoltaic power grid connection decision module, and a load reduction decision module. The second information interaction module is used to receive the reconstruction scheme of the distribution network reconstruction model based on the value function decomposition model with transformation. The two-stage distribution network operation status accurate perception module is used to perceive in real time the power interaction with the upper-level grid, the grid connection power of each photovoltaic power source, the load demand power, the current of each branch, the current time, and the output action of the value function decomposition model with transformation during the operation of the distribution network. The two-stage experience pool module is used to store the historical operating environment of the two-stage distribution network, the corresponding decision actions of the distribution network operation optimization model, and the reward value obtained after the decision. The photovoltaic power grid connection decision module is used to determine the photovoltaic power grid connection quantity under the current state according to the distribution network operation optimization model. The load reduction decision module is used to determine the load reduction quantity under the current state according to the distribution network operation optimization model.

[0136] In the distribution network reconfiguration model based on the value function decomposition algorithm with transformation,

[0137] The sub-agent's state space:

[0138]

[0139] In the formula: o n,t This represents the observation of the power distribution network environment by a single agent n at time t; This represents the amount of interaction between node i and the upstream power grid at time t; The amount of photovoltaic power generated at node i at time t; t' represents the load demand of node i at time t; t' represents the per-unit value at time t.

[0140] The action space of the sub-agent:

[0141]

[0142] In the formula: w represents the action space of the sub-agent n; ij This indicates the on / off state of branch ij.

[0143] The joint state space of the value function decomposition model with transformation:

[0144]

[0145] In the formula: Represents the set of observations of all single agents; This represents the set of actions of all single agents.

[0146] The joint action space of the value function decomposition model with transformation:

[0147]

[0148] In the formula: This represents the set of actions of all single agents.

[0149] The reward function of the value function decomposition model with transformation:

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164]

[0165] In the formula: This represents the economic operation indicators of the distribution network for the value function decomposition model with transformation during the distribution network reconfiguration phase. express Invert; c loss This indicates the active power loss price for distribution network operators; r ij This represents the resistance of branch ij; These represent the single-operation cost of changing the on / off state of feeder tie switches, transformer tie switches, substation tie switches, and branch section switches, respectively. Specifically, a feeder tie switch connects two feeders belonging to the same transformer within the same substation; a transformer tie switch connects two feeders belonging to different transformers within the same substation; a substation tie switch connects two feeders belonging to different substations; and a branch section switch is a switch in a feeder branch that is in a default closed state. This is to suppress large-scale power flow shifts. These flags represent the on / off status of the feeder tie switch, transformer tie switch, substation tie switch, and branch section switch in the reconfiguration area, respectively. 0 indicates open and 1 indicates closed. Voltage offset index; represents V i N and V i,t These are the rated voltage value of node i and the actual voltage value during time period t, respectively. Indicates load balance; R i,t This represents the load rate of node i during time period t; P represents the average load factor of the distribution network during time period t; i,t P represents the active power injected into node i during time period t. i max The maximum allowable active power injected into node i; N represents the number of nodes in the distribution network; This indicates a penalty for exceeding the limits of trendsetting; λ represents the radiality and connectivity penalty; if the agent's output topology does not satisfy the constraints, a penalty is imposed on the agent; t Indicates step count reward; c r represents a positive reward bias, which ensures that the reward value is greater than 0 when the agent performs an action that satisfies the constraints; c represents the maximum penalty.

[0166] The state transition function of the value function decomposition model with transformation:

[0167] s t+1 =f(s) t ,a t ,ρ)

[0168]

[0169] In the formula: ρ represents a random variable; N represents a normal distribution; μ N,PV σ represents the mean of a normal distribution; N,PV This represents the standard deviation of the normal distribution of noise.

[0170] The radiometric criterion:

[0171]

[0172] Where: N B N represents the number of nodes in the distribution network. Sub Indicates the number of substations in the distribution network; E Always This indicates a branch in which no switch is always in the connected state.

[0173] The connectivity criterion:

[0174] Based on the distribution network topology, construct an n×n connectivity matrix T. Let T(i,j) represent whether there is an electrical connection between node i and node j, while also satisfying i < j. If there is an electrical connection between node i and node j, then T(i,j) = 1; otherwise, T(i,j) = 0. The diagonal elements in the matrix are the sum of the row and column of the element. Determine whether a node is an isolated node or located in an island by tracing its final parent node to see if it is a power source node.

[0175] In the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm:

[0176] The state space of the depth deterministic strategy gradient algorithm is:

[0177]

[0178] The action space of the depth deterministic policy gradient algorithm is:

[0179]

[0180] The reward function based on the deep deterministic policy gradient algorithm is:

[0181]

[0182]

[0183]

[0184] In the formula: This represents the economic operation reward of the distribution network in the second stage of the deep deterministic strategy gradient algorithm; since the model also aims to maximize the reward... express Inverse; This indicates a reduction in load costs; This represents the cost of abandoning light.

[0185] Numerical example verification and analysis:

[0186] The proposed method was verified using a modified 148-node system, and the system was optimized and verified using the model proposed in this invention based on 24-hour data.

[0187] like Figure 4 As shown, the example employs a two-stage dynamic reconfiguration optimization method for urban power distribution networks based on multi-agent reinforcement learning to optimize network operation. An agent is assigned to each switch, while a deep deterministic policy gradient model is responsible for optimizing the variables of curtailment of solar power and load. Its reward value reaches near its maximum after 180,000 training iterations. The reward trend shows that the reward value is greater than 0 after 150,000 iterations, indicating that the strategy of centralized training and distributed execution not only enables the agents to make their own topology decisions but also ensures that the decided topology satisfies the constraints of the distribution network's radiality and connectivity. Furthermore, although the reward value oscillates continuously, the optimization effect continuously improves and eventually stabilizes, exceeding the reward value of the original topology, thus verifying the effectiveness of the multi-agent optimization model presented in this paper.

[0188] This invention considers the economic and safety requirements of urban power distribution network operation, as well as the need for rapid decision-making. It focuses on a two-stage dynamic reconfiguration optimization method for urban power distribution networks based on a multi-agent reinforcement learning algorithm. First, a multi-agent deep reinforcement learning model with transform-based value function decomposition is used to assign an agent to each switch to achieve adaptive output for reconfigured topology. Then, during the agent's exploration process, considering the radial and connectivity aspects of the power distribution network, only actions that satisfy the reconfigured topology constraints are output. Finally, based on a deep deterministic policy gradient algorithm, the state space considers the power of each node, branch current, current time, and the output actions of the first-stage model; the action space only considers photovoltaic power output and load curtailment; and the reward function considers network losses, curtailment, voltage balance, etc., optimizing continuous variables in the operation of the power distribution network.

[0189] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

[0190] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning, characterized in that, Includes the following steps: Step S1, the first stage, constructs a multi-agent deep reinforcement learning model based on value function decomposition with transformation, assigns a sub-agent to each switch to achieve adaptive output of topology reconstruction; wherein the joint state space of the sub-agents in the multi-agent model is constructed with the photovoltaic power generation of photovoltaic nodes, the load demand of load nodes, the power interaction with the upper-level grid, the current flowing through each branch and the current time as features of the distribution network at each time. Taking the on / off state of branch switches as the action space of sub-agents, the reward function of the value function decomposition model with transformation comprehensively considers the network loss of distribution network operation, switch on / off loss, and voltage deviation index; the state transition function of the value function decomposition model with transformation considers the randomness of photovoltaic power output, and a joint action space of the value function decomposition model with transformation is constructed. Step S2: During the exploration process, the agent considers the radiation and connectivity of the distribution network and outputs actions that satisfy the topology constraints of the distribution network reconfiguration. Step S3, the second stage, considers photovoltaic power output and curtailment in the action space of the sub-agent; the reward function of the value function decomposition model with transformation considers the distribution network operation loss, curtailment of photovoltaic power and load, and voltage balance. Based on the deep deterministic policy gradient algorithm, the power of each node, branch current, time of day, and the output action of the multi-agent deep reinforcement learning model based on the value function decomposition model with transformation in the state space of the sub-agent in the multi-agent model are considered to optimize the continuous variables in the operation of the distribution network. The joint action space of the value function decomposition model with transformation is: Joint action space of the value function decomposition model with transformation: In the formula: Represents the set of actions of all single agents; Reward function of the value function decomposition model with transformation: In the formula: This represents the economic operation indicators of the distribution network for the value function decomposition model with transformation during the distribution network reconfiguration phase. express Inverse of; This indicates the active power loss price for distribution network operators; Indicates a branch ij The resistance; , , , These represent the single-operation cost of changing the on / off state of feeder tie switches, transformer tie switches, substation tie switches, and branch section switches, respectively. Specifically, a feeder tie switch connects two feeders belonging to the same transformer within the same substation; a transformer tie switch connects two feeders belonging to different transformers within the same substation; a substation tie switch connects two feeders belonging to different substations; and a branch section switch is a switch in a feeder branch that is in a default closed state. This is to suppress large-scale power flow shifts. ; , , , These flags represent the on / off status of the feeder tie switch, transformer tie switch, substation tie switch, and branch section switch in the reconfiguration area, respectively. 0 indicates open and 1 indicates closed. Voltage offset index; indicates and They are nodes i voltage rating and t Actual value for the time period; Indicates the degree of load balance; express t Time period nodes i Load rate; Indicates distribution network t Average load factor over time period; For nodes i exist t Active power injected during a given time period; For nodes i Maximum permissible injected active power; N Indicates the number of nodes in the distribution network; This indicates a penalty for exceeding the limits of trendsetting; This represents the radiality and connectivity penalty; if the topology output by the agent does not satisfy the constraints, a penalty is imposed on the agent. Indicates step count reward; This indicates a positive reward bias, which ensures that the reward value is greater than 0 when the agent performs an action that satisfies the constraints. Indicates the maximum penalty; State transition function of the value function decomposition model with transformation: In the formula: Represents a random quantity; Represents a normal distribution; This represents the mean of a normal distribution; This represents the standard deviation of the normal distribution of noise.

2. The two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning according to claim 1, characterized in that, The joint state space of the sub-agents in the multi-agent model is as follows: Sub-agent state space: In the formula: Represents a single agent n exist t Constantly observed values ​​of the power distribution network environment; Represents a node i exist t The amount of interaction with the upper-level power grid at any given time; node i exist t Photovoltaic power generation at any given moment; Represents a node i exist t The load demand at any given time; Indicates the current time t The per-unit value; Sub-agent action space: In the formula: Representing sub-agents n Action space; Indicates a branch ij On / off state; Joint state space of the value function decomposition model with transformation: In the formula: This represents the set of observations for all single agents.

3. The two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning according to claim 1, characterized in that, The radioactivity is as follows: In the formula: Indicates the number of nodes in the distribution network; Indicates the number of substations in the distribution network; This indicates a branch in which no switch remains connected at all times; The connectivity is: Establish a network based on the distribution network topology Connectivity matrix ,use Represents a node i and nodes j Is there an electrical connection between them, and must it also meet the following requirements? If node i If there is an electrical connection between node j and node j, then = 1; otherwise = 0, while the diagonal elements in the matrix are the sum of the rows and columns of the element; by tracing whether the final parent node of each node is a power node, it can be determined whether the node is an isolated point or in an island.

4. The two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning according to claim 3, characterized in that, The optimization of continuous variables in the operation of the distribution network includes: exist t The deep deterministic policy gradient algorithm at time t first determines the reconfiguration topology of the distribution network through the value function decomposition model with transformation in steps S1 and S2. Then, the output action of the value function decomposition model with transformation is used as the state space of the deep deterministic policy gradient model in step S3. After the two-stage optimization at time t is completed, the algorithm transitions to time t+1 for further optimization.

5. A two-stage dynamic reconfiguration operation system for power distribution networks based on multi-agent reinforcement learning, characterized in that, The two-stage dynamic reconfiguration operation method for distribution networks based on multi-agent reinforcement learning as described in any one of claims 1-4 is applied to the distribution network reconfiguration model based on the value function decomposition model with transformation and the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm. The distribution network reconfiguration model based on the value function decomposition model with transformation includes: a first-stage distribution network operation status accurate perception module, a reconfiguration scheme decision module, a first-stage experience pool module, a tie switch action module, and a first information interaction module; The first-stage distribution network operation status accurate sensing module is used to sense in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, and the current time in the operation of the distribution network. The reconfiguration scheme decision module is used to directly optimize the distribution network topology based on the real-time operating status of the distribution network perceived by the system, and adaptively output the reconfigured distribution network topology. The first-stage experience pool module is used to store the historical operating status of the distribution network, the historical actions of the model, and the corresponding reward values. The communication switch action module is used to remotely control the switch to open and close according to the reconstruction scheme; The first information interaction module is used to provide the distribution network reconfiguration results to the distribution network operation optimization model based on the deep deterministic strategy gradient algorithm; The distribution network operation optimization model based on the deep deterministic strategy gradient algorithm includes a second information interaction module, a two-stage distribution network operation status accurate perception module, a two-stage experience pool module, a photovoltaic power grid connection decision module, and a load reduction decision module. The second information interaction module is used to receive the reconfiguration scheme of the distribution network reconfiguration model based on the value function decomposition model with transformation; The second-stage distribution network operation status accurate sensing module is used to sense in real time the power interaction with the upper-level grid, the grid-connected power of each photovoltaic power source, the load demand power, the current of each branch, the current time, and the output action based on the value function decomposition model with transformation during the operation of the distribution network. The two-stage experience pool module is used to store the historical operating environment of the two-stage distribution network, the corresponding decision-making actions of the distribution network operation optimization model, and the reward value obtained after the decision. The photovoltaic power grid connection decision module is used to determine the photovoltaic power grid connection quantity under the current state based on the distribution network operation optimization model; The load reduction decision module is used to determine the load reduction amount under the current state based on the power distribution network operation optimization model.

Images