A multi-voltage level power distribution network reconstruction method based on action value decomposition framework

Abstract

The application discloses a multi-voltage level distribution network reconstruction method based on an action value decomposition framework, comprising the following steps: constructing a multi-voltage level distribution network operation model; formalizing the multi-voltage level distribution network reconstruction problem into a cooperative Markov game; based on a centralized training-decentralized execution framework, using an "actor-critic" algorithm based on value decomposition to train and control the agent, the centralized evaluation network receives the value information and the global state of each local agent, outputs the global state value, and realizes credit distribution across voltage levels. The application overcomes the defects of poor convergence, insufficient coordination and lack of topological adaptability of the existing method in the multi-voltage level distribution network reconstruction, can effectively improve the operation performance of the multi-voltage level distribution network in voltage regulation, transformer protection and load balancing, is significantly superior to the existing reinforcement learning method, and has a wide engineering application prospect.

Description

Technical Field

[0001] This invention belongs to the field of distribution network operation and control technology, and specifically relates to a multi-voltage level distribution network reconfiguration method based on an action value decomposition framework. Background Technology

[0002] With the continuous expansion of distribution network scale and the deep integration of distributed renewable energy, the complexity and uncertainty of power system operation have increased significantly. Multi-voltage level distribution networks, due to cross-level energy interaction and power flow coupling, are more prone to operational problems such as voltage exceeding limits, transformer overload, and feeder load imbalance. Dynamic network reconfiguration, as an effective means to improve system flexibility and security, can improve operational status through switching operations and resource scheduling. However, most existing research focuses on single-voltage level scenarios, making it difficult to achieve coordinated optimization across multiple voltage levels, thus limiting the improvement of global operational goals. Traditional mathematical programming or heuristic algorithms face combinatorial explosion and insufficient computational efficiency when dealing with multi-voltage level reconfiguration, making it difficult to meet near real-time control requirements. In recent years, deep reinforcement learning methods have made some progress in single-voltage level reconfiguration, but still have three shortcomings in multi-voltage level environments: first, the training process is susceptible to environmental non-stationarity, leading to unstable convergence; second, the lack of global value modeling among agents makes it difficult to solve credit allocation problems, reducing the coordination effect across voltage levels; and third, existing methods generally lack effective perception of dynamic changes in the grid topology, resulting in insufficient adaptability of control strategies in complex operating environments. Summary of the Invention

[0003] The purpose of this invention is to provide a multi-voltage-level distribution network reconfiguration method based on an action value decomposition framework, which can take into account global coordination, convergence stability and dynamic topology awareness, so as to improve the reconfiguration performance of multi-voltage-level distribution networks and the level of renewable energy consumption.

[0004] To achieve the above objectives, the solution of the present invention is:

[0005] A multi-voltage level distribution network reconfiguration method based on an action value decomposition framework includes,

[0006] Collect the operating topology information of multi-voltage level distribution networks, the voltage and power injection data of key nodes, and the operating status of switches and on-load tap-changing transformers to construct an operating model of multi-voltage level distribution networks;

[0007] The problem of reconfiguring distribution networks at multiple voltage levels is formalized as a cooperative Markov game, defining the observation space, action space, and global reward function for voltage level agents, feeder agents, and substation agents.

[0008] Based on a centralized training-decentralized execution framework, an agent-evaluator algorithm based on value decomposition is used for agent training and control. The centralized evaluation network receives the value information and global state of each local agent and outputs the global state value to achieve credit allocation across voltage levels.

[0009] The optimization objective function of the multi-voltage level distribution network operation model is:

[0010]

[0011] in, These are the weight coefficients for each term in the objective function; For branch index; For node indexing; Index for high-voltage transformers; For medium-voltage feeder index; For switch index; Index for on-load tap-changing transformers; For branch set; A set of nodes; A collection of high-voltage transformers; For medium-voltage feeder sets; For a set of switches; This is a collection of on-load tap-changing transformers; Branch resistance; For branch circuit reactance; For nodes At any moment The square of the voltage; branch road At any moment The square of the current; For switch At any moment The state; For switch At any moment The state of the previous moment; On-load tap-changing transformer At any moment The location of the tap; On-load tap-changing transformer At any moment The tap position at the previous moment; High voltage transformer At any moment Operating capacity; High voltage transformer Rated capacity; medium voltage feeder At any moment The active power; For a moment Average active power of all feeders;

[0012] The constraints are as follows:

[0013] The current balance constraint is,

[0014]

[0015] in, For nodes At any moment The active and reactive loads; Contribute to photovoltaic power; branch road Active power and reactive power; For the state variables of the branch;

[0016] The voltage-power flow relationship is constrained as follows:

[0017]

[0018]

[0019] in, As an auxiliary variable; It is a constant; This is the lower limit of the node voltage. This is the upper limit of the node voltage; branch road The maximum value of the current; For nodes The square of the voltage; For nodes The square of the voltage; branch road Active power; branch road reactive power; branch road The square of the current;

[0020] Voltage and current safety constraints are as follows:

[0021] .

[0022] Specifically, the multi-voltage-level distribution network reconfiguration problem is formalized as a cooperative Markov game, namely,

[0023] Cooperative Markov games are defined as quintuples. ,in The number of agents; The actual state space of the environment; and The first The local observation space and action space of an intelligent agent; For the reward function; Let be the state transition probability function;

[0024] At every moment intelligent agent Receive local observations And according to its policy function Select Action The joint action of all intelligent agents Execute within the environment to achieve the true state. According to the transition probability function Evolution, and by the reward function Generate instant rewards; the goal is to maximize global discount returns. maximize,

[0025]

[0026] in, As a discount factor, for The reward function at each time step.

[0027] In the local observation space, the high-voltage side reconfiguration agent is at time... Local observation for,

[0028]

[0029] in, This refers to the voltage at the high-voltage node. To inject power into active power, To inject power into reactive power, This is the topology of the high-voltage side distribution network system. High-voltage side moment The previous reconstruction plan, It is a set of nodes in the high-voltage side distribution network. Modeled in non-Euclidean graph form, node features are Topology connection by definition;

[0030] The medium-pressure side agent at time Observations for,

[0031]

[0032] in, This is the topology of the medium-voltage side distribution network system. For the intermediate pressure side moment The previous reconstruction plan, This refers to the set of nodes in the medium-voltage distribution network.

[0033] High-voltage and medium-voltage intelligent agents transmit voltage and power injection information from the feeder ends and photovoltaic access nodes to the substation intelligent agent, which observes... for,

[0034]

[0035] in, For a set of switches, For the set of feeder end nodes;

[0036] In the real state space, the global environment state is:

[0037]

[0038] in, For the global distribution network system topology, A set of nodes;

[0039] In the action space of the intelligent agent, the high-voltage side action is as follows: The action of the medium-pressure side is ;

[0040] The substation intelligent agent action is represented as Its combinations include Species, among which This refers to the number of tap positions. The number of substations;

[0041] All agents share the same global reward function. Its expression is,

[0042]

[0043] in, These are the weight coefficients for each term in the objective function; For branch set; The square of the branch current; Branch resistance; Square the node voltage; A collection of high-voltage transformers; This refers to the operating capacity of the high-voltage transformer. This refers to the rated capacity of the high-voltage transformer. For medium-voltage feeder sets; This refers to the active power of the feeder. Average active power; For a set of switches; It is in the on / off state; For a moment The on / off state at the previous moment; This is a collection of on-load tap-changing transformers; This refers to the tap position of an on-load tap-changing transformer. For a moment The previous switching status was the tap position of the on-load tap-changing transformer.

[0044] The overall architecture of the value decomposition-based "actor-evaluator" algorithm is as follows:

[0045] set up Each agent contains a local policy network and a local evaluation network, which output discrete action distribution and local state value, respectively. The agent's local policy network and local evaluation network share a non-output layer. The central evaluation network of the control center receives the local state value and the global state as input and outputs the global state value.

[0046] Each local reconstruction agent and the centralized evaluation network embeds a graph transformer layer to extract dynamic topological features.

[0047] The centralized evaluation network receives all local state values ​​and the global state, and outputs the global value function. for,

[0048]

[0049] in, for Global status of the time-period power distribution system; For intelligent agents At any moment State value;

[0050] To ensure consistency between the global optimum and the local optimum, the following monotonicity relationship must be satisfied:

[0051]

[0052] The weights are generated by a supernetwork based on the global state, ensuring non-negativity and maintaining monotonicity; the centralized evaluation network is trained by minimizing the temporal difference error, and its parameters are updated as follows.

[0053]

[0054] in, For mixed network parameters, For learning rate, To guide the goal;

[0055] The gradient for parameter update in the local policy network is:

[0056]

[0057] in, For local policy network parameters; It is the time-difference dominance function; This represents the action of the local agent at time t. For the observations of the local agent at time t;

[0058] Further simplified to a standard "actor-evaluator" format,

[0059]

[0060] in, For overall discount rewards; This is the global value function.

[0061] Each local reconstruction agent and the centralized evaluation network have an embedded graph transformer layer, specifically,

[0062] Position encoding using the Laplacian matrix.

[0063]

[0064] in, For Laplace matrix, The eigenvector matrix, For the eigenvalue diagonal matrix, select the largest... Each feature value and its corresponding feature vector are used as a positional encoding;

[0065] By using a multi-head attention mechanism and neighbor aggregation to update node features, the first... Nodes in the layer graph transformer Features Represented as:

[0066]

[0067] in, For nodes The neighborhood group, For the first The weight of each attention head, For value matrices, Indicates a splicing operation; For the first Layer nodes Features;

[0068] The formula for calculating attention weights is:

[0069]

[0070] in, These are the query and key matrices, respectively. Dimensions for each attention head;

[0071] After residual connection and normalization operations, the output of the graph transformer layer is:

[0072]

[0073] The output is concatenated with the action code from the previous moment and then sequentially input into a multilayer perceptron, a recurrent neural network with memory, a policy head, and a value head to generate a reconstructed policy and estimate the state value. The parameters of each graph transformer layer are updated synchronously with the parameters of the corresponding agent network or centralized evaluation network during training.

[0074] By adopting the above scheme, this invention can effectively alleviate transformer overload, improve feeder load balance, and enhance the absorption of renewable energy while ensuring voltage safety. This invention overcomes the shortcomings of existing methods in multi-voltage-level distribution network reconfiguration, such as poor convergence, insufficient coordination, and lack of topology adaptability. Under a centralized training-distributed execution framework, it introduces a centralized scalar commentator to achieve accurate credit allocation across voltage levels and combines a graph converter structure to dynamically extract grid topology features, thereby improving the algorithm's decision-making accuracy and stability in complex environments. This effectively enhances the operational performance of multi-voltage-level distribution networks in voltage regulation, transformer protection, and load balancing, significantly outperforming existing reinforcement learning methods and demonstrating broad engineering application prospects. Attached Figure Description

[0075] Figure 1 This is a flowchart of the method of the present invention;

[0076] Figure 2 This is a schematic diagram of the power distribution network topology used in the embodiments of the present invention;

[0077] Figure 3 This is a comparison chart of the training reward functions of the method proposed in this invention and other methods;

[0078] Figure 4 This is a comparison chart of the voltage control results of the method proposed in this invention and other methods. Detailed Implementation

[0079] The technical solution and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.

[0080] like Figure 1 As shown, this embodiment of the invention provides a multi-voltage level distribution network reconfiguration method based on an action value decomposition framework, comprising the following steps:

[0081] Step 1: Collect the operating topology information, key node voltage and power injection data, and operating status of switches and on-load tap-changing transformers for multi-voltage level distribution networks. Construct a multi-voltage level distribution network operation model. By rationally controlling the operating status of switches and on-load tap-changing transformers in the distribution network, reduce line losses, voltage deviations, and operating costs, while also considering voltage level-specific operating objectives, including mitigating high-voltage transformer overload risks and improving the load balancing capability of medium-voltage feeders.

[0082] Its optimization objective function is expressed as:

[0083] (1)

[0084] in: These are the weight coefficients for each term in the objective function; For branch index; For node indexing; Index for high-voltage transformers; For medium-voltage feeder index; For switch index; Index for on-load tap-changing transformers; For branch set; A set of nodes; A collection of high-voltage transformers; For medium-voltage feeder sets; For a set of switches; This is a collection of on-load tap-changing transformers; Branch resistance; For branch circuit reactance; For nodes At any moment The square of the voltage; branch road At any moment The square of the current; For switch At any moment The state (binary variable); On-load tap-changing transformer At any moment The tap position (integer variable); High voltage transformer At any moment Operating capacity; High voltage transformer Rated capacity; medium voltage feeder At any moment The active power; For a moment Average active power of all feeders.

[0085] The constraints are as follows:

[0086] 1) Power flow balance constraint:

[0087] (2)

[0088] in, For nodes At any moment The active and reactive loads; Contribute to photovoltaic power; branch road Active power and reactive power, For the state variables of the branch.

[0089] 2) Constraints on the relationship between voltage and power flow:

[0090] (3)

[0091] (4)

[0092] in, As an auxiliary variable, It is a constant; For nodes The square of the voltage; For nodes The square of the voltage; branch road Active power; branch road reactive power; branch road The square of the current.

[0093] 3) Voltage and current safety constraints:

[0094] (5)

[0095] in, These are the upper and lower limits of the node voltage; branch road The maximum value of the current.

[0096] Step 2: The multi-voltage-level distribution network reconfiguration problem is formalized as a cooperative Markov game, defining the observation space, action space, and global reward function for voltage level agents, feeder agents, and substation agents.

[0097] Step 2.1, clarify the principle of cooperative Markov process:

[0098] Cooperative Markov games are defined as quintuples. ,in The number of agents; The actual state space of the environment; and The first The local observation space and action space of an intelligent agent; For the reward function; Let be the state transition probability function.

[0099] At every moment intelligent agent Receive local observations And according to its policy function Select Action The joint action of all intelligent agents Execute within the environment to achieve the true state. According to the transition probability function Evolution, and by the reward function Generate immediate rewards. The global reward function integrates the operational objectives of multi-voltage level distribution networks. The goal of cooperative Markov game is to promote local coordination and maximize global discount rewards by optimizing the policies of each agent.

[0100] (6)

[0101] in, As a discount factor, for The reward function at each time step.

[0102] Step 2.2, clarify the local observations of each agent:

[0103] High-voltage side reconfiguration agent at time The observations include high-voltage node voltages. Active power injection Reactive power injection High-voltage side distribution network system topology and the reconfiguration scheme of the high-voltage side at the previous moment. Its local observation is represented as:

[0104] (7)

[0105] The first four items are modeled in non-Euclidean graph form, and the node features are: Topology connection by definition.

[0106] The medium-pressure side agent at time The observations are as follows:

[0107] (8)

[0108] The high-voltage and medium-voltage agents transmit voltage and power injection information from the feeder ends and photovoltaic access nodes to the substation agent, whose observations are as follows:

[0109] (9)

[0110] Step 2.3, define the global state:

[0111] The global environment state is determined by the global node voltage. Net power injection at nodes and global distribution network system topology constitute:

[0112] (10)

[0113] in, For the global distribution network system topology, A set of nodes;

[0114] Step 2.4, define the actions of each agent:

[0115] To reduce sparse rewards caused by ineffective actions, all switch combinations that satisfy the radial constraints are pre-enumerated. All candidate solutions are generated using a tool, and a depth-first search algorithm is employed to eliminate isolated or self-looping solutions.

[0116] High-voltage side action is represented as The action on the medium-pressure side is represented as For a substation intelligent agent, if each on-load tap-changing transformer has... There are 1 tap position and the number of substations is 1 Then it can form A combination, the action is represented as .

[0117] Step 2.5, Define the reward function:

[0118] All agents share the same global reward function. The single-step reward considers both general objectives and voltage-level-specific objectives, and its expression is:

[0119] (11)

[0120] in: The square of the branch current; Branch resistance; Square the node voltage; This refers to the operating capacity of the high-voltage transformer. This refers to the rated capacity of the high-voltage transformer. This refers to the active power of the feeder. Average active power; It is in the on / off state; This represents the tap position of the on-load tap-changing transformer. If no feasible solution is found in the power flow calculation at any given moment, the reward is assigned a large negative value.

[0121] Step 3: Based on the centralized training-distributed execution framework, the agent is trained and controlled using a value decomposition-based "actor-evaluator" algorithm.

[0122] Step 3.1: Define the overall architecture of the algorithm.

[0123] This invention combines the "actor-evaluator" framework with a value decomposition-based multi-agent Q-learning method to form the VDAC algorithm, thereby improving training stability and reducing estimation bias. Each agent includes a local policy network and a local evaluation network, which output discrete action distributions and local state values, respectively. To reduce computational overhead, the local policy network and the local evaluation network share a non-output layer. The central evaluation network receives the local state values ​​and the global state as input and outputs the global state value.

[0124] The parameter updates of each local policy network are guided by policy gradients, and their robustness does not depend on the accuracy of the local evaluation network, ensuring stable convergence even with estimation bias. To improve exploration efficiency and training speed, a multi-threaded parallel training mechanism is introduced to fully utilize multi-core CPU and GPU resources. Considering the rapid evolution of the distribution network topology, this invention embeds graph transformer layers within the local policy network, local evaluation network, and centralized evaluation network of each local reconstruction agent to extract dynamic topological features.

[0125] Step 3.2, Algorithm Convergence Analysis

[0126] The centralized evaluation network receives the values ​​of all local states and the global state, and outputs a global value function:

[0127] (12)

[0128] To ensure consistency between the global optimum and the local optimum, the following monotonicity relationship must be satisfied:

[0129] (13)

[0130] The weights are generated by a supernetwork based on the global state, ensuring non-negativity and maintaining monotonicity. The centralized evaluation network is trained by minimizing the temporal difference error, and its parameters are updated as follows:

[0131] (14)

[0132] in, For mixed network parameters, For learning rate, To guide the goal.

[0133] The gradient for parameter update in the local policy network is:

[0134] (15)

[0135] in, For local policy network parameters, This is the timing difference dominance function.

[0136] This can be further simplified to the standard "actor-evaluator" form:

[0137] (16)

[0138] Lemma 1: When using a TD-based evaluation network and the above policy gradient, the VDAC algorithm of this invention converges to the stationary point of the expected reward with probability 1.

[0139] Step 4: Embed graph transformers within the “actor” and “evaluator” networks to enhance the algorithm’s ability to mine topological features and evaluate global state value through position embedding and multi-head graph attention mechanisms.

[0140] Before reconstructing the agent network from local observations and inputting the global state into the centralized evaluation network, global topological features are extracted through a graph transformer layer. The non-Euclidean graph consists of node voltage and power injection information. Position encoding is first performed using the Laplacian matrix:

[0141] (17)

[0142] in, It is an adjacency matrix. For degree matrix, For Laplace matrix, The eigenvector matrix, For the eigenvalue diagonal matrix, select the largest... Each feature value and its corresponding feature vector is used as a positional encoding.

[0143] Then, the node features are updated by aggregating with neighbors through a multi-head attention mechanism. Layer nodes The features are represented as follows:

[0144] (18)

[0145] in, For nodes The neighborhood group, For the first The weight of each attention head, For value matrices, This indicates a splicing operation.

[0146] The formula for calculating attention weights is:

[0147] (19)

[0148] in, These are the query and key matrices, respectively. Dimensions for each attention head.

[0149] After residual connection and normalization operations, the final output of the graph transformer layer is:

[0150] (20)

[0151] The output is concatenated with the action code from the previous time step and then sequentially input into a multilayer perceptron, a recurrent neural network with memory, a policy head, and a value head to generate a reconstructed policy and estimate the state value. The parameters of each graph transformer layer are updated synchronously with the parameters of the corresponding agent network or centralized evaluation network during training.

[0152] The following is based on Figure 2 The effectiveness of this invention is verified using a multi-voltage-level power distribution network as an example. The system includes a high-voltage side and a medium-voltage side, and is configured with five intelligent agents: one on the high-voltage side, two on the medium-voltage side, and two substations. Several branches on the high-voltage side are connected to a 110 / 35kV transformer (capacity 40–50 MVA), and key branches on the medium-voltage side are equipped with 35 / 10kV on-load tap-changing transformers (OLTCs), forming a typical operating environment across voltage levels.

[0153] like Figure 3 As shown, the method of this invention initially exhibits some fluctuations due to the introduction of the graph transformer structure, but with training iterations, it achieves stable convergence after approximately 1700 rounds and maintains a high average reward value. In contrast, while QMIX can converge, it gets trapped in local optima earlier and lacks the ability to continuously explore dynamic environments; MASAC, due to the lack of explicit global value modeling and poor adaptability to high-dimensional discrete actions, shows significant fluctuations and convergence instability. The results demonstrate that this invention possesses significant advantages in stability and global optima in complex multi-voltage level environments.

[0154] like Figure 4As shown, during peak load periods (6 PM), the voltage at nodes of some medium-voltage feeders, such as feeder 2 in Figure (a) and feeder 5 in Figure (b), drops significantly under traditional methods, with the lowest voltage falling below 0.94 pu, posing an operational risk. After adopting the method of this invention, heavily loaded terminal nodes are rationally transferred to feeders with lower load rates. Simultaneously, combined with OLTC range adjustment, the voltage at each node of the system is maintained within a safe range of 0.95-1.05 pu, with fluctuations around 1.0 pu. The voltage deviation is significantly smaller than that of the comparative method, significantly improving voltage safety and stability.

[0155] This invention not only improves voltage levels but also effectively alleviates overload problems on the high-voltage side transformer, enhances load balance between feeders, and maintains low network losses throughout the 24-hour operating cycle. Compared to QMIX and MASAC, this invention outperforms in multiple metrics.

[0156] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The embodiments of the present invention can be implemented using various computer languages.

[0157] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0158] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A multi-voltage level distribution network reconfiguration method based on an action value decomposition framework, characterized in that: include, Collect the operating topology information, key node voltage and power injection data, and operating status of switches and on-load tap-changing transformers of multi-voltage level distribution networks to construct an operating model of multi-voltage level distribution networks; The problem of reconfiguring distribution networks at multiple voltage levels is formalized as a cooperative Markov game, defining the observation space, action space, and global reward function for voltage level agents, feeder agents, and substation agents. Based on a centralized training-decentralized execution framework, an agent-evaluator algorithm based on value decomposition is used for agent training and control. The centralized evaluation network receives the value information and global state of each local agent and outputs the global state value to achieve credit allocation across voltage levels.

2. The method as described in claim 1, characterized in that: The optimization objective function of the multi-voltage level distribution network operation model is: ， in, These are the weight coefficients for each term in the objective function; For branch index; For node indexing; Index for high-voltage transformers; For medium-voltage feeder index; For switch index; Index for on-load tap-changing transformers; For branch set; A set of nodes; A collection of high-voltage transformers; For medium-voltage feeder sets; For a set of switches; This is a collection of on-load tap-changing transformers; Branch resistance; For branch circuit reactance; For nodes At any moment The square of the voltage; branch road At any moment The square of the current; For switch At any moment The state; For switch At any moment The state of the previous moment; On-load tap-changing transformer At any moment The location of the tap; On-load tap-changing transformer At any moment The tap position at the previous moment; High voltage transformer At any moment Operating capacity; High voltage transformer Rated capacity; medium voltage feeder At any moment The active power; For a moment Average active power of all feeders; The constraints are as follows: The current balance constraint is, ， in, For nodes At any moment The active and reactive loads; Contribute to photovoltaic power; branch road Active power and reactive power; For the state variables of the branch; The voltage-power flow relationship is constrained as follows: ， ， in, As an auxiliary variable; It is a constant; This is the lower limit of the node voltage. This is the upper limit of the node voltage; branch road The maximum value of the current; For nodes The square of the voltage; For nodes The square of the voltage; branch road Active power; branch road reactive power; branch road The square of the current; Voltage and current safety constraints are as follows: 。 3. The method as described in claim 1, characterized in that: The multi-voltage-level distribution network reconfiguration problem is formalized as a cooperative Markov game, specifically... Cooperative Markov games are defined as quintuples. ,in The number of agents; The actual state space of the environment; and The first The local observation space and action space of an intelligent agent; For the reward function; Let be the state transition probability function; At every moment intelligent agent Receive local observations And according to its policy function Select Action The joint action of all intelligent agents Execute within the environment to achieve the true state. According to the transition probability function Evolution, and by the reward function Generate instant rewards; The goal is to maximize the overall discount return. maximize, ， in, As a discount factor, For a moment The reward function.

4. The method as described in claim 3, characterized in that: In the local observation space, the high-voltage side reconfiguration agent is at time... Local observation for, ， in, This refers to the voltage at the high-voltage node. To inject power into active power, To inject power into reactive power, This is the topology of the high-voltage side distribution network system. High-voltage side moment The previous reconstruction plan, It is a set of nodes in the high-voltage side distribution network. Modeled in non-Euclidean graph form, node features are Topology connection by definition; The medium-pressure side agent at time Observations for, ， in, This is the topology of the medium-voltage side distribution network system. For the intermediate pressure side moment The previous reconstruction plan, This refers to the set of nodes in the medium-voltage distribution network. High-voltage and medium-voltage intelligent agents transmit voltage and power injection information from the feeder ends and photovoltaic access nodes to the substation intelligent agent, which observes... for, ， in, For a set of switches, For the set of end nodes of the feeder; In the real state space, the global environment state is: ， in, For the global distribution network system topology, A set of nodes; In the action space of the intelligent agent, the high-voltage side action is as follows: The action of the medium-pressure side is ; The substation intelligent agent action is represented as Its combinations include Species, among which This refers to the number of tap positions. Number of substations; All agents share the same global reward function. Its expression is, ， in, These are the weight coefficients for each term in the objective function; For branch set; The square of the branch current; Branch resistance; Square the node voltage; A collection of high-voltage transformers; This refers to the operating capacity of the high-voltage transformer. This refers to the rated capacity of the high-voltage transformer. For medium-voltage feeder sets; This refers to the active power of the feeder. Average active power; For a set of switches; It is in the on / off state; For a moment The on / off state at the previous moment; This is a collection of on-load tap-changing transformers; This refers to the tap position of an on-load tap-changing transformer. For a moment The previous switching status was the tap position of the on-load tap-changing transformer.

5. The method as described in claim 1, characterized in that: The overall architecture of the value decomposition-based "actor-evaluator" algorithm is as follows: set up Each agent contains a local policy network and a local evaluation network, which output discrete action distribution and local state value, respectively. The agent's local policy network and local evaluation network share a non-output layer. The central evaluation network of the control center receives the local state value and the global state as input and outputs the global state value. Each local reconstruction agent and the centralized evaluation network embeds a graph transformer layer to extract dynamic topological features.

6. The method as described in claim 5, characterized in that: The centralized evaluation network receives the values ​​of all local states and the global state, and outputs the global value function. for, ， in, for Global status of the time-period power distribution system; For intelligent agents At any moment State value; To ensure consistency between the global optimum and the local optimum, the following monotonicity relationship must be satisfied: ， The weights are generated by a supernetwork based on the global state, ensuring non-negativity and maintaining monotonicity; the centralized evaluation network is trained by minimizing the temporal difference error, and its parameters are updated as follows. ， in, For mixed network parameters, For learning rate, To guide the goal; The gradient for parameter update in the local policy network is: ， in, For local policy network parameters; It is the time-difference dominance function; This represents the action of the local agent at time t. For the observations of the local agent at time t; Further simplified to the standard "actor-evaluator" format, ， in, For overall discount rewards; This is the global value function.

7. The method as described in claim 5, characterized in that: Each locally reconstructing agent and the centralized evaluation network is preceded by an embedded graph transformer layer, specifically... Position encoding using the Laplacian matrix. ， in, For Laplace matrix, The eigenvector matrix, For the eigenvalue diagonal matrix, select the largest... Each feature value and its corresponding feature vector are used as a positional encoding; By using a multi-head attention mechanism and neighbor aggregation to update node features, the first... Nodes in the layer graph transformer Features Represented as: ， in, For nodes The neighborhood group, For the first The weight of each attention head, For value matrices, Indicates a splicing operation; For the first Layer nodes Features; The formula for calculating attention weights is: ， in, These are the query and key matrices, respectively. Dimensions for each attention head; After residual connection and normalization operations, the output of the graph transformer layer is: ， The output is concatenated with the action code from the previous moment and then sequentially input into a multilayer perceptron, a recurrent neural network with memory, a policy head, and a value head to generate a reconstructed policy and estimate the state value. The parameters of each graph transformer layer are updated synchronously with the parameters of the corresponding agent network or centralized evaluation network during training.

Images