Power distribution network joint layout optimization method based on reinforcement learning

Abstract

The invention discloses a power distribution network joint layout optimization method based on reinforcement learning, and the method comprises the following steps: S1, building a power distribution network initial graph model containing source-load distribution, and defining a joint planning Markov decision process of a line and a switch; s2, constructing a decision-making agent based on a graph convolutional network and near-end strategy optimization; and S3, carrying out topological constraint processing and model training based on a union-check set, maintaining connected components of nodes of the power distribution network through the union-check set, generating a dynamic topological constraint mask, and completing optimization training and scheme output of the joint layout of the power distribution network in combination with the decision-making agent. According to the reinforcement learning-based power distribution network joint layout optimization method provided by the invention, collaborative automatic planning of lines and switches can be realized, and the economical efficiency and the reliability of the power distribution network are remarkably improved on the premise of meeting various technical constraints.

Description

Technical Field

[0001] This invention relates to a method for joint layout optimization of power distribution networks based on reinforcement learning, belonging to the field of power system planning and operation technology. Background Technology

[0002] Distribution network planning aims to meet future load growth and grid development requirements by determining the timing, location, and type of lines, substations, or distributed generation (DG) equipment to be constructed. An active distribution network is a public distribution network with a flexible topology, employing a proactive management model for distributed generation, energy storage devices, and bidirectional loads from customers. The results of distribution network planning directly impact distribution network investment, returns, and the safety, economy, and stability of future distribution network operation. To rationally plan DG, coordinate its optimal operation, and fully leverage the positive role of DG and other new power sources and loads, active management and planning of the distribution network are necessary.

[0003] Currently, existing distribution network planning mainly relies on rule-based automated methods, heuristic algorithms (such as genetic algorithms and particle swarm optimization), or statistical multi-index evaluation methods (such as entropy weighting and cluster analysis), for example, the patent with authorization announcement number CN120749874B, "A Distribution Network Planning Method Adapted to Distributed New Energy Access." These methods typically guide the layout of line routes and equipment by pre-setting empirical rules or calculating sub-area index scores.

[0004] The existing technology has the following drawbacks: 1. Existing technologies often adopt a step-by-step logic of "classification or path planning first, then equipment configuration", or process the line corridor and switch layout in stages, ignoring the strong coupling relationship between the two, making it difficult to achieve global optimization of reliability and economy.

[0005] 2. Over-reliance on manually preset empirical rules or statistical indicators (such as cluster center scores, weight allocation, etc.) makes the weight setting subjective and fails to deeply explore the nonlinear mapping relationship between topology and operational results through end-to-end feedback mechanisms.

[0006] 3. Traditional statistical or clustering methods usually treat power parameters as independent features, lack explicit extraction of non-Euclidean structural features of distribution networks, and are difficult to make highly generalizable planning decisions when the topology evolves dynamically.

[0007] 4. When dealing with large-scale combinatorial optimization, traditional methods often face the "curse of dimensionality" and often use posterior verification or soft penalties to deal with "radial" physical constraints, resulting in low algorithm convergence efficiency and a high likelihood of generating invalid solutions. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a reinforcement learning-based method for joint layout optimization of distribution networks, which can realize the coordinated automatic planning of lines and switches, and significantly improve the economy and reliability of distribution networks while meeting various technical constraints.

[0009] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A reinforcement learning-based method for joint layout optimization of power distribution networks includes the following steps: Step S1: Establish an initial diagram model of the distribution network including source-load distribution, and define a Markov decision process for joint planning of lines and switches; Step S2: Construct a decision-making agent based on graph convolutional networks and near-end policy optimization; Step S3: Perform topology constraint processing and model training based on disjoint-set data structure. Maintain the connectivity components of the distribution network nodes through disjoint-set data structure, generate a dynamic topology constraint mask, and combine it with the decision-making agent to complete the optimization training and scheme output of the joint layout of the distribution network.

[0010] Furthermore, the joint planning Markov decision process in step S1 includes a state space S, an action space A, a reward function R, and a constraint handling mechanism. Furthermore, the state space S is represented by the mathematical model G = (V, E, B), where V is the set of node features; E is the combination of edge features; and B is the topological information. In the node feature set V, the feature vector of each node i The expression is as follows: ; in, For active load, For reactive load, For node type, To estimate the voltage; In the edge feature set E, the feature vector of each candidate line ij The expression is as follows: ; in, For line impedance, For the cost of line construction, This is the current state. Cost of switchgear.

[0011] Furthermore, the expression for the reward function R is as follows: ; in, As a construction cost incentive, As a reliability reward, As a reward for operation and wear and tear, To restrain punishment; Construction cost incentive The expression is as follows: ; in, The unit price for construction of the line per unit length; This refers to the line length; Unit price for the purchase and installation of switchgear; This refers to the total number of switches installed. Reliability Bonus The expression is as follows: ; in, The conversion factor for unit power outage losses. For the expected power shortage; Operation and Loss Rewards The expression is as follows: ; in, This is the unit network loss cost coefficient. Penalties for network damage; Restraint and Punishment This is used to provide a negative reward when the generated network does not meet voltage or capacity constraints.

[0012] Furthermore, the constraint processing mechanism is an action mask, which specifically includes the following steps: Before the agent selects an action, a mask vector M is calculated, and valid actions are selected through loop detection and island detection.

[0013] Furthermore, in step S2, constructing a decision agent based on graph convolutional networks and near-end policy optimization specifically includes the following steps: Step S21: Construct a distribution network state feature extraction module based on graph convolutional network to extract features from non-Euclidean spatial data of distribution network. Step S22: Design the Actor-Critic dual network structure; Step S23: Update network parameters using a near-end strategy optimization algorithm.

[0014] Furthermore, in step S21, a distribution network state feature extraction module based on graph convolutional networks is constructed to extract features from the non-Euclidean spatial data of the distribution network. Specifically, this includes the following steps: Step S211: Define the feature matrix and adjacency matrix ; The feature matrix The expression is as follows: ; Where N is the number of distribution network nodes, and F is the feature dimension of each node; The adjacency matrix The expression is as follows: ; Where N is the number of distribution network nodes; Constructing an adjacency matrix Enhanced adjacency matrix The enhanced adjacency matrix The expression is as follows: = B + ; in, It is the identity matrix; Step S212: Encode the state by stacking L layers of graph convolutional network. The formula for calculating the node embedding vector of the (l+1)th layer is as follows: ; in, Let l be the feature matrix of the l-th layer. = X; yes The degree matrix; It is the trainable weight matrix of the l-th layer; It is a non-linear activation function; Step S213: After extraction by the graph convolutional network layer, a node feature matrix containing topological information is obtained. The global state vector of the entire distribution network diagram is obtained by employing global average pooling or global max pooling operations. The global state vector The expression is as follows: ; in, This indicates vector concatenation; GlobalFeatures represents global information. This represents the node feature matrix obtained after convolution operation on the Lth layer graph. The process of performing aggregation.

[0015] Furthermore, in step S22, the Actor-Critic dual network structure is designed, specifically including the following steps: Design an Actor-Critic dual-network structure, which includes an actor network and a critic network, and the actor network and critic network share features extracted by graph convolutional network layers; The executor network inputs a global state vector. The original action log probabilities Logits are output through a fully connected layer, and then the action probability distribution is output by a Softmax layer. The expression of the action probability distribution is as follows; Action probability distribution = ; in, For the policy function, For action vectors; Define output dimension = K ; Where K is the number of candidate corridors and D is the size of the action discrete space; The critic network is input with a global state vector. The scalar value is output after passing through the fully connected layer, and the expression for the scalar value is as follows: scalar value = ; in, The value function of the critic network, This is the set of trainable parameters for the critic network.

[0016] Furthermore, in step S23, the network parameters are updated using a near-end strategy optimization algorithm, specifically including the following steps: Step S231: Calculate the advantage function using generalized advantage estimation. The advantage function The expression is as follows: ; in, For TD error, ; Step S232: Construct the total loss function The total loss function is jointly optimized by the executor network and the critic network. The expression is as follows: ; in, For the inclusion strategy ratio The truncation objective function; To minimize the value loss, mean squared error (MSE) is used to minimize the gap between the critics' predictions and the actual returns. ; To maximize policy entropy; As the value error loss weight, is the entropy regularization coefficient.

[0017] Furthermore, in step S3, topological constraint processing and model training based on disjoint-set data structure are performed. The disjoint-set data structure is used to maintain the connectivity components of the distribution network nodes, generating a dynamic topological constraint mask. This mask, combined with the decision-making agent, completes the optimization training and scheme output for the joint layout of the distribution network. Specifically, this includes the following steps: Step S31: Use a connectivity detection method based on disjoint-set data structure to determine in real time whether a planned route will form a closed loop; Step S32: Generate a dynamic topological constraint mask; Step S33: After the Actor network constructed in step S2 outputs the original action log probabilities Logits, the calculated mask M is applied to the original action log probabilities Logits. t .

[0018] By adopting the above technical solution, the present invention has the following beneficial effects: 1. Existing technologies typically decouple line planning from switch configuration, which can easily lead to local optima where the grid structure limits the effectiveness of switch configuration. This invention establishes a unified Markov decision process, incorporating line construction and switch location into the same action space for collaborative optimization. By simultaneously evaluating construction costs and potential fault isolation benefits during decision-making, the generated planning scheme achieves higher power supply reliability at the same cost.

[0019] 2. To address the rigid constraint that power distribution networks must meet a radial (loop-free) structure, existing technologies often employ penalty function methods, leading to low training efficiency and a tendency to generate invalid solutions. This invention utilizes a disjoint-set data structure to compute action masks in real time, eliminating invalid actions that could lead to loops or islands during the action generation stage. This not only ensures that the output solution 100% satisfies the physical constraints but also significantly reduces the search space, substantially improving the convergence speed of model training.

[0020] 3. Distribution networks have irregular graph structures, making it difficult for traditional fully connected or convolutional neural networks to effectively extract their topological features. This invention employs a graph convolutional network (GCN) as the state extraction layer for the agent, which can directly aggregate electrical connection information between nodes. Compared to existing technologies, this method can more accurately capture the relationship between load centers and key interconnection nodes, giving the planning algorithm stronger generalization ability when facing distribution networks of different sizes and structures.

[0021] 4. Existing algorithms often confuse the functions of segment switches and tie switches. This invention, through specific masking logic, allows the construction of normally open tie switches only when a potential loop is detected, while only segment switches are allowed to be constructed in non-connected areas. This mechanism ensures that the generated network inherently possesses N-1 transfer capability, eliminating the need for manual secondary verification and improving the engineering practicality of the solution. Attached Figure Description

[0022] Figure 1 This is a flowchart of the reinforcement learning-based distribution network joint layout optimization method of the present invention. Detailed Implementation

[0023] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0024] like Figure 1 As shown in the figure, this embodiment provides a joint layout optimization method for power distribution networks based on reinforcement learning, which includes the following steps: Step S1: Establish an initial diagram model of the distribution network including source-load distribution, and define a Markov decision process (MDP) for joint planning of lines and switches.

[0025] This step aims to transform the complex problem of joint planning of power distribution networks into a computer-processable mathematical model. By constructing a Markov Decision Process (MDP) that includes geographical topology, electrical states, and source-load distribution, a full-dimensional "digital sandbox" is provided for agent decision-making. Its technical advantages lie in achieving synchronous layout of lines and switches through heterogeneous action space definition, and explicitly linking economic costs and reliability indicators using a multi-objective composite reward function, laying the underlying logical foundation for subsequent global optimization.

[0026] The joint planning Markov decision process includes a state space S, an action space A, a reward function R, and a constraint handling mechanism.

[0027] The state space S represents a snapshot of the power grid layout at decision time t, including the set of erected lines, the location and status of installed switches, the network topology, and the load and voltage information of each node. The state space S is represented by the mathematical model G = (V,E, B), where V is the set of node features; E is the combination of edge features; and B is the topology information, representing the current adjacency matrix. In the node feature set V, the feature vector of each node i The expression is as follows: ; in, For active load, For reactive load, For node type, To estimate the voltage; In the edge feature set E, the feature vector of each candidate line ij The expression is as follows: ; in, For line impedance, For the cost of line construction, This is the current state. To reduce the cost of the switching equipment, the switching equipment in this embodiment includes: sectionalizing switches, such as automatic circuit breakers and load switches, and also tie switches, such as normally open contacts; The values ​​include 0 (not built), 1 (built and without a switch), 2 (built and with a sectional switch), and 3 (built and with a tie switch).

[0028] Action space A represents the operations that an agent can perform in state space S. It is a discrete action space. For each candidate corridor k, a composite action set is defined, which includes: a k =0 indicates that no line will be built; a k =1 indicates the construction line (ordinary conductor); a k =2 indicates that the line is constructed and sectionalizing switches (automatic circuit breakers / load switches) are installed. a k =3 indicates that the line was constructed and a tie switch (normally open) was installed.

[0029] The reward function R is the immediate reward obtained by the agent after performing an action, and it determines the direction of optimization. In this embodiment, we mainly consider the factors of economy and reliability. The expression of the reward function R is as follows: ; in, As a construction cost incentive, As a reliability reward, As a reward for operation and wear and tear, To restrain punishment; Construction cost incentive This represents the sum of line construction cost and switchgear cost, rounded to the negative. Construction cost incentive. The expression is as follows: ; in, The unit price for construction of the line per unit length; This refers to the line length; Unit price for the purchase and installation of switchgear; This refers to the total number of switches installed. Reliability Bonus The expression is as follows: ; in, The unit power outage loss conversion factor (an adjustment factor preset based on the sensitivity of the target power distribution area to power supply reliability). To minimize the expected power shortage, the more switches there are, the smaller the fault isolation range, the lower the EENS, and the smaller the penalty. Operation and Loss Rewards The expression is as follows: ; in, The unit network loss cost coefficient (considering the economic expenditure weight corresponding to the power loss during the entire life cycle of the distribution network, used to guide the intelligent agent to optimize the network structure to reduce operating losses). Penalties for network damage; Restraint and Punishment This is used to provide a large negative reward when the generated network does not meet voltage or capacity constraints.

[0030] The constraint handling mechanism is action masking, which includes the following steps: Before the agent selects an action, a mask vector M is calculated, and valid actions are filtered through loop detection and island detection. Loop formation detection is as follows: if an action would cause a closed loop to form in a non-tethering switch line, then the action is masked (so that its normalized sampling probability is strictly 0). Islanding detection ensures that all load points can eventually be connected to the power source.

[0031] Step S2: Construct a decision agent based on graph convolutional network (GCN) and proximal policy optimization (PPO). The decision agent adopts an Actor-Critic architecture. The feature extraction part of the decision agent uses graph convolutional network, and the decision optimization part uses proximal policy optimization algorithm.

[0032] This step aims to create a decision-making brain with spatial awareness and policy evolution capabilities. By introducing Graph Convolutional Networks (GCNs) to extract features from non-Euclidean spatial data of the distribution network, the "curse of dimensionality" problem under large-scale nodes is effectively solved, enabling the agent to identify critical connection nodes affecting system reliability. Combined with the executor-critic architecture of the PPO algorithm, an end-to-end mapping from perceived topology to outputting the optimal joint layout scheme is achieved, significantly improving the generalization ability and optimization accuracy of the planning scheme. Specifically, the steps include: Step S21: Construct a distribution network state feature extraction module based on graph convolutional networks to extract features from the non-Euclidean spatial data of the distribution network. Specifically: Step S211: Define the feature matrix and adjacency matrix : Feature matrix The expression is as follows: ; Where N is the number of distribution network nodes, and F is the characteristic dimension of each node (including node active and reactive loads, voltage amplitude, node type code, etc.).

[0033] Adjacency Matrix Used to describe the current line connection status of the distribution network. Its expression is as follows: ; Where N is the number of distribution network nodes; Considering the characteristics of the nodes themselves, construct an adjacency matrix. Enhanced adjacency matrix Enhanced Adjacency Matrix The expression is as follows: = + ; in, It is an identity matrix, with 1s on the main diagonal and 0s elsewhere. During convolution, feature aggregation and numerical stability are achieved by making each node self-loop.

[0034] Step S212: Encode the state by stacking L layers of graph convolutional network. The formula for calculating the node embedding vector of the (l+1)th layer is as follows: ; in, Let l be the feature matrix of the l-th layer. = X; yes The degree matrix is ​​used for normalization to prevent numerical explosion; It is the trainable weight matrix of the l-th layer; It is a non-linear activation function (such as ReLU).

[0035] Step S213: After extraction by the graph convolutional network layer, a node feature matrix containing topological information is obtained. The global state vector of the entire distribution network diagram is obtained by employing global average pooling or global max pooling operations. Global state vector The expression is as follows: ; in, This indicates vector concatenation; GlobalFeatures contains global information, including the current total cost, number of construction steps, etc. This represents the node feature matrix obtained after convolution operation on the graph at the Lth layer (i.e., the last layer). The aggregation process primarily employs global average pooling to calculate the average value of all node feature vectors. The aim is to compress the feature matrix into a fixed-length vector, thereby extracting a descriptor representing the global topological characteristics of the entire distribution network.

[0036] Step S22: Design the Actor-Critic dual network structure. Specifically: The Actor-Critic dual-network architecture includes an Actor Network and a Critic Network, which share features extracted by graph convolutional network layers. The executor network is used to determine, in the current state, which construction action to take for each candidate corridor (e.g., no construction, construction of the line, construction of the line plus segmented switching, construction of the line plus tie-line switching). The executor network is input to the global state vector. The original action log probabilities Logits are output through a fully connected layer (MLP), and then the action probability distribution is output by a Softmax layer. The expression for the action probability distribution is as follows; Action probability distribution = ; in, The policy function represents a random policy defined by the agent network, which establishes a mapping from the state space to the action space and determines the logical rules by which the agent arranges the circuits and switches. The parameter set of the executor network represents all trainable weights and biases. Let be the action vector, representing the specific planning action executed by the decision-making agent at time t (such as choosing "build the line" or "do not build the line" for a candidate route). The global state vector represents the environmental state input at time t, generated by step S213 above.

[0037] Define output dimension = K ; Where K is the number of candidate corridors, which in this embodiment refers to the total number of all preset, possible construction routes; D represents the size of the discrete space of the action. In this embodiment, D=4 is preferred. These four dimensions correspond to the four sets of composite actions mentioned in step S1.

[0038] The critic network is used to evaluate the current global state vector. The value of this, namely the expected cumulative reward in the future, is used to assist the agent in updating its policy to reduce variance. The critic network is input with the global state vector. The fully connected layer (MLP) outputs a scalar value, which is the commentator network's view of the current global state vector of the distribution network. The score is used to evaluate the total cumulative reward that the agent is expected to obtain in the future under the current layout. The scalar value is expressed as follows: scalar value = ; in, The value function of the commentator network is used to quantify the long-term expected return of the current global state of the distribution network; The set of trainable parameters for the critic network helps the executor network update its planning policy more stably by minimizing the value estimation error during training.

[0039] Step S23: To ensure training stability and avoid convergence failure due to excessively large policy update steps, the Proximal Policy Optimization (PPO) algorithm is used to update network parameters. Specifically: Step S231: Calculate the advantage function using generalized advantage estimation (GAE). The advantage function is used to measure how well an action is performed compared to the average level. The expression is as follows: ; in, For TD error, .

[0040] Step S232: Construct the total loss function Jointly optimize the Actor Network and Critic Network, with the total loss function... The expression is as follows: ; in, For the inclusion strategy ratio The truncation objective function is used to limit the ratio of the new and old strategies; In [1- , 1+ Within a certain range, prevent drastic fluctuations in the strategy; To minimize the value loss, mean squared error (MSE) is used to minimize the gap between the critics' predictions and the actual returns. ; To maximize policy entropy, agents are encouraged to explore more possible layout schemes in the early stages to avoid getting trapped in local optima too early. This is the value error loss weight, used to balance the estimation accuracy of the value function; is the entropy regularization coefficient, used to maintain the agent's exploratory ability by maximizing policy entropy.

[0041] Step S3: Perform topology constraint processing and model training based on Union-Find. Maintain the connectivity components of distribution network nodes through Union-Find, generate dynamic topology constraint masks, and combine them with decision-making agents to complete the optimization training and scheme output of the joint layout of the distribution network.

[0042] This step aims to ensure that AI decisions strictly conform to the realities of power engineering through a hard constraint filtering mechanism. The Union-Find algorithm is used to monitor the connectivity of the evolving topology in real time, and action masking technology is combined to eliminate invalid operations that could lead to illegal loops or isolated networks before decision-making. The technical effect of this step is that it guarantees 100% satisfaction of the radial operational constraints at the algorithmic level, while significantly accelerating the model's training and convergence process exponentially by pruning the invalid search space. Specifically, it includes the following steps: Step S31: Employ a connectivity detection method based on disjoint-set data structure to determine in real time whether a planned route will form a closed loop. Specifically: In order to determine in real time whether a planned line will form a closed loop, this embodiment introduces a disjoint-set data structure to maintain the connectivity components of the distribution network nodes.

[0043] Define an array parent, where parent[i] represents the parent node of node i; initially, all nodes are not connected to each other, so let parent[i] = i; To speed up the query, path compression is used to find the root node of the set to which node i belongs. The expression for the search operation is as follows: ; When constructing a line between nodes i and j, the two sets are merged. The expression for the merge operation is as follows: .

[0044] Step S32: Generate dynamic topology constraint mask (Action Mask Generation). Specifically: Before inputting the state into the neural network at each decision step t of the agent, the mask vector M is first calculated based on the current topology. t M t It is a binary vector with the same dimension as the action space (0 represents prohibition, 1 represents permission); for each candidate corridor k (connecting nodes u and v), the mask generation rule is determined based on the disjoint-set data structure. The mask generation rule includes: Rule 1: Ring Network Testing (for ordinary lines and sectionalizing switches): Calculate Root u = Find(u) and Root v = Find(v); If Root u = Root v This indicates that nodes u and v are already in the same electrically connected area. In this case, if ordinary lines or sectionalizing switches are constructed, an electrical closed loop will be formed, violating the radial operation principle. Therefore, the following masking formula is used for shielding: Mask[k, action: line creation] = 0; Mask[k, action: create segment switch] = 0; Rule 2, Island Elimination Guide: If the root node is Root u and Root v If none of the connected components represented by a candidate corridor k contain power nodes (such as substation nodes), then the construction action corresponding to that candidate corridor k is subjected to probability downweighting or temporary masking. This approach aims to guide the agent to prioritize extending the planned route outward from the connected region containing power nodes through dynamic constraints on the action space, thereby ensuring real-time connectivity during the network evolution process.

[0045] Step S33: Combining the mask with the near-end policy optimization (PPO) algorithm network. Specifically: After the Actor network constructed in step S2 outputs the original action log probabilities Logits, the calculated mask M is applied to the original action log probabilities (Logits). t The expression for the mask operation is as follows: ; in, The output vector after topological constraint correction is used as the input to the next Softmax normalization layer. This represents the raw values ​​output by the fully connected layer (MLP) in step S22. It indicates the agent's initial tendency towards each planned action without considering physical constraints. In actual code, a very small negative number (such as -10) is usually chosen. 9 ); This formula processes values ​​in the following way, ensuring that the probability of performing an illegal action is 0: 1. For legal actions =1, Logits masked = Logits raw The original probability remains unaffected; 2. Regarding illegal actions =0, Logits masked = Logits raw -10 9 The probability is processed into a minimum value, and this minimum value is then normalized using Softmax. The corresponding probability is... Strictly approximates 0; The Softmax normalization calculation expression is as follows: .

[0046] This ensures that the agent will never sample actions that violate topological constraints, guaranteeing 100% physical feasibility of the output solution.

[0047] The specific embodiments described above further illustrate the technical problems, technical solutions, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A power distribution network joint layout optimization method based on reinforcement learning, characterized in that, It comprises the following steps: Step S1, establishing an initial graph model of a power distribution network containing source-load distribution, and defining a joint planning Markov decision process of lines and switches; Step S2, constructing a decision-making agent based on a graph convolution network and a proximal policy optimization; Step S3, performing topological constraint processing and model training based on a union-find set, maintaining the connected components of the nodes of the power distribution network by the union-find set, generating a dynamic topological constraint mask, and combining the decision-making agent to complete the optimization training and scheme output of the joint layout of the power distribution network.

2. The method of claim 1, wherein, The joint planning Markov decision process in the step S1 comprises a state space S, an action space A, a reward function R, and a constraint processing mechanism.

3. The method of claim 2, wherein, The state space S is represented by a mathematical model G = (V, E, B), wherein V is a node feature set; E is an edge feature combination; and B is topological information. The feature vector of each node i in the node feature set V The expression is as follows: ； wherein, is the active load, is the reactive load, is the node type, is the estimated voltage; The feature vector of each candidate line route ij in the edge feature set E The expression is as follows: ； wherein, is the line impedance, is the line construction cost, is the current state, is the switching device cost.

4. The method of claim 2, wherein, The expression of the reward function R is as follows: ； wherein, is a construction cost reward, is a reliability reward, is an operation and wear reward, is a constraint penalty; Construction cost incentive The expression is as follows: ； wherein, Construction unit price of per unit length line; L is the length of the line; Purchase and installation unit price for switchgear; is the total number of switches to be installed; Reliability rewards The expression of the above is as follows: ； wherein, is the unit outage loss conversion factor, is the expected amount of power not supplied; Running and loss rewards The expression for the running and loss rewards is as follows: ； wherein, is the unit network loss cost coefficient, is the network loss penalty; Constraint penalty to give negative reward when the generated network does not satisfy voltage constraints or capacity constraints.

5. The method of claim 2, wherein, The constraint processing mechanism is an action mask, which specifically comprises the following steps: Before the agent selects an action, a mask vector M is calculated to filter valid actions through loop detection and island detection.

6. The method of claim 1, wherein, In the step S2, the decision-making agent based on the graph convolution network and the proximal policy optimization is constructed, which specifically comprises the following steps: Step S21, constructing a power distribution network state feature extraction module based on a graph convolution network to extract features from non-Euclidean space data of the power distribution network; Step S22, designing an Actor-Critic double network structure; Step S23, updating network parameters by using a proximal policy optimization algorithm.

7. The method of claim 6, wherein, In the step S21, the power distribution network state feature extraction module based on the graph convolution network is constructed to extract features from non-Euclidean space data of the power distribution network, which specifically comprises the following steps: Step S211, defining a feature matrix and an adjacency matrix ; The feature matrix The expression is as follows: ； Wherein N is the number of nodes of the power distribution network, and F is the feature dimension of each node. The adjacency matrix The expression is as follows: ； Wherein N is the number of nodes of the power distribution network. Constructing an adjacency matrix An enhanced adjacency matrix An enhanced adjacency matrix The expression of the enhanced adjacency matrix is as follows: = B + ； wherein is the identity matrix; Step S212, encoding the state by stacking L layers of graph convolution network layers, and the calculation formula of the node embedding vector of the (l+1)th layer is as follows: ； wherein is a feature matrix for the lth layer, = X; is a degree matrix; is a trainable weight matrix of the first layer; is a non-linear activation function; Step S213, after passing through the graph convolution network layer, a node feature matrix containing topological information is obtained ; a global state vector of the entire power distribution network graph is obtained by using a global average pooling or global maximum pooling operation , and an expression of the global state vector is as follows: ； wherein, denotes vector concatenation; GlobalFeatures is global information; denotes a process of performing aggregation processing on a node feature matrix obtained after L-th layer graph convolution operation denotes a process of performing aggregation processing on a node feature matrix obtained after L-th layer graph convolution operation 8. The method of claim 6, wherein, In the step S22, the Actor-Critic double network structure is designed, which specifically comprises the following steps: The Actor-Critic double network structure comprises an actor network and a critic network, and the actor network and the critic network share the features extracted by the graph convolution network layer; The actor network inputs a global state vector The original action pair Logits are output by a fully connected layer, and then the action probability distribution is output by a Softmax layer, and the expression of the action probability distribution is as follows: Action probability distribution = f(a | s, θ) ; wherein, is a policy function, is an action vector; define output dimensions = K ; Wherein K is the number of candidate corridors, and D is the size of the action discrete space. The critic network inputs a global state vector through a fully connected layer outputs a scalar value, whose expression is as follows: Scalar value = 0x00000000 ; wherein, is a value function for the critic network, is a set of trainable parameters for the critic network.

9. The method of claim 6, wherein, In the step S23, the network parameters are updated by using the proximal policy optimization algorithm, which specifically comprises the following steps: Step S231, calculating the advantage function using generalized advantage estimation , the expression of the advantage function is as follows: ； wherein is the TD error, ; Step S232, constructing a total loss function , jointly optimizing the performer network and the critic network, an expression of the total loss function is as follows: ； wherein, is a truncated objective function comprising a policy ratio ​ For value loss, mean squared error (MSE) is used to minimize the difference between critic prediction and true return, ; to maximize the policy entropy; is a value error loss weight, is an entropy regularization coefficient.

10. The method of claim 1, wherein, In the step S3, the topological constraint processing and model training based on the union-find set are performed, the connected components of the nodes of the power distribution network are maintained by the union-find set, a dynamic topological constraint mask is generated, and the optimization training and scheme output of the joint layout of the power distribution network are completed in combination with the decision-making agent, which specifically comprises the following steps: Step S31, using connectivity detection based on the union-find set to determine whether a planned line will form a closed loop in real time; Step S32, generating a dynamic topological constraint mask; Step S33, after the Actor network constructed in the step S2 outputs the original action pair log probability Logits, applying the calculated mask M to the original action pair log probability Logits t .

Images