Power distribution area topology optimization generation method based on deep reinforcement learning

Abstract

The invention relates to the technical field of power system automation, in particular to a power distribution area topology optimization generation method based on deep reinforcement learning, and the method comprises the steps: obtaining the number of area users, the power supply radius and the node spacing of a power distribution network, and coding a topological structure and dynamic operation characteristics through a dual-channel state matrix; based on a deep reinforcement learning network, an action space of a power distribution network branch decision is decoupled as a two-dimensional decision combination of trunk node selection and branch extension length, a power grid topology is dynamically updated through an intelligent agent based on an action decision result, and a multi-stage priority strategy is adopted to distribute user loads; and setting a composite reward function, and training the deep reinforcement learning network to converge by adopting a progressive training strategy to obtain an optimal topological structure. Compared with the prior art, the method is based on deep reinforcement learning, through intelligent agent dynamic decision branch extension and user distribution, optimal topology generation under power supply radius constraint is realized, line loss is effectively reduced, and real-time access of distributed energy is supported.

Description

Technical Field

[0001] The present invention relates to the technical field of power system automation, and in particular to a power system automation technology field. Background Art

[0002] The topological optimization of distribution transformer areas is the core technology to improve the power supply reliability of the power grid and reduce line losses. The current methods rely on manual experience or static algorithms, and there are problems such as poor dynamic response ability, unreasonable branch configuration, and unbalanced user distribution, resulting in high line loss rates and low resource utilization rates.

[0003] For example, Chinese Patent Application CN202411782677.5 discloses a method for reconstructing the topology of a distribution network based on improved DQN. In this scheme, a state space based on historical load is constructed according to the autocorrelation of the load time series, and the topological information and load characteristics are not separated and expressed, resulting in the inability to reflect the influence of the node depth change on the load characteristics with the topological expansion in this scheme. And it changes the action object of the model decision from the topological structure to the action switch, there are problems of insufficient extraction of topological structure features and non-support for dynamic expansion of branches. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above-mentioned deficiencies in the prior art, such as insufficient extraction of topological structure features and non-support for dynamic expansion of branches, and provide a method for generating topological optimization of distribution transformer areas based on deep reinforcement learning.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A method for generating topological optimization of distribution transformer areas based on deep reinforcement learning, the steps include: Obtain the physical parameters of the distribution network, including the number of users in the transformer area, the power supply radius, and the node spacing, convert the physical parameters of the power grid into the state space, and encode the topological structure and dynamic operation characteristics through a dual-channel state matrix; Input the dual-channel state matrix into the deep reinforcement learning network, decouple the action space of the distribution network branch decision into a two-dimensional decision combination of main trunk node selection and branch extension length, dynamically update the power grid topology through the agent based on the action decision result, and adopt a multi-stage priority strategy to allocate user loads; Set a composite reward function in the deep reinforcement learning network to guide the agent to balance fast power supply and reduce line losses, train the deep reinforcement learning network to convergence by an incremental training strategy, and output and generate the optimal topological structure.

[0006] As a preferred technical solution, the dual-channel state matrix includes two channels representing the topological structure and dynamic operation characteristics respectively, and the state matrix is expressed as follows: , In the formula, is the adjacency matrix, is the user distribution feature, is the depth feature, is the total number of nodes in the substation area; The adjacency matrix is used to represent the topological structure of the substation area: , In the formula, , represents that there is a connection line between node and node , represents that there is no connection line between node and node ; The user distribution feature of node is: , The depth feature is the number of hops from node to the nearest backbone node.

[0007] As a preferred technical solution, the deep reinforcement learning network extracts topological space features through a dual-channel convolutional neural network, performs value prediction through a fully connected layer, and adopts a gradient clipping and weight decay strategy to improve the generalization ability of the model: The adjacency matrix in the input dual-channel state matrix and the feature matrix containing user distribution and node depth are used to extract spatial features through two convolutional layers with different convolutional kernel sizes respectively, and the feature maps are uniformly output through adaptive average pooling; The feature maps are flattened and gradually reduced in dimension through three fully connected layers. A random inactivation layer is added after the second fully connected layer to prevent overfitting, and the final output dimension is the size of the action space; The loss function is designed as follows: , In the formula: is the predicted value, that is, the output of the deep reinforcement learning network; is the target value, that is, the value calculated by the target network, and B is the batch size.

[0008] As a preferred technical solution, the reward function of the deep reinforcement learning network is as follows: , , , , , wherein, is the stability reward, is the nominal voltage value, is the th node voltage of the distribution substation area; is the economic reward, is the th equivalent resistance of the line, is the th current value of the line; is the risk penalty, is the load factor of node , is the current value of line , is the current value of line 's rated current-carrying capacity.

[0009] As a preferred technical solution, the deep reinforcement learning network adopts a three-stage training mechanism to dynamically adjust the exploration rate and the reward weight ; The exploration rate decay formula is shown as follows: , wherein, is the exploration rate of the th round, which is used to control the proportion of random actions; is the number of training rounds; Based on the increase of the training rounds, the reward weights of each stage of training are successively reduced : Set an early stopping mechanism. When the sum of the absolute values of the differences between the reward values of each round and the average reward value is less than the set reward value difference threshold within the set number of rounds before the previous round, the training stops and the generated topological structure is output.

[0010] As a preferred technical solution, the two-dimensional decision combination of the backbone node selection and the branch extension length is represented as follows: , wherein, represents the floor function; [[ID= , representing a high-dimensional discrete action space; is the maximum number of branches of a single backbone, , is the number of nodes in the backbone line.

[0011] As a preferred technical solution, the action space embeds the physical constraints of the power grid to verify the effectiveness of the constraints, and excludes invalid actions through rule screening, as shown in the following formula: , In the formula, is the current number of nodes, that is, the number of created nodes, and l is the branch extension length, that is, the number of newly added nodes in a single extension.

[0012] As a preferred technical solution, the dynamic update of the power grid topology based on the action decision result includes: creating new nodes according to the action decision, and updating the adjacency matrix and node depth, as shown in the following formula: , In the formula, is the backbone node number, is the new node, is the current node, is the branch extension length, is the node depth.

[0013] As a preferred technical solution, the multi-stage priority strategy distributes the user load, based on real-time load prediction and dynamic priority adjustment, to achieve intelligent distribution of the user load to ensure the balanced operation of the power grid, specifically as follows: Calculate the node load rate : , In the formula, is the current load power of node , is the rated capacity of node ; Conduct node priority evaluation: , In the formula, is the node depth, the backbone node is 0, and the branch node is the hop count; is the voltage deviation; ， ， is the weight coefficient, which is dynamically adjusted through reinforcement learning; The user access principle is as follows: New users access the node with the highest priority. If the priorities of multiple nodes are the same, select the node with a smaller depth. When the node load rate is greater than the set node load rate threshold, new user access is rejected, and branch expansion is triggered. The branch expansion mechanism is as follows: When a continuous set number of users are rejected access due to node overload, select the backbone node with the largest available capacity to expand the branch. If the node voltage deviation is greater than the set node voltage deviation threshold, remove the node from the candidate list.

[0014] As a preferred technical solution, dynamically adjusting the weight coefficient in the node priority evaluation specifically is: In a high-load scenario, that is, when the average node load rate is greater than the set average load rate threshold, increase the weight coefficient α , and preferentially select low-load nodes; In a low-voltage scenario, that is, when the average node voltage deviation is greater than the set average voltage deviation threshold, increase the weight coefficient γ , and preferentially select nodes with stable voltage.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1) The present invention proposes a method for generating a topology optimization of a distribution substation area based on deep reinforcement learning. The agent creates new nodes based on the action decision results, updates the adjacency matrix and node depth, and dynamically updates the power grid topology. And a multi-stage priority strategy is adopted to allocate user loads. Based on real-time load prediction and dynamic priority adjustment, intelligent allocation of user loads is realized to ensure the balanced operation of the power grid. Ensure the optimal matching of branch expansion and load distribution, solve the problem of separated optimization of topology and load in traditional methods. Realize the generation of the optimal topology under the constraint of the power supply radius, effectively reduce line losses, and support the real-time access of distributed energy.

[0016] 2) The present invention converts the physical parameters of the power grid into a state space that can be processed by deep reinforcement learning. By encoding the topology structure and dynamic operation characteristics through a dual-channel matrix, the problem of separate expression of topology information and load characteristics in traditional methods is solved. And the discrete action space is decoupled into a two-dimensional combination of backbone node selection - branch expansion decision, reducing the decision complexity through mathematical mapping, and at the same time being able to embed the physical constraint rules of the power grid. Reduce the original action dimension by more than 50%, and be able to rigidly constrain to ensure that all actions comply with the power grid safety specifications.

[0017] 3) The present invention designs a composite reward function, which quantifies the power grid operation quality by combining multi-dimensional indicators such as construction rewards, depth rewards, and empty branch penalties, balances economy, stability, and user experience, guides the agent to balance short-term benefits (fast power supply) and long-term goals (reducing line losses), and guides the agent to generate the optimal topology.

[0018] 4) The topology optimization generation network of the present invention adopts gradient clipping and weight decay strategies to improve the model generalization ability and solve the problem of insufficient extraction of topology structure features by traditional DQN networks. At the same time, the topology optimization generation network adopts a progressive training strategy, dynamically adjusts the exploration rate and reward weights through a three-stage training mechanism, focuses on topology exploration in the early stage, balances exploration and exploitation in the middle stage, and converges to the optimal strategy in the later stage. And to reduce the number of ineffective training rounds, an early stopping mechanism is set. Solve the problems of slow convergence and easy to fall into local optimum in traditional DRL training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of the distribution substation area topology optimization generation method based on deep reinforcement learning of the present invention; Figure 2 is a diagram of the distribution substation area topology generation result in a specific embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and gives detailed implementation manners and specific operation processes, but the protection scope of the present invention is not limited to the following embodiments.

[0021] Embodiment 1 Step 1, environmental modeling and state encoding, convert the physical parameters of the power grid into a state space that can be processed by deep reinforcement learning, and encode the topology structure and dynamic operation characteristics through a dual-channel matrix to solve the problem of separate expression of topology information and load characteristics in traditional methods.

[0022] Step 2, hierarchical action space design, aiming at the high-dimensional discrete characteristics of the distribution network branch decision, decouple the action space into a two-dimensional combination of "main trunk node selection - branch extension decision", reduce the decision complexity through mathematical mapping, and at the same time embed the physical constraint rules of the power grid.

[0023] Step 3, topology update and user allocation, dynamically update the power grid topology based on the action decision result, adopt a multi-stage priority strategy to allocate user loads, ensure the optimal matching of branch extension and load distribution, and solve the problem of separate optimization of topology and load in traditional methods.

[0024] Step 4, Composite Reward Function Design: By non-linearly combining multi-dimensional indicators such as construction rewards, deep rewards, and empty branch penalties, the topological quality is quantitatively evaluated to guide the agent to balance short-term benefits (rapid power supply) and long-term goals (reducing line losses).

[0025] Step 5, Network Design and Optimization: Construct a dual-channel convolutional neural network (CNN) to extract topological space features, combine with a fully connected layer to achieve Q-value prediction, and improve the generalization ability of the model through gradient clipping and weight decay strategies.

[0026] Step 6, Progressive Training Strategy: Adopt a three-stage training mechanism to dynamically adjust the exploration rate and reward weights. In the early stage, it focuses on topological exploration, in the middle stage, it balances exploration and exploitation, and in the later stage, it converges to the optimal strategy, solving the problems of slow convergence and easy entrapment in local optima in DRL training.

[0027] The specific implementation of each step is as follows: Step 1: Environment Modeling and State Encoding: Convert the physical parameters of the power grid into a state space that can be processed by deep reinforcement learning, and encode the topological structure and dynamic operation characteristics through a dual-channel matrix to solve the problem of separate expression of topological information and load characteristics in traditional methods.

[0028] 1.1, Parameter Initialization According to the power supply radius and the node spacing calculate the number of nodes on the main line of the distribution area : ， In the formula, represents the floor function.

[0029] To avoid incomplete topological results, set the following constraint conditions: ， That is, ensure that the number of nodes on the main line of the distribution area is at least 5.

[0030] ， That is, the total number of nodes in the distribution area is greater than the number of nodes on the main line.

[0031] 1.2, Dual-Channel State Matrix Construction The dual-channel state matrix includes two channels representing topological structure and operation characteristics respectively.

[0032] For channel 1, an adjacency matrix A is used to represent the topological structure of the distribution area: ， [[ID= , represents a node and node have a connection line, represents a node and node do not have a connection line between them, A is the adjacency matrix with dimension , is the total number of nodes in the substation area.

[0033] Corresponding to Channel 2, it includes user distribution characteristics and depth characteristics, where the user distribution characteristics are: ， The depth characteristic is the hop count from node to the nearest backbone node, which is used to reflect the depth of the branch extension and suppress over-expansion.

[0034] The finally obtained state matrix is shown as follows: ， By separating the static topology and dynamic load characteristics through the two-channel in Step 1, the DRL learning efficiency is improved. Dynamic normalization adapts to the depth changes during the topology expansion process.

[0035] The main program to implement Step 2 is as follows: class PowerGridEnv: def __init_(self, params): self.max_main_nodes = int(params['supply_radius'] / / params['node_interval']) assert self.max_main_nodes>= 5, "The number of backbone nodes is insufficient. Please adjust the supply radius or node spacing." assert params['max_total_nodes']>self.max_main_nodes, "The total number of nodes must be greater than the number of backbone nodes." def _get_state(self): state = np.zeros((2, self.max_total_nodes, self.max_total_nodes),dtype=np.float32) state[0] = self.adj_matrix # Adjacency matrix max_depth = max(self.branch_depth.values()) if self.branch_depth else 1 for node in range(self.current_nodes): if node < self.max_main_nodes: capacity = 3 # Main node capacity else: capacity = 4.8 # Branch node capacity (increased by 20%) state[1, 0, node] = self.user_distribution[node] / capacity # User distribution state[1, 1, node] = self.branch_depth.get(node, 0) / (max_depth + 1e - 5) # Node depth return state Step 2: Hierarchical action space design 1) Action space decomposition Decouple the discrete action space into a two - dimensional decision of "main node selection + branch extension length" to reduce the action dimension. As shown in the following formula: ， In the formula, is the original action number, , representing the high - dimensional discrete action space; is the maximum number of branches of a single main node, .

[0036] 2) Validity constraint verification Exclude invalid actions (such as node 0 branches, excessive number of branches, etc.) through rule screening. As shown in the following formula: ， In the formula, is the current number of nodes, that is, the number of nodes that have been created, and l is the branch extension length, that is, the number of nodes newly added in a single extension.

[0037] The two - dimensional action space in Step 2 reduces the original action dimension by more than 50%. Hard constraints ensure that all actions comply with the grid safety specifications.

[0038] The main procedures for implementing Step 2 are as follows: max_branch = int(0.5 * self.max_main_nodes) main_node = action / / (max_branch + 1) branch_type = action % (max_branch + 1) def _is_valid(self, main_node, branch_type): if main_node == 0 and branch_type>0: return False # Node 0 prohibits branch expansion if self.main_branches[main_node]>= 5: return False # The number of single main branches exceeds the upper limit if self.current_nodes + branch_type>self.max_total_nodes: return False # Exceeds the total node capacity return True Step 3: Topology update and user allocation 3.1) Branch expansion algorithm Create new nodes according to the action decision, and update the adjacency matrix and node depth. As shown in the following formula: ， In the formula, is the main node number, is the branch expansion length, is the node depth.

[0039] 3.2) User allocation strategy Based on real-time load prediction and dynamic priority adjustment, realize the intelligent allocation of user loads to ensure the balanced operation of the power grid.

[0040] Calculate the node load rate : ， In the formula, is the current load power of node For the node Rated capacity.

[0041] Then, perform node priority evaluation: ， Wherein: is the node depth (the backbone node is 0, and the branch node is the hop count), is the voltage deviation coefficient, ， ， is the weight coefficient, which is dynamically adjusted through reinforcement learning.

[0042] The dynamic weight adjustment is as follows: High-load scenario ( > 0.8): Increase the α weight and preferentially select low-load nodes.

[0043] Low-voltage scenario ( > 5%): Increase the γ weight and preferentially select nodes with stable voltage.

[0044] The user access principle is: New users access the node with the highest priority. If the priorities of multiple nodes are the same, select the node with a smaller depth (closer to the backbone). When the node load rate > 0.9, automatically reject new user access and trigger branch capacity expansion. If the node voltage deviation > 10%, remove the node from the candidate list.

[0045] Set the automatic capacity expansion mechanism as follows: The trigger condition is that 3 consecutive users are rejected access due to node overload. When the condition is met, select the backbone node m with the largest available capacity to expand the branch, and the branch length = 1 - 3.

[0046] Step 4: Design the composite reward function, quantify the power grid operation quality through multi-dimensional reward terms, balance economy, stability and user experience, and guide the agent to generate the optimal topology.

[0047] The reward function is designed as shown in the following formula: ， Wherein: is the stability reward: ， Wherein: The nominal value is taken as 400V, is the node voltage, so that the closer the node voltage is to the nominal value, the higher the reward.

[0048] For economic rewards: ， ， In the formula, is the equivalent resistance of the i-th line, is the current value of the i-th line.

[0049] For risk penalties: ， In the formula, is the load rate of node i, is the current value of line j, is the rated current-carrying capacity of line j.

[0050] To achieve the penalty for overload phenomenon.

[0051] Step 5, Design and optimization of the dual-channel convolutional neural network, construct a dual-channel convolutional neural network (CNN) to extract topological space features, combine with a fully connected layer to achieve Q-value prediction, and improve the generalization ability of the model through gradient clipping and weight decay strategies to solve the problem of insufficient extraction of topological structure features by traditional DQN networks.

[0052] 5.1) Network architecture design Dual-channel input processing: Input1: Adjacency matrix A ∈ ; Input2: Feature matrix F ∈ (User distribution + node depth); Table 1 Network structure of the dual-channel convolutional neural network Network layer Output dimension Input (2, N, N) Conv2d → ReLU → BN (32, N, N) Conv2d → ReLU (64, N, N) AdaptiveAvgPool2d (64, 8, 8) Flatten 4096 (i.e., 64×8×8) Linear 512 Dropout 512 Linear 256 Output num_actions For the designed dual-channel convolutional neural network, its design features are as follows: The input is a dual-channel N×N matrix (i.e., input1 and input2), spatial features are extracted through two convolutional layers (using 32 and 64 3×3 convolutional kernels respectively, retaining the resolution and introducing ReLU non-linear activation), and are uniformly output as a 64-channel 8×8 feature map through adaptive average pooling, then flattened and gradually reduced in dimension through a fully connected layer (4096→512→256), and 30% Dropout is added after the 512-dimensional layer to prevent overfitting, and the final output dimension is the size of the action space.

[0053] 5.2) Optimization strategy Gradient clipping mechanism: ， Wherein: is the gradient, representing the parameter update direction in backpropagation; is the gradient clipping threshold, used to prevent gradient explosion.

[0054] Weight decay: ， Wherein: is the regularization coefficient, used to control the model complexity and suppress overfitting; is the network weight.

[0055] The loss function is designed as follows: ， Wherein: is the predicted Q value, i.e., the output of the deep reinforcement learning network; is the target Q value, i.e., the calculated value of the target network, and B is the batch size.

[0056] Step 6, Progressive training strategy, adopting a three-stage training mechanism to dynamically adjust the exploration rate ( ) and the reward weight ( ), focusing on topological exploration in the early stage, balancing exploration and exploitation in the middle stage, and converging to the optimal strategy in the late stage, to solve the problems of slow convergence and easy getting stuck in local optimum in DRL training.

[0057] The designed exploration rate decay formula is as follows: ， Wherein, is the exploration rate at the t-th round, used to control the proportion of random actions; t is the training round.

[0058] To help the deep reinforcement learning model find a balance between complexity and generalization ability and avoid overfitting, the designed stage weight adjustment rule is as follows: ， Finally, to reduce the number of invalid training rounds, an early stopping mechanism is set. When the following formula is satisfied, the training stops and the generated topological structure is output: ， Wherein, is the reward value, is the reward at the -th epoch, is from the -th epoch to Average reward of the next epoch.

[0059] Example 2 In this example, a specific implementation example of generating the distribution transformer area topology by the method described in the above Example 1 is given. The topology generation result is as Figure 2 shown. Only three parameters, namely the number of users in the transformer area, the power supply radius, and the node spacing, need to be input. Through deep reinforcement learning optimization, an optimal topology that meets the relevant conditions can be generated.

[0060] The method for optimizing the distribution transformer area topology driven by deep reinforcement learning (DRL) proposed by the present invention generates a topology structure with high reliability, low loss, and balanced scalability as shown in the figure through the dynamic interaction between the agent and the environment. Compared with the traditional heuristic algorithm, this method has achieved a breakthrough improvement in core indicators such as structural rationality, economy, and scalability. The specific effects are shown in the following table.

[0061] Table 2 Comparison of the optimization results of the topology optimization method of the present invention and the traditional genetic algorithm

[0062] The preferred specific embodiments of the present invention have been described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and variations based on the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art in the technical field of the present invention through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art shall fall within the protection scope determined by the claims.

Claims

1. A method for generating a topology optimization of a distribution transformer area based on deep reinforcement learning, characterized in that the steps Including: Obtain the physical parameters of the distribution network, including the number of users in the substation area, the power supply radius, and the node spacing, transform the physical parameters of the power grid into the state space, and encode the topological structure and dynamic operation characteristics through a two-channel state matrix; Input the two-channel state matrix into the deep reinforcement learning network, decouple the action space of the distribution network branch decision into a two-dimensional decision combination of backbone node selection and branch extension length, dynamically update the power grid topology by the agent based on the action decision result, and adopt a multi-stage priority strategy to allocate user loads; Set a composite reward function in the deep reinforcement learning network to guide the agent to balance fast power supply and reduce line losses, adopt a progressive training strategy to train the deep reinforcement learning network until convergence, and output to generate the optimal topological structure.

2. The topology optimization generation method for a distribution transformer area based on deep reinforcement learning according to claim 1, wherein The dual-channel state matrix includes two channels representing the topological structure and dynamic operation characteristics respectively, and the state matrix is expressed as follows: , Wherein, is the adjacency matrix, is the user distribution feature, is the depth feature, is the total number of nodes in the substation area; Adjacency matrix Used to represent the topological structure of the substation area: , In the formula, , represents that there is a connection line between node and node ; represents that there is no connection line between node and node . Node User distribution characteristics are as follows: , Deep feature is the node hops to the nearest backbone node.

3. A topology optimization generation method for a distribution transformer area based on deep reinforcement learning according to claim 1, characterized in that, The described deep reinforcement learning network extracts topological space features through a dual-channel convolutional neural network and performs value prediction, and adopts gradient clipping and weight decay strategies to improve the generalization ability of the model: For the adjacency matrix and the feature matrix containing user distribution and node depth in the input two-channel state matrix, spatial features are extracted through two convolutional layers with different convolutional kernel sizes respectively, and the feature maps are uniformly output through adaptive average pooling; Flatten the feature maps and gradually reduce the dimension through three fully connected layers. A dropout layer added after the second fully connected layer is used to prevent overfitting, and the final output dimension is the size of the action space; The loss function is designed as follows: , Wherein: is the predicted value, i.e., the output of the deep reinforcement learning network; is the target value, i.e., the value calculated by the target network, and B is the batch size.

4. A topology optimization generation method for a distribution transformer area based on deep reinforcement learning according to claim 1, characterized in that, The reward function of the deep reinforcement learning network is as follows: , , , , , Wherein, is the stability reward, is the rated voltage value, is the node voltage of the th node in the distribution substation area; is the economic reward, is the equivalent resistance of the th line, is the current value of the th line; is the risk penalty, is the load rate of node is the current value of line and is the rated current-carrying capacity of line .​ 5. A topology optimization generation method for a distribution substation area based on deep reinforcement learning according to claim 4, characterized in that The deep reinforcement learning network adopts a three-stage training mechanism to dynamically adjust the exploration rate and the reward weight ; The exploration rate decay formula is as follows: , In the formula, is the exploration rate of the th round, which is used to control the proportion of random actions; is the number of training rounds; Based on the increase in the number of training rounds, sequentially lower the reward weights for training in each stage : Set an early stopping mechanism. When the sum of the absolute values of the differences between the reward values of each round and the average reward value is less than the set reward value difference threshold within the set number of rounds before the previous round, the training stops and outputs to generate the topological structure.

6. A topology optimization generation method for a distribution substation area based on deep reinforcement learning according to claim 1, characterized in that The two-dimensional decision combination of the backbone node selection and the branch extension length is expressed as follows: , In the formula, represents the floor function; is the original action number, , representing the high-dimensional discrete action space; is the maximum number of branches of a single backbone, , is the number of nodes in the backbone line.

7. A topology optimization generation method for a distribution transformer area based on deep reinforcement learning according to claim 6, characterized in that The action space embeds the physical constraint rules of the power grid to perform validity constraint verification, and invalid actions are excluded through rule screening, as follows: , Wherein, is the current number of nodes, i.e., the number of nodes that have been created, and l is the branch extension length, i.e., the number of nodes newly added in a single extension.

8. A topology optimization generation method for a distribution transformer area based on deep reinforcement learning according to claim 1, characterized in that, The dynamic update of the power grid topology based on the action decision result includes: creating new nodes according to the action decision, and updating the adjacency matrix and node depth, as follows: , In the formula, is the main trunk node number, is the new node, is the current node, is the branch extension length, is the node depth.

9. A topology optimization generation method for a distribution transformer area based on deep reinforcement learning according to claim 1, characterized in that The multi-stage priority strategy allocates user loads, based on real-time load prediction and dynamic priority adjustment, to achieve intelligent allocation of user loads to ensure the balanced operation of the power grid, specifically as follows: Calculation node load rate : , In the formula, is the node current load power, is the node rated capacity; Conduct node priority evaluation: , Wherein, is the node depth, with the main node being 0 and the branch node being the hop count; is the voltage deviation; ， ， is the weight coefficient, which is dynamically adjusted through reinforcement learning; The user access principle is: new users access the node with the highest priority. If the priorities of multiple nodes are the same, select the node with a smaller depth; When the node load rate is greater than the set node load rate threshold, new user access is rejected and branch expansion is triggered. The branch expansion mechanism is as follows: when the access of a continuously set number of users is rejected due to node overload, the backbone node with the largest available capacity is selected to expand the branch; If the node voltage deviation is greater than the set node voltage deviation threshold, remove the node from the candidate list.

10. A method for generating a topology optimization of a distribution transformer area based on deep reinforcement learning according to claim 9, characterized in that, Dynamically adjust the weight coefficient in the node priority evaluation, specifically as: In a high-load scenario, that is, when the average node load rate is greater than the set average load rate threshold, increase the weight coefficient α , and preferentially select low-load nodes; In the low-voltage scenario, that is, when the average voltage deviation of the nodes is greater than the set average voltage deviation threshold, increase the weight coefficient γ , and preferentially select the nodes with stable voltage.

Images