Power distribution network topology reconstruction method based on improved DQN

Abstract

The invention provides a power distribution network topology reconstruction method based on an improved DQN, and the method comprises the steps: building a power distribution network dynamic reconstruction optimization model, constructing a state space based on a historical load in the power distribution network dynamic reconstruction optimization model according to the autocorrelation of a load time sequence, and coding an actual real-time load into reward function setting; based on a radial topological optimization and Q value decoupling method of a spanning tree, an action object decided by the dynamic reconstruction optimization model of the power distribution network is changed from a topological structure to an action switch; an existing DQN experience playback mechanism is improved by using a multi-experience optimization action selection strategy, and a power distribution network dynamic reconstruction optimization model based on an improved DQN algorithm is obtained; and solving the power distribution network dynamic reconstruction optimization model based on the improved DQN algorithm to obtain an optimal power distribution network topology reconstruction scheme. According to the method, the decision-making capability of the reinforcement learning model is fully exerted, and the power distribution network topology reconstruction is realized based on the improved DQN, so that the reliability and economy of the power distribution system are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of distribution network reconstruction, and in particular to a distribution network topology reconstruction method based on improved DQN. Background Art

[0002] The distribution network adopts a radial structure and has a unidirectional power flow characteristic. Topology reconstruction will change the topology structure and power flow distribution of the distribution network by opening the normally closed section switches and closing the normally open parallel switches, thereby improving the performance of the system. The main goals of network reconstruction are to improve system safety and reliability, improve power quality, reduce energy losses, and improve voltage curves, see references [1-3]. In traditional distribution network topology reconstruction, it is hoped that losses can be minimized by transitioning from one topology to another. However, with the increase in the use of highly intermittent and volatile renewable energy and demand response programs, the volatility and uncertainty of the load will also increase accordingly, which leads to greater changes in load distribution and poses major challenges to the distribution system, see references [4-6].

[0003] In existing studies, a population-based intelligent search method has been proposed to solve the topology reconstruction problem. Reference [7] proposed a distribution network topology reconstruction scheme based on genetic algorithm to improve power quality and reliability. Reference [8] proposed to improve PSO by combining the taboo search algorithm, and improved the global search ability of the algorithm by setting a taboo table, thus realizing the dynamic reconstruction of the distribution network based on the improved binary PSO. In reference [9], mixed integer quadratic programming has been used to minimize power loss under reliability constraints. The scheme of using population-based intelligent search methods to reconstruct the distribution network topology to improve reliability has the following shortcomings: the accuracy and effectiveness of this method depend on the accuracy of the model used, and an accurate model will bring other modeling challenges; in addition, the mathematical model is usually derived based on approximate values, which requires complete power system information; at the same time, due to the huge search space, the intelligent search method is computationally inefficient, especially when the system size increases.

[0004] In addition to population-based intelligent search methods, learning-driven methods are also gaining increasing attention in the field of distribution network topology reconstruction. Reference

[10] uses batch constrained reinforcement learning to perform dynamic topology reconstruction to minimize grid operation costs. Reference

[11] uses deep learning to minimize network power loss and the number of switch actions in the distribution network. In reference

[12] , reinforcement learning is used to simultaneously reduce network power loss and improve voltage quality, and the Noisy Net deep Q network is combined with cyclic coding to improve training efficiency and computational efficiency. Reference

[13] uses deep Q learning to minimize the number of switch switching actions when performing topology reconstruction, and tests the computational cost and scalability of the brute force search algorithm and genetic algorithm. It is found that although there are some similarities between distribution network topology reconstructions with different objectives, the distribution network topology reconstruction scheme for improving reliability is still a challenging task because it requires estimating each possible topology to find the optimal reconstruction scheme. Therefore, it is crucial to develop intelligent learning-based methods for distribution network topology reconstruction to improve reliability.

[0005] References are as follows:

[0006] [1]D.Das, "Reconfiguration of distribution system using fuzzy multi-objective approach", International Journal of Electrical Power&Energy Systems, vol.28,

[0007] pp.331-338,Jun 2006.

[0008] [2]F.Ding and K.A.Loparo,"Hierarchical decentralized networkreconfiguration for smart distribution systems-Part I:Problem formulation andalgorithm development",IEEE Trans.Power Syst.,vol.30,no.2,pp.734-743,March2015.[3]F.Ding and K.A.Loparo,"Feeder reconfiguration for unbalanceddistribution systems with distributed generation:A hierarchical decentralizedapproach",

[0009] IEEE Trans.Power Systems,vol.31,no.2,pp.1,633-1,642,March 2016.

[0010] [4]H.Jiang,Y.Zhang,J.J.Zhang,D.W.Gao and E.Muljadi,

[0011] "Synchrophasor-based auxiliary controller to enhance the voltagestability of adistribution system with high renewable energy penetration",IEEE Trans.Smart Grid,vol.6,no.4,pp.2,107-2,115,2015.

[0012] [5]H.Jiang,X.Dai,W.Gao,J.J.Zhang,Y.Zhang and E.Muljadi,

[0013] "Spatial-temporal synchrophasor data characterization and analytics in smart grid fault detection identification and impact causal analysis", IEEETrans.Smart Grid, vol.7, no.5, pp.2,525-2,536.

[0014] [6] Mao Deyong, Zhou Zhangbin, Cao Tao. Reliability of dynamic reconstruction of distribution network based on dynamic time warping[J].

[0015] Environmental Technology, 2024, 42(06): 36-42.

[0016] [7] Liu Dangwu, Zheng Gaofeng, Liu Pengxi, et al. Application research of an optimization genetic algorithm in distribution network reconstruction[J].

[0017] Information Technology, 2018, 42(07): 107-112. DOI: 10.13274 / j.cnki.hdzj.2018.07.025.

[0018] [8] Wu Xiaomeng, Li Chenchen, Dang Bo. Dynamic Reconfiguration of Distribution Network Based on Improved Binary PSO[J]. Xi'an Shiyou University

[0019] Journal of Natural Science, 2024, 39(04): 124-131.

[0020] [9] J.Jose and A.Kowli, "Reliability constrained distribution feederreconfiguration for power loss minimization", 2016National Power Systems Conference (NPSC),

[0021] pp.1-6,2016.

[0022]

[10] Y. Gao, W. Wang, J. Shi and N. Yu, "Batch-constrained reinforcement learning for dynamic distribution network reconfiguration", IEEE Trans-actionson Smart Grid, vol. 11, no. 6, pp. 5357-5369, 2020.

[0023]

[11] X.Ji,Z.Yin,Y.Zhang,B.Xu et al., "Real-time autonomous dynamic reconfiguration based on deep learning algorithm for distribution net-work",

[0024] Electric Power Systems Research,vol.195,pp.107132,2021.

[0025]

[12] B.Wang, H.Zhu, H.Xu, Y.Bao and H.Di, "Distribution network reconfiguration based on noisynet deep q-learning network", IEEE Access, vol.9,

[0026] pp.90358-90365,2021.

[0027]

[13] SHOh, YTYoon and SWKim, "Online reconfiguration scheme of self-sufficient distribution network based on a reinforcement learning approach", Applied Energy, vol. 280, pp. 115900, 2020. Summary of the invention

[0028] The present invention provides a distribution network topology reconstruction method based on improved DQN, which gives full play to the decision-making ability of the reinforcement learning model and realizes the distribution network topology reconstruction based on the improved DQN, thereby improving the reliability and economy of the distribution system.

[0029] A distribution network topology reconstruction method based on improved DQN, comprising:

[0030] Step 1: Establish a distribution network dynamic reconstruction optimization model. In the distribution network dynamic reconstruction optimization model, a state space based on historical loads is constructed according to the autocorrelation of load time series, and the actual real-time load is encoded into the reward function setting;

[0031] Step 2: Based on the radial topology optimization and Q-value decoupling method of the spanning tree, the action object of the distribution network dynamic reconstruction optimization model decision is changed from the topological structure to the action switch;

[0032] Step 3: Use multiple experience optimization action selection strategies to improve the existing DQN experience replay mechanism and obtain a distribution network dynamic reconstruction optimization model based on the improved DQN algorithm;

[0033] Step 4: Solve the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm to obtain the optimal distribution network topology reconstruction solution.

[0034] Furthermore, the step 1 specifically includes:

[0035] Step 1.1, establish a dynamic reconstruction optimization model for the distribution network: comprehensively consider the voltage fluctuations and system losses brought to the distribution network by the access of new energy and the increase in the use of demand response plans, take maintaining voltage stability and reducing network losses as the optimization goals, and after satisfying the constraints including voltage constraints, current constraints, and node power balance constraints, design the following objective function:

[0036]

[0037] Where: F is the optimization target; P loss is the distribution network loss; N is the number of nodes; N T N is the number of time periods that a single day is divided into in hours. T =24; Δu i N T The sum of the differences between the voltage per unit value of node i and the reference value in a period; u i,t represents the per unit value of the actual voltage amplitude of node i in time period t; u i,m Represents the per-unit value of the reference voltage at node i, i.e. u i,m =1;

[0038] Step 1.2, analyze the autocorrelation of the load time series and calculate the autocorrelation coefficients at different time lags;

[0039] Step 1.3: Construct the state space based on the autocorrelation of the load time series analyzed in step 1.2: According to the autocorrelation coefficients of different time lags calculated in step 1.2, identify the key historical moments that affect the current load state, and determine how long the historical load should be included in the state space. Based on this, define the state of the previous moment as the load value of several moments in the past: S t =[p t_ ,q t_ ], p t_ Indicates the historical active load; q t_ Indicates historical reactive load;

[0040] Step 1.4: Set a reward function based on real-time load data encoding and encode the real-time load data into the reward function.

[0041] Furthermore, the autocorrelation function R(τ) used to calculate the autocorrelation coefficients at different time lags in step 1.2 is as follows:

[0042]

[0043] Where: p t is the load at time t; p t+τ is the load with a time lag of τ relative to time t; p is the average value of the load time series; σ is the standard deviation of the load time series; n is the number of time points in the load time series.

[0044] Furthermore, the reward function based on real-time load data encoding set in step 1.4 is as follows:

[0045] R t,1 = r1(P loss0,t -P loss,t ) (5)

[0046]

[0047] r t =R t,1 +R t,2 (7)

[0048] Where: r1 represents the reward coefficient for network loss optimization, r1>0; r2 represents the reward coefficient for voltage optimization, r2>0; P loss0,t is the distribution network loss in the original topology state; P loss,t Δu is the distribution network loss after adopting the topology reconstruction solution provided by the intelligent agent; i0,t Indicates the difference between the voltage per unit value of node i in the original topology state and the reference value; Δu i,t It represents the difference between the voltage per unit value and the reference value of node i after adopting the topology reconstruction solution provided by the intelligent agent.

[0049] Furthermore, the step 2 specifically includes:

[0050] Step 2.1, radial topology optimization based on spanning tree: According to the current distribution network structure, an undirected graph can be constructed with each node as the vertex and the line as the edge; the topology set that satisfies the radial topology is obtained by traversing the undirected graph using the spanning tree algorithm; the distribution network topology structure set A that satisfies the radial topology constraint condition is used to determine the action decision A provided by the intelligent agent. t =[α1,α2,K,α n ] Whether the radial topology conditions of the distribution network are met, so that the environment can judge whether to give the agent a high penalty value in the reward link;

[0051] Step 2.2, Q-value decoupling and action space simplification: When making decisions, the agent selects the switch closures with the highest scores as states S t Next action A t The sum of the scores q of the switches with the highest scores is Q(S t ,A t ).

[0052] Furthermore, the step three specifically includes:

[0053] Step 3.1, construct a multiple experience pool, and store a number of historical experiences in the multiple experience pool;

[0054] Step 3.2, action selection strategy optimization: adopt the improved ε-greedy strategy to take action. When selecting actions, the agent considers the current Q value, and also refers to the historical experience stored in the multiple experience pools, and dynamically adjusts the ε value according to the progress of training. In the early stage of training, the ε value is large to increase the diversity of exploration. In the later stage of training, the ε value gradually decreases to make more use of the excellent strategies that have been learned;

[0055] Step 3.3: Train the distribution network dynamic reconstruction optimization model based on the improved ε-greedy strategy to obtain a distribution network dynamic reconstruction optimization model based on the improved DQN algorithm.

[0056] Furthermore, the steps of constructing multiple experience pools in step 3.1 include:

[0057] 4) Experience classification: All the interaction experiences between the agent and the environment are classified according to their performance, and the experiences with excellent performance are stored in the advanced experience pool, while the experiences with average performance are stored in the ordinary experience pool;

[0058] 5) Priority allocation: Different priorities are assigned to the experiences in the experience pool, where experiences with excellent performance are given higher priorities so that they can be sampled and used more often in subsequent training;

[0059] 6) Multi-layer sampling: During each training, random sampling is performed from each layer of the experience pool according to the priority ratio to ensure that different types of experience can be reasonably utilized.

[0060] 9. Further, step 3.3 specifically includes:

[0061] 1) Initialize the environment and parameters

[0062] Initialize the topology of the distribution network and set the initial state; initialize the neural network parameters of the DQN model, including weights and biases; build multiple experience pools and set the priority and capacity of the corresponding experience pools;

[0063] 2) Agent-environment interaction

[0064] Under the current distribution network state, the agent selects actions according to the existing ε-greedy strategy to obtain a new topology and corresponding reward feedback;

[0065] 3) Experience storage

[0066] The experience data including the current state, the selected action, the feedback reward and the new state are stored in multiple experience pools. The experience is classified and stored according to the size of the reward value, and different priorities are assigned;

[0067] 4) Experience replay and model training

[0068] Randomly sample from multiple experience pools according to priority ratio to form training batches; use the sampled experience data to update the Q value function of the DQN model through the back propagation algorithm; combine historical experience feedback to further optimize the action selection strategy and improve the decision-making accuracy of the model;

[0069] 5) Dynamically adjust strategies

[0070] As training progresses, the ε value is dynamically adjusted to achieve a better balance between exploration and exploitation; high-priority experience is used multiple times to ensure that valuable experience can be fully used for training;

[0071] 6) Iterative training

[0072] Repeat the process of interaction between the agent and the environment and experience playback, and continuously update the parameters of the DQN model until the number of iterations reaches the preset number or the reward value reaches the preset value;

[0073] After completing the training according to the above process, the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm can be obtained.

[0074] The present invention proposes a distribution network topology reconstruction scheme based on improved DQN, aiming to meet the operation constraints while minimizing the active power loss of the line and maintaining voltage stability; in addition, in order to solve the problem of dimensionality curse of action space caused by topology reconstruction, a radial topology optimization and Q-value decoupling method based on spanning tree is proposed; at the same time, the existing DQN experience replay mechanism is improved, and multiple experience pools are used to optimize the action selection strategy and improve the training performance, thereby improving the reliability and economy of the distribution system. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic diagram of the structure of the IEEE 16-node distribution network according to an embodiment of the present invention;

[0076] Figure 2 It is a schematic diagram of the structure of an undirected graph and a spanning tree according to an embodiment of the present invention;

[0077] Figure 3 It is a flow chart of a distribution network topology reconstruction method based on improved DQN according to an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] See also Figure 3 The present invention provides a method for reconfiguring the topology of a distribution network based on an improved DQN, comprising the following steps:

[0080] Step 1: Establish a distribution network dynamic reconstruction optimization model. In the distribution network dynamic reconstruction optimization model, a state space based on historical loads is constructed according to the autocorrelation of load time series, and the actual real-time load is encoded into the reward function setting;

[0081] Step 2: Based on the radial topology optimization and Q-value decoupling method of the spanning tree, the action object of the distribution network dynamic reconstruction optimization model decision is changed from the topological structure to the action switch;

[0082] Step 3: Use multiple experience optimization action selection strategies to improve the existing DQN experience replay mechanism and obtain a distribution network dynamic reconstruction optimization model based on the improved DQN algorithm;

[0083] Step 4: Solve the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm to obtain the optimal distribution network topology reconstruction solution.

[0084] The step 1 specifically includes:

[0085] Step 1.1, establish a dynamic reconstruction optimization model for the distribution network: comprehensively consider the voltage fluctuations and system losses brought to the distribution network by the access of new energy and the increase in the use of demand response plans, and take maintaining voltage stability and reducing network losses as the optimization goals. After satisfying the constraints including voltage constraints, current constraints, and node power balance constraints, design the objective function as shown below:

[0086]

[0087] Where: F is the optimization target; P loss is the distribution network loss; N is the number of nodes; N T N is the number of time periods that a single day is divided into in hours. T =24; Δu i N T The sum of the differences between the voltage per unit value of node i and the reference value in a period; u i,t represents the per unit value of the actual voltage amplitude of node i in time period t; u i,m Represents the per-unit value of the reference voltage at node i, i.e. u i,m =1.

[0088] Step 1.2: Analyze the autocorrelation of the load time series and calculate the autocorrelation coefficients at different time lags.

[0089] Autocorrelation, also known as serial correlation, is used in statistics to describe the similarity between a random process or time series and itself at different time points. It measures the degree of correlation of a time series under time lag, that is, the linear correlation between the value at a certain moment and the values ​​at several previous moments.

[0090] In the distribution network load time series, autocorrelation reflects the degree of influence of past load change patterns on future loads. High autocorrelation means that load changes have strong continuity and predictability. This characteristic can be used to construct a state space based on historical data, thereby realizing a direct mapping relationship from historical load data to topology reconstruction decisions, avoiding complex load forecasting models, and reducing decision deviations caused by forecast errors.

[0091] In traditional distribution network topology reconstruction, prediction models are usually used to predict future load conditions to guide the decision-making process. However, the load uncertainty and complexity brought about by the access of new energy sources make it difficult for the prediction model to accurately predict future loads, which can easily lead to decision-making errors in topology reconstruction. Therefore, the present invention avoids predictive modeling of uncertain loads by directly utilizing the autocorrelation of the load time series. The autocorrelation function R(τ) applied to the load time series is as follows:

[0092]

[0093] Where: p t is the load at time t; p t+τ is the load with a time lag of τ relative to time t; is the average value of the load time series; σ is the standard deviation of the load time series; n is the number of time points in the load time series.

[0094] Step 1.3: Construct the state space based on the autocorrelation of the load time series analyzed in step 1.2.

[0095] By analyzing the time series of historical load data, we determine its autocorrelation and use it as a basis to construct the state space. In the load time series, the current load state is related to the load state at several moments in the past. Based on this relationship, we can use the historical load data as the input of the current state. In this way, the intelligent agent can predict the current load state by observing the historical load data, and then make an optimization decision based on the historical data. This method not only reduces the dependence on future load forecasts, but also can utilize the implicit laws in the load time series to improve the accuracy of decision-making.

[0096] The key to the DQN method is to learn the Q value by establishing and training a neural network, whose input is the state and the output is the Q value corresponding to different actions selected under the state. The state is part or all of the characteristics of the current environment observed by the agent. The present invention aims to use historical load data to determine the topological reconstruction scheme of the distribution network, so the state space can be constructed based on the autocorrelation of the load time series. Step 1.3 specifically includes:

[0097] According to the autocorrelation coefficients of different time lags calculated in step 1.2, the key historical moments that affect the current load state are identified, and the length of historical load that should be included in the state space is determined. Based on this, the state of the previous moment is defined as the load value of several moments in the past: S t =[p t_ ,q t_ ], p t_ Indicates the historical active load; q t_ Indicates historical reactive load.

[0098] Step 1.4: Set a reward function based on real-time load data encoding and encode the real-time load data into the reward function.

[0099] After the state space is constructed, the real-time load data needs to be encoded into the reward function to guide the optimization process of the intelligent agent. The real-time load data can be used to update the state in real time and affect the calculation of the reward function, thereby achieving a dynamic response of load optimization.

[0100] DQN training sets the reward function based on the objective function. The action selection strategy of the agent is continuously iterated and optimized according to the guidance of the reward function, and finally the optimal action selection strategy is found. Therefore, the design of the reward function should fully consider the actual operation goals of the distribution network, in order to minimize the voltage deviation and network loss. Therefore, the reward function is set as follows:

[0101] R t,1 = r1(P loss0,t -P loss,t ) (5)

[0102]

[0103] r t =R t,1 +R t,2 (7)

[0104] Where: r1 represents the reward coefficient for network loss optimization, r1>0; r2 represents the reward coefficient for voltage optimization, r2>0; P loss0,t is the distribution network loss in the original topology state; P loss,t Δu is the distribution network loss after adopting the topology reconstruction solution provided by the intelligent agent; i0,t Indicates the difference between the voltage per unit value of node i in the original topology state and the reference value; Δu i,t It represents the difference between the voltage per unit value and the reference value of node i after adopting the topology reconstruction solution provided by the intelligent agent.

[0105] Since it is inevitable that the topological structure of the selected action combination does not satisfy the radial topological constraint during the DQN training process, when the decision provided by the agent does not satisfy the topological radial constraint, the reward function should be set to a high penalty value. In addition, if the voltage of each node is not within the allowable range of ±5% deviation, the reward function should also be set to a high penalty value and the training of this round should be ended; otherwise, no penalty is given and the training of this round continues.

[0106] As mentioned above, the construction of the state space avoids the predictive modeling of uncertain loads. When combined with the real-time load of the distribution network in the reward function setting, the intelligent agent can directly make topological reconstruction plans for the distribution network based on historical loads, thus avoiding incorrect decisions caused by predictive modeling errors.

[0107] One of the difficulties in reconfiguring the topology of distribution networks is that every possible distribution network structure needs to be tested. The difficulty of using intelligent search algorithms is that they need to be recalculated each time they are used, and the online application effect is poor. However, if data-driven algorithms, such as deep reinforcement learning, are used to deal with this problem, it is inevitable to face the curse of dimensionality. Because the change in topology structure originates from the action of switches, as the number of switches increases, the types of topology structures will grow geometrically, which not only increases the computational complexity, but also may make it difficult for the intelligent agent to find the optimal topology reconstruction solution within a reasonable time. In order to solve this problem, the radial topology optimization and Q-value decoupling method based on the spanning tree described in step 2 is proposed.

[0108] The step 2 specifically includes:

[0109] Step 2.1, radial topology optimization based on spanning tree: First, the distribution network structure is converted into an undirected graph. Taking the IEEE 16-node distribution network as an example, according to the current distribution network structure, a graph can be constructed with each node as a vertex and the line as an edge. Figure 1 The undirected graph shown.

[0110] Since nodes 1 to 3 of the IEEE 16-node distribution network are feeders, they can be merged into one node, thus obtaining an undirected graph with 14 nodes, as shown in Figure 2 shown.

[0111] Because the distribution network is generally a radial network with unidirectional power flow, two constraints must be met during the distribution network topology reconstruction process: radial constraints and full-node traversal constraints. Based on this, the spanning tree algorithm can be used to traverse the undirected graph to obtain a topology set that satisfies the radial topology. For the same distribution network without adding other nodes, the radial topology set can be used for DQN training and decision-making without updating, so unlike the intelligent search algorithm, this scheme uses the spanning tree to traverse the set only once. The distribution network topology structure set A that satisfies the radial topology constraints is used to determine the action decision A provided by the intelligent agent. t =[α1,α2,K,α n ] Whether the radial topology conditions of the distribution network are met, so that the environment can judge whether to give the intelligent agent a high penalty value in the reward link.

[0112] After radial topology optimization based on spanning tree, the number of action spaces is reduced from 2 to n The number of topologies in set A is reduced. Although the action airborne has been optimized to a certain extent, if it is directly used for DQN training, there will still be the problem of low training efficiency. Therefore, it is necessary to further reduce the dimension of the action space through the Q value decoupling method.

[0113] Step 2.2, Q-value decoupling and action space simplification: Traditional distribution network topology reconstruction methods usually take the entire topology structure as the decision object, resulting in a huge decision space and making it difficult for the agent to effectively learn the optimal strategy during training. In order to simplify the decision process, a Q-value decoupling method is proposed to convert the agent's decision object from a complex topology structure to a specific operation switch.

[0114] The core of the Q-value decoupling method is that the action decision object in the action space is no longer a topological structure, that is, the action space is no longer a set A, but a topological switch set: {α1, α2, K, α n}. DQN will give action A t =[α1,α2,K,α n ] to score each switch and get [q1,q2,K,q n ], in order to map these scores to each distribution network topology that meets the radial topology conditions, a new mapping strategy is designed: Figure 1 Taking the IEEE 16-node distribution network shown in the figure as an example, the agent only selects the 13 switch closures with the highest scores as state S when making decisions. t Next action A t The sum of the scores q of these 13 switches is Q(S t ,A t ). This achieves Q-value decoupling, which greatly reduces the dimension of the action space and enables it to be directly applied to deep reinforcement learning models.

[0115] The step three specifically includes:

[0116] Step 3.1, build multiple experience pools, and store several historical experiences in multiple experience pools: The core of this strategy is to classify the experience collected during the training process. The traditional experience playback mechanism uses a single experience pool, while the improved mechanism adopts a multiple experience pool structure to store specific types of experience according to different criteria. The construction of multiple experience pools includes: 1) Experience classification: All the interaction experiences of the agent and the environment are classified according to performance, and the experience with excellent performance is stored in the advanced experience pool, while the general experience is stored in the ordinary experience pool. 2) Priority allocation: Different priorities are assigned to the experience in the experience pool, among which the experience with excellent performance is given a higher priority so that it can be sampled and used more times in subsequent training. 3) Multi-layer sampling: During each training, random sampling is performed from each layer of the experience pool according to the priority ratio to ensure that different types of experience can be reasonably used. In this way, the convergence process can be effectively accelerated while preventing the model from falling into a local optimal solution.

[0117] Step 3.2, action selection strategy optimization: In traditional DQN, the action selection strategy is usually based on the ε-greedy strategy, which balances between exploring new action strategies and using known optimal action strategies. However, in the distribution network topology reconstruction problem, due to the large and complex action space, directly applying the ε-greedy strategy may lead to low exploration efficiency. Therefore, the improved DQN introduces an action selection strategy based on multiple experience optimization to further improve the decision-making performance of the intelligent agent.

[0118] When selecting actions, the agent not only considers the current Q value, but also refers to the historical experience stored in the multiple experience pools. In this way, the agent can learn more effective strategies from past experience, thereby improving the accuracy of current decisions. In addition, the improved ε-greedy strategy is no longer static, but dynamically adjusts the ε value according to the progress of training. In the early stage of training, the ε value is large to increase the diversity of exploration; in the later stage of training, the ε value gradually decreases to make more use of the excellent strategies that have been learned. This dynamic adjustment mechanism helps to accelerate model convergence and improve DQN performance.

[0119] Step 3.3: Train the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm: Combine the above DQN framework and the improved learning technology to train the optimization model in the following process:

[0120] 1) Initialize the environment and parameters

[0121] Initialize the topology of the distribution network and set the initial state; initialize the neural network parameters of the DQN model, including weights and biases; build multiple experience pools and set the priority and capacity of the corresponding experience pools.

[0122] 2) Agent-environment interaction

[0123] Under the current state of the distribution network, the agent selects actions according to the existing ε-greedy strategy to obtain a new topological structure and corresponding reward feedback.

[0124] 3) Experience storage

[0125] The experience data including the current state, the selected action, the feedback reward and the new state are stored in multiple experience pools. The experience is classified and stored according to the size of the reward value, and different priorities are given.

[0126] 4) Experience replay and model training

[0127] Random sampling is performed from multiple experience pools according to priority ratio to form training batches. The Q-value function of the DQN model is updated through the back-propagation algorithm using the sampled experience data. Combined with historical experience feedback, the action selection strategy is further optimized to improve the decision-making accuracy of the model.

[0128] 5) Dynamically adjust strategies

[0129] As training progresses, the ε value is dynamically adjusted to enable the agent to achieve a better balance between exploration and utilization; high-priority experience is used multiple times to ensure that valuable experience can be fully used for training.

[0130] 6) Iterative training

[0131] Repeat the process of interaction between the agent and the environment and experience replay, and continuously update the parameters of the DQN model until the number of iterations reaches the preset number or the reward value reaches the preset value.

[0132] After completing the training according to the above process, the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm can be obtained.

[0133] After training DQN, the deep neural network in DQN will save relevant parameters, and then the obtained intelligent agent can directly make decisions to obtain the optimal distribution network topology reconstruction plan, that is, solve the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm to obtain the optimal distribution network topology reconstruction plan.

[0134] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A distribution network topology reconstruction method based on improved DQN, characterized in that: The steps include: Step 1: Establish a distribution network dynamic reconstruction optimization model. In the distribution network dynamic reconstruction optimization model, a state space based on historical loads is constructed according to the autocorrelation of load time series, and the actual real-time load is encoded into the reward function setting; Step 2: Based on the radial topology optimization and Q-value decoupling method of the spanning tree, the action object of the distribution network dynamic reconstruction optimization model decision is changed from the topological structure to the action switch; Step 3: Use multiple experience optimization action selection strategies to improve the existing DQN experience replay mechanism and obtain a distribution network dynamic reconstruction optimization model based on the improved DQN algorithm; Step 4: Solve the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm to obtain the optimal distribution network topology reconstruction solution.

2. The distribution network topology reconstruction method based on improved DQN according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1, establish a dynamic reconstruction optimization model for the distribution network: comprehensively consider the voltage fluctuations and system losses brought to the distribution network by the access of new energy and the increase in the use of demand response plans, take maintaining voltage stability and reducing network losses as the optimization goals, and after satisfying the constraints including voltage constraints, current constraints, and node power balance constraints, design the following objective function: Where: F is the optimization target; P loss is the distribution network loss; N is the number of nodes; N T N is the number of time periods that a single day is divided into in hours. T =24; Δu i N T The sum of the differences between the voltage per unit value of node i and the reference value in a period; u i,t represents the per unit value of the actual voltage amplitude of node i in time period t; u i,m Represents the per-unit value of the reference voltage at node i, i.e. u i,m =1; Step 1.2, analyze the autocorrelation of the load time series and calculate the autocorrelation coefficients at different time lags; Step 1.3: Construct the state space based on the autocorrelation of the load time series analyzed in step 1.2: According to the autocorrelation coefficients of different time lags calculated in step 1.2, identify the key historical moments that affect the current load state, and determine how long the historical load should be included in the state space. Based on this, define the state of the previous moment as the load value of several moments in the past: S t =[p t_ ,q t_ ], p t_ Indicates the historical active load; q t_ Indicates historical reactive load; Step 1.4: Set a reward function based on real-time load data encoding and encode the real-time load data into the reward function.

3. The distribution network topology reconstruction method based on improved DQN according to claim 2, characterized in that: The autocorrelation function R(τ) used to calculate the autocorrelation coefficients at different time lags in step 1.2 is as follows: Where: p t is the load at time t; p t+τ is the load with a time lag of τ relative to time t; is the average value of the load time series; σ is the standard deviation of the load time series; n is the number of time points in the load time series.

4. The distribution network topology reconstruction method based on improved DQN according to claim 2, characterized in that: The reward function based on real-time load data encoding set in step 1.4 is as follows: R t,1 =r1(P loss0,t -P loss,t ) (5) r t =R t,1 +R t,2 (7) Where: r1 represents the reward coefficient for network loss optimization, r1>0; r2 represents the reward coefficient for voltage optimization, r2>0; P loss0,t is the distribution network loss in the original topology state; P loss,t Δu is the distribution network loss after adopting the topology reconstruction solution provided by the intelligent agent; i0,t Indicates the difference between the voltage per unit value of node i in the original topology state and the reference value; Δu i,t It represents the difference between the voltage per unit value and the reference value of node i after adopting the topology reconstruction solution provided by the intelligent agent.

5. The distribution network topology reconstruction method based on improved DQN according to claim 1, characterized in that: The step 2 specifically includes: Step 2.1, radial topology optimization based on spanning tree: According to the current distribution network structure, an undirected graph can be constructed with each node as the vertex and the line as the edge; the topology set that satisfies the radial topology is obtained by traversing the undirected graph using the spanning tree algorithm; the distribution network topology structure set A that satisfies the radial topology constraint condition is used to determine the action decision A provided by the intelligent agent. t =[α1,α2,K,α n ] Whether the radial topology conditions of the distribution network are met, so that the environment can judge whether to give the agent a high penalty value in the reward link; Step 2.2, Q-value decoupling and action space simplification: When making decisions, the agent selects the switch closures with the highest scores as states S t Next action A t The sum of the scores q of the switches with the highest scores is Q(S t ,A t ).

6. The distribution network topology reconstruction method based on improved DQN according to claim 1, characterized in that: The step three specifically includes: Step 3.1, construct a multiple experience pool, and store a number of historical experiences in the multiple experience pool; Step 3.2, action selection strategy optimization: adopt the improved ε-greedy strategy to take action. When selecting actions, the agent considers the current Q value, and also refers to the historical experience stored in the multiple experience pools, and dynamically adjusts the ε value according to the progress of training. In the early stage of training, the ε value is large to increase the diversity of exploration. In the later stage of training, the ε value gradually decreases to make more use of the excellent strategies that have been learned; Step 3.3: Train the distribution network dynamic reconstruction optimization model based on the improved ε-greedy strategy to obtain a distribution network dynamic reconstruction optimization model based on the improved DQN algorithm.

7. The distribution network topology reconstruction method based on improved DQN according to claim 6, characterized in that: The steps to construct multiple experience pools in step 3.1 include: 1) Experience classification: All the interaction experiences between the agent and the environment are classified according to their performance, and the experiences with excellent performance are stored in the advanced experience pool, while the experiences with average performance are stored in the ordinary experience pool; 2) Priority allocation: Different priorities are assigned to the experiences in the experience pool, where experiences with excellent performance are given higher priorities so that they can be sampled and used more often in subsequent training; 3) Multi-layer sampling: During each training, random sampling is performed from each layer of the experience pool according to the priority ratio to ensure that different types of experience can be reasonably utilized.

8. The method for reconfiguring the distribution network topology based on the improved DQN according to claim 6, characterized in that: Step 3.3 specifically includes: 1) Initialize the environment and parameters Initialize the topology of the distribution network and set the initial state; initialize the neural network parameters of the DQN model, including weights and biases; build multiple experience pools and set the priority and capacity of the corresponding experience pools; 2) Agent-environment interaction Under the current distribution network state, the agent selects actions according to the existing ε-greedy strategy to obtain a new topology and corresponding reward feedback; 3) Experience storage The experience data including the current state, the selected action, the feedback reward and the new state are stored in multiple experience pools. The experience is classified and stored according to the reward value and given different priorities. 4) Experience replay and model training Randomly sample from multiple experience pools according to priority ratio to form training batches; use the sampled experience data to update the Q value function of the DQN model through the back propagation algorithm; combine historical experience feedback to further optimize the action selection strategy and improve the decision-making accuracy of the model; 5) Dynamically adjust strategies As training progresses, the ε value is dynamically adjusted to achieve a better balance between exploration and exploitation; high-priority experience is used multiple times to ensure that valuable experience can be fully used for training; 6) Iterative training Repeat the process of interaction between the agent and the environment and experience playback, and continuously update the parameters of the DQN model until the number of iterations reaches the preset number or the reward value reaches the preset value; After completing the training according to the above process, the distribution network dynamic reconstruction optimization model based on the improved DQN algorithm can be obtained.

Images