Power distribution network reconstruction method and system based on topology security constraint and integrated reinforcement learning

Abstract

The invention discloses a power distribution network reconstruction method and system based on topology security constraint and integrated reinforcement learning. The method comprises the steps of power grid environment modeling including system state quantity design, reward function design, power grid constraint embedding and system power flow calculation; constructing a topology security layer, and correcting a topology reconstruction action by using a topology mask through a detection algorithm; pre-judging different actions by using an integrated action network, a parameter doping mechanism and a network deletion mechanism; building a reinforcement learning training framework to train the initial strategy network, and obtaining network parameters; and obtaining a scheduling strategy of power distribution network reconstruction according to the state data of the system by using the trained strategy network. According to the method, a topology security layer is superposed on the basis of reinforcement learning, and mechanisms such as action network integration are introduced, so that the problem of lack of security of reinforcement learning model output strategies is solved, the volatility of the parameter training process is relieved, and the reconstruction scheduling strategy of the power distribution network can be efficiently obtained in real time.

Description

Technical Field

[0001] The present invention relates to the technical field of power system optimization and dispatching, and in particular to a distribution network reconstruction method and system based on topological security constraints and integrated reinforcement learning. Background Art

[0002] As fossil energy is gradually depleted, the low-carbon transformation of energy structure has attracted widespread attention from countries around the world, prompting large-scale renewable distributed energy to be widely connected to the power grid to meet the growing local energy demand. However, the randomness and volatility of renewable energy generation frequently cause problems such as system voltage deviation and power flow changes, which seriously threaten the safe and stable operation of the distribution network system.

[0003] In order to effectively deal with the challenges brought by the uncertainty of new energy, the current dispatch optimization research focuses on integrating diversified dispatchable resources on the source and load side, such as energy storage devices, reactive compensation devices, and flexible loads, and combines the dynamic reconstruction strategy on the grid side to directly control the power flow distribution of the system to achieve the coordinated optimization of the system operation status. In the traditional research framework, the mathematical modeling optimization algorithm has a significantly reduced solution efficiency when faced with high-dimensional discrete topological variables mixed with uncertainty factors, and it is difficult to meet the stringent requirements of real-time dispatch of power grids. In this context, reinforcement learning theory has opened up a new path for the optimization dispatch problem of complex power grids. This method does not rely on precise system parameters and physical models. It can quickly adapt to environmental changes and directly train a general decision model for complex optimization problems instead of a single strategy by mining and characterizing the key features and knowledge models of the power system, with extremely high solution speed. However, when it comes to reconstruction optimization tasks, the huge discrete topological space and the high proportion of invalid actions also make it obvious that the neural network has difficulty in fitting during training. The full enumeration of the action space is not only computationally expensive, but also difficult to meet the application of actual engineering. Although continuous processing can be tried, it is easy to complicate the problem. Some studies have narrowed the action space by screening effective topology sets. Although this has alleviated the fitting pressure to a certain extent, it has also indirectly affected the optimization performance of the model. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a distribution network reconstruction method and system based on topological security constraints and integrated reinforcement learning. By improving the topological encoding method, using multi-dimensional discrete space for representation, and introducing a topological mask mechanism to ensure the security and computational efficiency of the model; further using cluster strategies to construct action network groups to achieve action prediction and screening, thereby enhancing the stability of the training process.

[0005] Technical solution: In order to achieve the above-mentioned invention object, the distribution network reconstruction method based on topological security constraints and integrated reinforcement learning proposed in the present invention includes the following steps:

[0006] Step 1: Construct a power grid environment, which is used to interact with the reinforcement learning model to obtain training data, including system state quantity design, reward function design, power grid constraint embedding and system steady-state quantity calculation. The purpose of the reinforcement learning model is to minimize network loss and the number of line switch actions by adjusting the system's grid structure and energy storage resources; the model's strategy network gives a reconstruction action based on the current state data of the power grid environment, calculates the reward for this action based on the reward function, and returns the reward and the next state data of the power grid environment; the state data includes the topological action of the network at the previous moment, the energy storage capacity state at the previous moment, and the node power of the previous period; the reconstruction action includes topological action and energy storage action, wherein the topological reconstruction action is designed to selectively disconnect a controllable branch in different independent loops in turn;

[0007] Step 2: Construct a topology security layer. The topology security layer uses an action mask through a topology detection algorithm to detect and correct the topology reconstruction action initially calculated by the model. When it is detected that the branch currently operated is a tree branch, a negative infinite mask element is superimposed on the branch so that it will not be selected.

[0008] Step 3: Combine multiple action networks into an integrated action group, and predict and screen different reconstruction actions of the integrated action group through parameter doping mechanism and network deletion mechanism. The parameter doping mechanism sets a cycle counter for each action network, counts the number of optimal actions of each action network in the first cycle, passes the parameters of the optimal action network to other action networks, and perturbs their parameters; the network deletion mechanism records the optimal action frequency of each action network in the second cycle and deletes it from the action network group;

[0009] Step 4: Use the dual-delay-deterministic policy gradient framework to train the reinforcement learning model to obtain network parameters for generating the reconstruction scheduling strategy;

[0010] Step 5: Use the trained reinforcement learning model to obtain the corresponding distribution network reconstruction scheduling strategy based on the current system status data.

[0011] The present invention also provides a distribution network reconstruction system based on topological security constraints and integrated reinforcement learning, comprising:

[0012] A power grid environment modeling module is used to construct a power grid environment, which is used to interact with a reinforcement learning model to obtain training data, including system state quantity design, reward function design, power grid constraint embedding and system steady-state quantity calculation. The purpose of the reinforcement learning model is to minimize network loss and the number of line switch actions by adjusting the system's grid structure and energy storage resources. The model's strategy network gives a reconstruction action based on the current state data of the power grid environment, calculates the reward for this action based on the reward function, and returns the reward and the next state data of the power grid environment. The state data includes the topological action of the network at the previous moment, the energy storage capacity state at the previous moment, and the node power of the previous period. The reconstruction action includes topological action and energy storage action, wherein the topological reconstruction action is designed to selectively disconnect a controllable branch in different independent loops in turn.

[0013] A topology security detection module is used to construct a topology security layer. The topology security layer uses an action mask through a topology detection algorithm to detect and correct the topology reconstruction action initially calculated by the model. When it is detected that the branch currently operated is a tree branch, a negative infinite mask element is superimposed on the branch so that it will not be selected.

[0014] The action network pre-screening module is used to combine multiple action networks into an integrated action group, and to prejudge and screen different reconstruction actions of the integrated action group through a parameter doping mechanism and a network deletion mechanism. The parameter doping mechanism sets a cycle counter for each action network, counts the number of optimal actions of each action network in the first cycle, and passes the parameters of the optimal action network to other action networks to perturb their parameters; the network deletion mechanism records the optimal action frequency of each action network in the second cycle and deletes it from the action network group;

[0015] A reinforcement learning model training module is used to train the reinforcement learning model using a dual-delay-deterministic policy gradient framework to obtain network parameters for generating a reconstruction scheduling strategy;

[0016] The distribution network reconstruction strategy determination module is used to use the trained reinforcement learning model to obtain the corresponding distribution network reconstruction scheduling strategy according to the current system status data.

[0017] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the distribution network reconstruction method based on topological safety constraints and integrated reinforcement learning as described above are implemented.

[0018] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the distribution network reconstruction method based on topological safety constraints and integrated reinforcement learning as described above are implemented.

[0019] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0020] (1) With the continuous grid connection of large-scale distributed photovoltaics, distribution networks face severe challenges such as intensified voltage fluctuations and increased network losses. Deep reinforcement learning has greatly improved the efficiency of solving optimization problems compared to traditional algorithms, but when it comes to collaborative optimization problems involving network reconstruction, the model is usually difficult to converge and the training stability is poor. To this end, the present invention proposes a distribution network reconstruction optimization method based on topological security constraints and integrated reinforcement learning, embeds the topological security layer into reinforcement learning, and alleviates the problem of stable optimization of large topological discrete spaces through coding design, achieving extremely strong security guarantees. At the same time, a special topological connectivity detection algorithm is introduced to meet the topological security requirements of the power system with extremely high computational efficiency;

[0021] (2) The present invention proposes an integrated reinforcement learning training technique that can ensure the rationality of the early action strategy through multi-network prediction, parameter soft update, and network deletion while occupying only a small amount of computing power, assist model training, and improve the optimization performance of the model, thereby ensuring the efficient and economical operation of the power system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the model structure of the distribution network reconstruction method of the present invention;

[0023] Figure 2 It is a schematic diagram of topological connectivity detection of the present invention;

[0024] Figure 3 The reward curve diagram of the method of the present invention and other comparison algorithms;

[0025] Figure 4 This is a comparison diagram of network loss before and after reconstruction using the method of the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0027] Reference Figure 1 The present invention proposes a distribution network reconstruction method based on topological security constraints and integrated reinforcement learning, comprising the following steps:

[0028] Step 1: Grid environment modeling, including system state quantity design, reward function design, grid constraint embedding and system steady-state solution;

[0029] Step 2: Build a topology security layer and use the topology mask to correct the topology reconstruction action through the detection algorithm;

[0030] Step 3: Introduce integrated action network, parameter doping mechanism and network deletion mechanism to predict different actions, while ensuring the calculation speed and improving the stability of model training;

[0031] Step 4: Establish a reinforcement learning training framework to train the initial policy network and obtain network parameters;

[0032] Step 5: Use the trained strategy network to obtain the dispatching strategy for distribution network reconstruction based on the system status data.

[0033] The following is a detailed description of the specific implementation process of the distribution network reconstruction optimization using the method of the present invention in conjunction with specific embodiments. The present invention uses energy storage and topology coordinated scheduling as the optimization scenario, and selects the IEEE 33-node system as the benchmark case to design a simulation experiment. In this system, 16 nodes are connected to distributed photovoltaics to simulate the uncertainty factors of new energy. At the same time, 3 energy storage devices and 12 branch switches are configured to ensure the stable operation of the system. The photovoltaic power generation data uses typical photovoltaic scene data in summer, and on this basis, proportional noise and bias noise are superimposed to meet the data scale requirements of the reinforcement learning training process. The distribution characteristics of these noises are fitted based on the actual photovoltaic data in a certain area of ​​Belgium to ensure that the simulation environment can more realistically reflect the actual operating conditions. Based on the constructed model data, the specific implementation steps of the method are as follows:

[0034] Step (1), construct a distribution network environment for interactive training.

[0035] The distribution network environment is a virtual simulation environment used to interact with the reinforcement learning model to obtain training data. Its input policy network gives a reconstruction action, calculates the reward obtained by this action through the reward function according to the current system state data, and returns the reward and the next state data of the power grid environment.

[0036] In general, the environment is a modeling of the task, and the reinforcement learning algorithm (the policy network of the intelligent agent) is a tool for solving the task. Different from traditional algorithms such as using second-order cones to model the optimization task of the power grid flow, and then using the random Robin algorithm to solve the scheduling optimization problem, the present invention first constructs the distribution network environment, and transforms it into a Markov decision environment according to the differences in solving the task when combining the actual distribution network scheduling optimization task. Similar reinforcement learning algorithms under excellent environmental modeling, such as a highly condensed state space and a clear reward function, will have better optimization performance after training.

[0037] The design of the environment includes the reward function (which directly determines the direction of the task), the state (the characteristic information of the task decision), and the action (the agent optimizes the action space). According to the embodiment of the present invention, the construction of the distribution network system environment can be divided into the following steps:

[0038] First, interactive variables such as reward functions, states, and actions are designed according to the different tasks and environments. The state data reflects the essential characteristics of the power system, and the reward function determines the optimization direction and tasks of the strategy model training. The distribution network reconstruction task set by this method minimizes the network loss and the number of line switch actions by adjusting the system's grid structure and load-side resources such as energy storage. The calculation method is:

[0039]

[0040] Among them, S is the system state, A topo,t-1 is the topological action at the previous moment, E t-1 is the energy storage capacity state at the last moment, P is the node power in the last period, R is the reward function, k price is the economic coefficient of electricity purchase, P ess,j,t is the charge and discharge power of the jth energy storage at the current moment, c pur is the current electricity price, c pur,ess is the current average electricity price of energy storage, k loss is the network loss coefficient, P loss is the system network loss after the action is executed. switch is the switching operation coefficient, N switch is the number of switch actions, and A is the strategy network action. topo , P ess are topology and energy storage actions, respectively. According to graph theory, the necessary but not sufficient condition for the conversion from a ring network to a radial topology is that at least one branch switch is turned on in each independent loop. Therefore, the topology reconstruction action of the model is designed to selectively disconnect a controllable branch in different independent loops in turn. L is the set of independent loop numbers, ε switch,l is the set of all controllable branch numbers under independent loop l, a l is the discrete action given for independent loop l, c l,i It is the corresponding action code, 0 means connected, 1 means disconnected.

[0041] Secondly, physical constraints are incorporated into the distribution network environment. The power grid environment includes topological constraints, power flow constraints, and energy storage capacity constraints. In addition to reasonably limiting the number of switch actions, the network structure adopted by the system must strictly satisfy the radial network topology. The system steady state is obtained by solving the power flow equation. The constraint calculation formula is:

[0042]

[0043] Among them, G is the total number of actions of all switches in the scheduling period, G t,m is the total number of actions of switch m within the scheduling period; H m,t is the state of switch m at time t, 0 means on, 1 means off; G max is the upper limit of the total number of switch actions within the scheduling period, G max,m is the upper limit of the total number of switches m; P i,t , Q i,t are the active power and reactive power injected into the node, G i,j , B i,j are the branch conductance and branch susceptance between nodes ij, V i,t is the voltage of node i at time t, V min 、V max are the upper and lower limits of the node voltage respectively; P ess,min,j , P ess,max,j are the upper and lower limits of energy storage charging and discharging power, respectively, c,j , η d,j are the charge and discharge coefficients of the jth energy storage, E j,t is the capacity of the jth energy storage at the tth time, E max,j 、E min,j is the upper and lower limits of the corresponding energy storage capacity, δ SOC,t It is the charge state of the energy storage at the current moment.

[0044] Step (2), construct a topological security layer based on security constraints.

[0045] The present invention introduces a topological security layer to modify topological actions without affecting energy storage. This is because energy storage actions can be limited by a limit function (such as a clamp function) based on the input current capacity of energy storage, which is relatively simple and is therefore not the focus of the present invention.

[0046] First, the topological mask is introduced to correct the initial topological action (also called action mask) output by the action network. The calculation process of generating and using the topological mask is as follows:

[0047]

[0048] Among them, m l is the mask vector of loop l, m l,i is the mask of the controllable branch i in the loop, They are respectively the original data and action-masked data of the controllable branch No. i in the loop (i.e., topological state, [0,1] represents opening and closing); is the original output of the action network, is the masked action output, which is randomly sampled using the re-parameterization technique, gi is the distributed noise to ensure the consistency of action probability, N switch,l is the number of loop branches, c l,i Encode the final action. Here, the re-parameterization technique is used to randomly sample the masked actions in order to optimize the action distribution. Assuming that the masked action A has four dimensions, corresponding to four action options, the maximum value is usually selected as the execution action after softmax, but the max() function is gradient split, such as 0.4, 0.2, 0.2, 0.2. In fact, the first action will always be executed. After the next update, it will become 0.5, 0.1, 0.2, 0.2, which is still the same and discontinuous. The re-parameterization technique randomly generates distributed noise (normal) according to the value of action A and superimposes it to make the distribution of the four actions continuous, which is convenient for training.

[0049] Then, the above specific mask element m is obtained through a special topological connectivity detection algorithm. l This detection algorithm can effectively obtain the constraint relationship between masks and reduce the amount of detection calculation. The tree node is defined as: a node whose linked non-tree branch is less than or equal to 1; the tree branch is defined as: a branch containing a tree node; all branches are non-tree branches in the initial state. Figure 2 , the yellow dots are tree nodes. At the beginning, there are no tree nodes or tree branches, all of which are non-tree nodes and non-tree branches. After disconnecting a branch, the two ends of the disconnected branch are detected. The number of non-tree branches connected to a node becomes 1, so the node becomes a tree node, and the connection branch of the newly generated tree node becomes a tree branch. The other node of the new tree branch is detected, and its number of non-tree branches is found to be 2, and the end is completed. Therefore, the above definition is a dynamic and complete marking process. According to graph theory, it can be understood that a tree branch cannot be disconnected. Once disconnected, an island will be formed. Non-tree branches can be disconnected at will. Therefore, the encoding of the tree branch is a mask element. If a branch is a tree branch during the current operation, it is necessary to superimpose a negative infinite mask element on the branch to make it small enough and not selected. In other words, the topology monitoring algorithm of the present invention regards the process of topology generation as extracting a tree topology from the original closed topology. Once the tree nodes and tree branches are formed, they are included in the final tree topology. The disconnection of any tree branch will inevitably form an island. This means that there is no need to repeatedly determine the connectivity of tree branches, and in the absence of isolated islands, a closed-loop topology will not be formed. Under this mechanism, the tree topology constraints of the grid are further equivalent to the tree branch connectivity constraints. The action mask only needs to ensure that the tree branch is not disconnected by the subsequent loop. By dynamically updating all branch categories, all mask elements under the corresponding loop can be obtained at one time without duplication. The principle of the topological connectivity detection algorithm is Figure 2 Shown in.

[0050] Finally, in order to further improve the calculation speed of mask actions, a hash table is used to store action masks. The process of storing and reading the hash table is:

[0051]

[0052] Among them, N' switch is the total number of switches that can be operated before the current circuit, n switch is the total number of valid topologies that can be generated by the branch switch. Matrix B stores all valid states in a one-hot encoding format, and matrix M contains the corresponding mask elements. i,k is the topological code of the current operation, h mask (b i ) is a hash function that retrieves the hash table through direct addressing to obtain the mask element mask(b i ).

[0053] Step (3), predict the initial scheduling strategy based on the integrated action network.

[0054] The present invention predicts and screens the initial actions by integrating multiple action networks, avoiding a large number of low-quality actions in the early stage that affect the training of the model. Different action networks use different initialization parameters, input the state into the action network group to obtain different scheduling schemes, and select the optimal action as the actual scheduling strategy to input into the power grid environment. For each action network, in order to maintain the uniqueness and diversity of its parameters, a cycle counter is set to count the cycle T best The optimal number of actions for each action network in the training set is calculated, and the parameters of the optimal action network are passed to other action networks in a soft update manner to perturb their parameters. At the same time, since the parameter update only relies on the output of the value network for gradient ascent optimization, the probability of inferior actions significantly decreases as the training progresses. Therefore, in a longer period T cut The optimal action frequency of each action network is recorded, and it is deleted from the action network group, and repeated until it degenerates into the original algorithm. The soft update calculation process is expressed as:

[0055] φ others =τ best φ ibest +(1-τ best )φ others

[0056] where φ others To exclude other network parameters of the optimal network, φ ibest is the optimal network parameter, τ best is the corresponding soft update parameter. That is, the model parameters with the subscript best are mixed with the model parameters with the subscript other in a small ratio τ, which is different from the parameter update using gradient, which is usually called soft update in Chinese.

[0057] In step (4), the model parameters are trained using a dual-delay-deterministic policy gradient framework, and the trained model output is used to verify the reconstructed scheduling strategy.

[0058] The double-delayed-deterministic policy gradient algorithm is a reinforcement learning algorithm under the Actor-Critic framework, which consists of two parts: the Actor network, i.e., the policy network, and the Critic network, i.e., the value network. The Actor network outputs actions based on the state, and the Critic network inputs actions and states to evaluate the actions given by the policy network to obtain the value of the actions. On this basis, the framework uses two sets of Critic networks, and takes the smaller one when calculating the target value to suppress the overestimation problem of the network. At the same time, it uses a delayed update method. After the Critic network is updated multiple times, the Actor network is updated to ensure the stability of the policy network training. The calculation process is:

[0059]

[0060] Where Q(·) is the value network, γ is the discount factor, y is the target value, s' and a' are the state and action at the next moment, s and a are the state and action at the current moment. θ is the value network parameter, that is, the parameter to be trained, and N is the number of batches for training. φ is the parameter of the policy network, ▽ φ represents the gradient of the policy network, so the parameter update is obtained through the gradient.

[0061] The present invention fine-tunes the training process in the Actor-Critic framework, and implements dual delays by training the policy network and the value network at different frequencies, one fast and one slow, to ensure stable training of the policy network.

[0062] The reconstructed scheduling strategy is verified using the trained model output. The state data S is input and the model gives the reconstructed scheduling strategy, i.e., topology and energy storage action.

[0063] In order to verify the performance of the proposed method, the actual optimization effect of the model was evaluated based on the constructed optimization scenario.

[0064] Figure 3The figure shows the comparison of the effect of the method of the present invention and other methods in learning scheduling strategies in the same scenario. The solid line represents the reward mean, and the shaded area is the upper and lower limits of the reward fluctuation. It can be observed that the efficiency of the method of the present invention is significantly higher than that of other methods in the early stage of training. This is because the cluster strategy is used to perform a priori screening of actions to ensure that the actions always remain in a better range. At the same time, the model of the present invention tends to converge after 10,000 interactive trainings, ahead of other comparison models. Therefore, although the method of the present invention consumes more computing power in the early stage, its training speed is also improved at the same time. It is noted that the TD3 algorithm and the DDPG algorithm achieved similar effects to the improved algorithm of the present invention in certain periods of some experiments, but the fluctuation of their training reward values ​​was more obvious, and the final reward was smaller than the reward of the present invention.

[0065] Figure 4 The corresponding changes in the network loss of the reconstructed system show that by changing the system topology at different times, the power transmission path in different time scenarios can be reasonably allocated, which effectively reduces the line loss of the distribution network system, and the total daily network loss is reduced by 5.7%.

[0066] In summary, the method of the present invention proposes a reinforcement learning method based on cluster strategy to solve the reconstruction optimization scheduling strategy of the power system. The effectiveness of the proposed method is verified through example analysis.

[0067] Based on the same technical concept as the method embodiment, the present invention also provides a distribution network reconstruction system based on topological security constraints and integrated reinforcement learning, including:

[0068] A power grid environment modeling module is used to construct a reinforcement learning model for distribution network reconstruction tasks based on the power grid environment, including system state quantity design, reward function design, power grid constraint embedding and system steady-state quantity calculation. The purpose of the reinforcement learning model is to minimize network loss and the number of line switch actions by adjusting the system's grid structure and energy storage resources. The model's strategy network gives a reconstruction action based on the current state data of the power grid environment, and calculates the reward for this action based on the reward function, and returns the reward and the next state data of the power grid environment. The state data includes the topological action of the network at the previous moment, the energy storage capacity state at the previous moment, and the node power of the previous period. The reconstruction action includes topological action and energy storage action, wherein the topological reconstruction action is designed to selectively disconnect a controllable branch in different independent loops in turn;

[0069] A topology security layer construction module is used to construct a topology security layer, which is used to detect and correct the topology reconstruction action of the initial calculation of the model using a topology mask through a topology detection algorithm;

[0070] An action combination module is used to combine multiple action networks into an integrated action group, which is used to predict different reconstruction actions through parameter doping mechanism and network deletion mechanism;

[0071] A network training module is used to train the initial policy network based on the reinforcement learning training framework to obtain network parameters for generating a reconfiguration scheduling strategy;

[0072] The network application module is used to use the trained strategy network to obtain the dispatching strategy for distribution network reconstruction according to the system status data.

[0073] It should be understood that the distribution network reconstruction system based on topological safety constraints and integrated reinforcement learning in the embodiment of the present invention can implement all the technical solutions in the above-mentioned method embodiment, and the functions of its various functional modules can be specifically implemented according to the methods in the above-mentioned method embodiments. The specific implementation process can refer to the relevant description in the above-mentioned embodiments, which will not be repeated here.

[0074] The present invention also provides a computer device, comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the distribution network reconstruction method based on topological safety constraints and integrated reinforcement learning as described above are implemented.

[0075] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the distribution network reconstruction method based on topological safety constraints and integrated reinforcement learning as described above are implemented.

[0076] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices (systems), computer equipment or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0077] The present invention is described with reference to a flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and a combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1A device that specifies functions in a process or multiple processes.

[0078] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A function specified in a process or multiple processes.

[0079] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 The steps of a specified function in a process or multiple processes.

Claims

1. A distribution network reconstruction method based on topological security constraints and integrated reinforcement learning, characterized in that: The following steps are involved: Step 1: Construct a power grid environment, which is used to interact with the reinforcement learning model to obtain training data, including system state quantity design, reward function design, power grid constraint embedding and system steady-state quantity calculation. The purpose of the reinforcement learning model is to minimize network loss and the number of line switch actions by adjusting the system's grid structure and energy storage resources; the model's strategy network gives a reconstruction action based on the current state data of the power grid environment, calculates the reward for this action based on the reward function, and returns the reward and the next state data of the power grid environment; the state data includes the topological action of the network at the previous moment, the energy storage capacity state at the previous moment, and the node power of the previous period; the reconstruction action includes topological action and energy storage action, wherein the topological reconstruction action is designed to selectively disconnect a controllable branch in different independent loops in turn; Step 2: Construct a topology security layer. The topology security layer uses an action mask through a topology detection algorithm to detect and correct the topology reconstruction action initially calculated by the model. When it is detected that the branch currently operated is a tree branch, a negative infinite mask element is superimposed on the branch so that it will not be selected. Step 3: Combine multiple action networks into an integrated action group, and predict and screen different reconstruction actions of the integrated action group through parameter doping mechanism and network deletion mechanism. The parameter doping mechanism sets a cycle counter for each action network, counts the number of optimal actions of each action network in the first cycle, passes the parameters of the optimal action network to other action networks, and perturbs their parameters; the network deletion mechanism records the optimal action frequency of each action network in the second cycle and deletes it from the action network group; Step 4: Use the dual-delay-deterministic policy gradient framework to train the reinforcement learning model to obtain network parameters for generating the reconstruction scheduling strategy; Step 5: Use the trained reinforcement learning model to obtain the corresponding distribution network reconstruction scheduling strategy based on the current system status data.

2. The method according to claim 1, characterized in that The variable design of the power grid environment is as follows: Among them, S is the system state, A topo,t-1 is the topological action at the previous moment, E t-1 is the energy storage capacity state at the last moment, P is the node power in the last period, R is the reward function, k price is the economic coefficient of electricity purchase, P ess,j,t is the charge and discharge power of the jth energy storage at the current moment, c pur is the current electricity price, c pur,ess is the current average electricity price of energy storage, k loss is the network loss coefficient, P loss is the system network loss after the action is executed, k switch is the switching operation coefficient, N switch is the number of switch actions, A is the strategy network action, and A topo , P ess are topology and energy storage action respectively, L is the set of independent loop numbers, ε switch,l is the set of all controllable branch numbers under independent loop l, a l is the discrete action given for independent loop l, c l,i It is the corresponding action code, 0 means connected, 1 means disconnected.

3. The method according to claim 2, characterized in that The power grid environment includes topology constraints, power flow constraints, and energy storage capacity constraints. The system steady state is obtained by solving the power flow equation. The constraint calculation formula is: Among them, G is the total number of actions of all switches in the scheduling period, G t,m is the total number of actions of switch m within the scheduling period; H m,t is the state of switch m at time t, 0 means on, 1 means off; G max is the upper limit of the total number of switch actions within the scheduling period, G max,m is the upper limit of the total number of switches m; P i,t , Q i,t are the active power and reactive power injected into the node, G i,j , B i,j are the branch conductance and branch susceptance between nodes ij, V i,t is the voltage of node i at time t, V min 、V max are the upper and lower limits of the node voltage respectively; P ess,min,j , P ess,max,j are the upper and lower limits of energy storage charging and discharging power, respectively, c,j , η d,j are the charge and discharge coefficients of the jth energy storage, E j,t is the capacity of the jth energy storage at the tth time, E max,j 、E min,j is the upper and lower limits of the corresponding energy storage capacity, δ SOC,t It is the charge state of the energy storage at the current moment.

4. The method according to claim 1, characterized in that: The computational process of generating and using action masks is expressed as: Among them, m l is the mask vector of loop l, m l,i is the mask of the controllable branch i in the loop, They are respectively the original data and the action-masked data of the controllable branch No. i in the loop; is the original output of the action network, is the masked action output, randomly sampled using the re-parameterization technique, g i is the distributed noise to ensure the consistency of action probability, N switch,l is the number of loop branches, c l,i Encode the final action.

5. The method according to claim 4, characterized in that The topology detection algorithm uses action masks to detect and correct the topology reconstruction actions initially calculated by the model, including: defining a tree node as a node whose linked non-tree branches are less than or equal to 1; defining a tree branch as a branch that contains a tree node; all branches in the initial state are non-tree branches; the topology detection algorithm regards the process of topology generation as extracting a tree topology from the original closed topology. Once the tree nodes and tree branches are formed, they are included in the final tree topology. The encoding of the tree branch is the mask element. When it is detected that the branch currently being operated is a tree branch, a negative infinite mask element is superimposed on the branch to prevent it from being selected.

6. The method according to claim 4, characterized in that The action mask is stored in a hash table. The process of storing and reading the hash table is expressed as: Among them, N' switch is the total number of switches that can be operated before the current circuit, n switch is the total number of valid topologies that can be generated by the branch switch. Matrix B stores all valid states in a one-hot encoding format, and matrix M contains the corresponding mask elements. i,k is the topological code of the current operation, h mask (b i ) is a hash function that retrieves the hash table through direct addressing to obtain the mask element mask(b i ).

7. The method according to claim 1, characterized in that The double-delayed-deterministic policy gradient framework trains the reinforcement learning model. The calculation process is: Among them, Q(·) is the value network, γ is the discount coefficient, y is the target value, s' and a' are the state and action at the next moment, s and a are the state and action at the current moment, θ is the value network parameter, that is, the parameter to be trained, N is the number of batches for training, and φ is the parameter of the policy network. represents the gradient of the policy network.

8. A distribution network reconstruction system based on topological security constraints and integrated reinforcement learning, characterized in that: include: A power grid environment modeling module is used to construct a power grid environment, which is used to interact with a reinforcement learning model to obtain training data, including system state quantity design, reward function design, power grid constraint embedding and system steady-state quantity calculation. The purpose of the reinforcement learning model is to minimize network loss and the number of line switch actions by adjusting the system's grid structure and energy storage resources. The model's strategy network gives a reconstruction action based on the current state data of the power grid environment, calculates the reward for this action based on the reward function, and returns the reward and the next state data of the power grid environment. The state data includes the topological action of the network at the previous moment, the energy storage capacity state at the previous moment, and the node power of the previous period. The reconstruction action includes topological action and energy storage action, wherein the topological reconstruction action is designed to selectively disconnect a controllable branch in different independent loops in turn. A topology security detection module is used to construct a topology security layer. The topology security layer uses an action mask through a topology detection algorithm to detect and correct the topology reconstruction action initially calculated by the model. When it is detected that the branch currently operated is a tree branch, a negative infinite mask element is superimposed on the branch so that it will not be selected. The action network pre-screening module is used to combine multiple action networks into an integrated action group, and to prejudge and screen different reconstruction actions of the integrated action group through a parameter doping mechanism and a network deletion mechanism. The parameter doping mechanism sets a cycle counter for each action network, counts the number of optimal actions of each action network in the first cycle, and passes the parameters of the optimal action network to other action networks to perturb their parameters; the network deletion mechanism records the optimal action frequency of each action network in the second cycle and deletes it from the action network group; A reinforcement learning model training module is used to train the reinforcement learning model using a dual-delay-deterministic policy gradient framework to obtain network parameters for generating a reconstruction scheduling strategy; The distribution network reconstruction strategy determination module is used to use the trained reinforcement learning model to obtain the corresponding distribution network reconstruction scheduling strategy according to the current system status data.

9. A computer device, characterized in that: include: one or more processors; Memory; And one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the programs are executed by the processors, the steps of the distribution network reconstruction method based on topological safety constraints and integrated reinforcement learning as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distribution network reconstruction method based on topological safety constraints and integrated reinforcement learning are implemented as described in any one of claims 1 to 7.

Images