Power distribution system dynamic reconstruction method based on deep reinforcement learning and computer device

Abstract

The invention relates to the field of power distribution system recovery, in particular to a power distribution system dynamic reconstruction method based on deep reinforcement learning and a computer device, so that a power distribution system operator can make a real-time decision according to the current state of a power distribution system, and the elasticity of the power distribution system is greatly improved. The method comprises the following steps: modeling the whole recovery process of the power distribution system as a Markov decision process, wherein the decision process is solved by using a deep reinforcement learning model; adding a cold load impact constraint, a power flow constraint, a radial topology constraint and a frequency response constraint in the whole recovery process of the power distribution system; optimizing the whole recovery process of the power distribution system; a deep reinforcement learning model fused with a physical information neural network is adopted to update and optimize parameters; training the deep reinforcement learning model; and performing dynamic reconstruction and sequential recovery of the power distribution system according to the trained deep reinforcement learning model. The method is suitable for power distribution system recovery.

Description

Technical Field

[0001] The present invention relates to the field of power distribution system restoration, and in particular to a power distribution system dynamic reconstruction method and a computer device based on deep reinforcement learning. Background Art

[0002] Nowadays, distribution systems are developing towards a more flexible, efficient and reliable direction thanks to the integration of intelligent devices (such as distributed generators, smart meters, and remote control switchgear). However, frequent extreme weather events pose a great threat to the safe and reliable operation of modern distribution systems such as green building clusters. They face the risk of large-scale power outages, which will cause serious damage to infrastructure and bring huge economic losses. In the past decade, due to the frequent occurrence of extreme weather caused by global climate change and the large-scale application of renewable energy generation that is highly dependent on weather, enhancing the resilience of power systems to cope with extreme weather conditions has become a major issue emphasized by the global energy system. At the same time, with the grid connection of massive renewable energy generation, power electronics devices, and information devices and other fast-changing resources, the physical and information characteristics of the new generation of power systems are more complex. In addition, due to the diversity of extreme weather conditions, the traditional power system's operation mode of formulating resilience improvement strategies one by one for given extreme weather scenarios based on traditional component characteristics and known complete information cannot reflect the essential response characteristics of the system structure. Therefore, it is urgent to develop new systematic theories and methods for evaluating and enhancing the resilience of power system structures based on the essential characteristics of the new generation of power systems. Smart facilities can ensure power supply to important loads when the distribution system is forced to disconnect from the main grid.

[0003] At present, most of the research on distribution system network reconstruction focuses on considering various uncertainties, and their solutions are mostly based on traditional optimization algorithms. However, the use of optimization algorithms to solve the problem requires obtaining the complete data parameters of the system before calculation. On the one hand, it is impractical to collect complete system data and obtain accurate network parameters after the disaster occurs. On the other hand, the solution of traditional optimization algorithms is very time-consuming, especially when considering various system uncertainties, the computational complexity often grows exponentially.

[0004] Considering that AI-based algorithms have great difficulty in dealing with many complex physical constraints. Existing research usually terminates the current round of training when the agent violates any operational constraints, which will lead to insufficient learning of the strategy by the agent or even failure to complete the learning. More importantly, few studies focus on how to improve AI algorithms to adapt to the optimization problem of improving actual resilience, and models that lack physical information cannot allow the calculation results to be actually applied to real power systems. Therefore, how to incorporate physical information into AI algorithms is the key to applying AI to the problem of improving the resilience of distribution systems. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and computer device for dynamic reconstruction of a distribution system based on deep reinforcement learning, so that the distribution system operator can make real-time decisions based on the current state of the distribution system, greatly improving the elasticity of the distribution system.

[0006] The present invention adopts the following technical solutions to achieve the above-mentioned purpose. In a first aspect, the present invention provides a method for dynamic reconstruction of a power distribution system based on deep reinforcement learning, comprising:

[0007] S1. The entire recovery process of the distribution system is modeled as a Markov decision process, which consists of a state set, an action set, a transition probability, a reward function, and a discount factor. The decision maker selects an action, and the environment transfers to the next state according to the action, and obtains corresponding rewards according to the transfer result. By solving the Markov decision process, the optimal strategy for the recovery of the distribution system under the current state is found;

[0008] S2, adding cold load impact constraints, power flow constraints, radial topology constraints and frequency response constraints during the entire restoration process of the distribution system;

[0009] S3, optimize the entire recovery process of the power distribution system;

[0010] S4, using a deep reinforcement learning model that integrates physical information neural network to perform parameter update optimization;

[0011] S5. Train the deep reinforcement learning model;

[0012] S6. Dynamically reconstruct and sequentially restore the power distribution system based on the trained deep reinforcement learning model.

[0013] Furthermore, in step S1, the state set of the Markov decision process includes the following elements:

[0014] The system topology status at time t, the status of the line switch at time t, the load recovery status at time t, the active / reactive power of the restored load at time t, and the voltage of each node of the system at time t;

[0015] The action set of a Markov decision process consists of the following elements:

[0016] The circuit breaker behavior, whether it is closed at the current moment;

[0017] Load recovery decision behavior, whether to restore at the current moment;

[0018] The reward function considers multiple angles based on the given behavior. When the given behavior violates the physical operation constraints, a negative reward will be obtained. At the same time, a corresponding positive reward will be obtained based on the importance of restoring the load at the current moment. The elements considered by the reward function include:

[0019] Priority of restoring loads, whether diesel generators are overloaded, whether radial constraints are violated, and whether frequency response constraints are violated;

[0020] The Markov decision process is solved using a deep reinforcement learning algorithm. This state transition probability (matrix) can be implemented using a neural network. The neural network learns the state transition process by updating parameters. The deep learning algorithm uses a policy neural network to achieve the mapping of states to actions. The mapping process is as follows:

[0021] The agent observes the state of the environment and models it as a vector;

[0022] Transcode the vector into a tensor, input the tensor into the neural network, encode the information, and output the action vector;

[0023] The agent executes the action vector.

[0024] Furthermore, in step S2, the cold load impact constraint is as follows:

[0025]

[0026] in and are the maximum deviation coefficient and the normal deviation coefficient of the constant temperature load under the influence of the cold load shock, N represents the total number of recovery operation periods, D represents the total number of loads, and a d,t is the decay rate of the cold load shock phenomenon, D d,t The time period from when the load starts to charge to when the diversity is obtained; C d,t is the time period from when the load starts to obtain diversity to time t, expressed as:

[0027] C d,t =(t-1)Δt-D d,t ;

[0028] The active power demand and reactive power demand of the load when it is restored are:

[0029]

[0030] in, is a binary variable, indicating the charging state of the load, ΔP d (k) and ΔQ d (k) According to the formula L d(t) is calculated to represent the difference between active power and reactive power at time t and time t-1, respectively. and They represent the rated active power and reactive power of load d respectively.

[0031] The power flow constraints are as follows:

[0032]

[0033] in, They are active power flow, reactive power flow, active generator output, reactive generator output, remote switch control state, and node i voltage state; and They represent the maximum active and reactive power flows allowed to pass through the line, δ ij represents the connectivity status of the line, ε represents the set of connected edges, r ij With x ij They represent the resistance and reactance of the line respectively.

[0034] The radial topology constraints are as follows:

[0035] σ∈Ω

[0036]

[0037] Among them, σ ij,t Describes all possible tree topologies of the system, δ swi,t Indicates the open or closed state of the switch. Indicates the number of nodes in the system. Indicates the number of root nodes in the tree structure. represents the flow rate of the virtual flow, σ ij,t Indicates whether the line has a switch that can be opened and closed, and M is a real number.

[0038] The frequency response constraints are as follows:

[0039]

[0040] The frequency response constraint indicates that the load restored by the distribution system at each step is limited by the frequency response constraints of all units, γ FRR Represents the system frequency response control coefficient.

[0041] Furthermore, in step S3, the objective function of optimization is to maximize the load recovery amount, and the objective function of the optimization problem is:

[0042] where ω d represents the weight of the load, o d,t Indicates the recovery status of the load, is the load demand of load d at time t.

[0043] Furthermore, step S4 specifically includes:

[0044] The load information and grid topology information are input in the form of a matrix, and the system state information is encoded using a convolutional neural network. After the state information is input, it is processed by the policy neural network and the initial operation action of the intelligent agent is output. The action includes the load position to be restored in the current state and the line switch action;

[0045] At the same time, according to the current state information, the feasible solution of the recovery strategy under the current state is obtained by solving the single-period optimization problem. The objective function of the optimization problem is to maximize the number of load restoration. After linearizing the nonlinear constraints, the commercial solver is used to solve the mixed integer programming to obtain the solution of the problem.

[0046] The optimization problem is integrated into the parameter update of the neural network as knowledge, the behavior vector of the feasible solution obtained by the optimization problem is calculated, and the cosine similarity between it and the initial action vector given by the intelligent agent is calculated, and the cosine similarity is combined with the original policy gradient function for optimization.

[0047] Furthermore, step S5 specifically includes:

[0048] The policy network of the deep reinforcement learning model makes decisions based on the observed environmental state, and the value network of the deep reinforcement learning model gives an evaluation value of the decision. At each time step, the agent outputs an action from the policy network based on the current state and interacts with the environment to obtain rewards and the next state. Then, the state, action, reward, and next state are stored in a buffer using the experience replay mechanism. Next, these experiences are randomly sampled to update the policy network and value network. The value network is optimized by minimizing the mean square error, and the deterministic policy gradient is used to update the policy network. By continuously iterating this process, the agent learns the optimal strategy.

[0049] Furthermore, in step S6, performing sequential recovery specifically includes:

[0050] When the distribution system detects a fault, data collection is carried out. The collected data includes load recovery status, power demand of restored load, line connectivity status, system voltage parameters and system flow parameters. The collected information is then encoded and input into the deep reinforcement learning model. The strategy network of the deep reinforcement learning model will derive the optimal recovery decision for the system at the current moment based on the obtained system status information. The recovery decision will be transmitted to each sensor through communication equipment and executed by the distribution system operator. After that, the system status information will be updated. Through this iterative behavior, the distribution system will execute sequential recovery decisions step by step.

[0051] In a second aspect, the present invention provides a computer device, including a memory, wherein the memory stores program instructions, and is characterized in that when the program instructions are run, the above-mentioned dynamic reconstruction method of the distribution system based on deep reinforcement learning is executed.

[0052] The beneficial effects of the present invention are:

[0053] The present invention is based on a deep reinforcement learning algorithm model for the sequential restoration of distribution systems and the formation of dynamic microgrids, and takes into account the uncertainty of distributed low-voltage user conditions by utilizing topological flexibility. The load uncertainty inherited from the distributed low-voltage user conditions during the restoration process is solved through a Markov decision process, and a deep deterministic policy gradient algorithm framework integrating physical information neural networks is developed, through which the agent can better learn the physical information of operational constraints. The method of the present invention can be trained through offline and online learning. In offline training, OpenDSS is used to simulate the system simulation environment. In particular, online learning can improve the adaptability of the strategy to actual restoration scenarios. In addition, the recovery strategy also fully considers the cold load shock phenomenon of the building constant temperature load under the uncertainty of environmental factors.

[0054] This invention integrates physical information neural network into reinforcement learning algorithm for the first time, and incorporates physical information into the learning strategy of intelligent agent, which further improves the robustness and reliability of recovery strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of a method for dynamic reconstruction of a power distribution system based on deep reinforcement learning provided by an embodiment of the present invention;

[0056] Figure 2 is a schematic diagram of the topological structure of the power distribution system provided by an embodiment of the present invention, the left side of the dotted line is a schematic diagram of the sequential restoration process without considering dynamic reconstruction, and the right side of the dotted line is a schematic diagram of the sequential restoration process with dynamic reconstruction;

[0057] Figure 3 is a schematic diagram of a sequential recovery framework provided by an embodiment of the present invention;

[0058] Figure 4 is a schematic diagram of cold load impact provided by an embodiment of the present invention;

[0059] Figure 5 is a schematic diagram of actual load power fluctuation provided by an embodiment of the present invention;

[0060] Figure 6 is a schematic diagram of a physical information neural network framework provided by an embodiment of the present invention;

[0061] Figure 7is a schematic diagram of a microgrid after dynamic reconstruction provided by an embodiment of the present invention;

[0062] Figure 8 It is a flow chart of a sequential recovery process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0064] Common approaches to sequential restoration of power systems can be divided into two categories: model-based and AI-based. AI-based approaches enable distribution system operators to make real-time decisions based on the current state of the system, thus providing greater potential for improving system resilience. Although the above-mentioned model-free approaches have shown great potential in sequential restoration decision-making, there are still some challenges that need to be addressed. This work focuses on deep reinforcement learning algorithms. On the one hand, the training environment should contain more real-world scenario elements (e.g., uncertainties in distributed low-voltage users) to make the strategy practical. On the other hand, more physical information, such as physical operating constraints, should be integrated into such model-free approaches. Given that most deep reinforcement learning methods use neural networks as state transition functions, integrating physical information can significantly improve interpretability and enable the agent to learn strategies more effectively.

[0065] Based on this, the present invention uses a deep reinforcement learning algorithm model for the sequential restoration of the distribution system and the formation of a dynamic microgrid, and takes into account the uncertainty of the conditions of distributed low-voltage users using topological flexibility. The load uncertainty inherited from the conditions of distributed low-voltage users in the restoration process is solved through a Markov decision process, and a deep reinforcement learning model integrating physical information neural network is developed, through which the agent can better learn the physical information of the operation constraints.

[0066] When the main grid faces a security threat from the distribution system, the distribution system will be disconnected from the main grid and continue to supply power using its own backup energy. Therefore, efficient microgrid reconstruction technology and sequential restoration are of great significance to improving the resilience of the distribution system. Considering that when the load experiences a long power outage, a cold load shock phenomenon will occur, that is, some constant temperature loads, such as air-conditioning systems or refrigeration equipment, when these devices are started, the sudden increase in load demand causes the instantaneous load of the equipment to increase. This phenomenon may cause the transformer to overload, thereby hindering system recovery. At this time, making full use of the system's topological flexibility can improve recovery efficiency, such as Figure 2 shown.

[0067] The left side shows the sequential recovery process without considering dynamic reconstruction, that is, without taking advantage of topological flexibility. It can be seen that the generator cannot be fully utilized. The right side makes full use of the topological flexibility of the network when performing load sequential recovery. It can be seen that the recovery strategy on the right has a great advantage in terms of recovery time.

[0068] In addition, there will be multiple uncertainties during restoration, such as environmental factors, load demand, etc. And in extreme scenarios, it is very difficult to obtain power grid data, and it is difficult to use traditional optimization algorithms to solve. Therefore, the present invention uses artificial intelligence algorithms to solve the sequential restoration problem and considers the uncertainty of load demand. The proposed algorithm can be combined with the fault management system in the distribution system. The proposed framework is as follows Figure 3 When the system detects a fault, the system data information is uploaded to the control center. The control center uses the algorithm to obtain the optimal scheduling decision of the system under this state and remotely control the corresponding equipment.

[0069] Specifically, the present invention provides a method for dynamic reconstruction of a power distribution system based on deep reinforcement learning, such as Figure 1 As shown, specifically including:

[0070] S1. The entire restoration process of the distribution system is modeled as a Markov decision process, which is solved using a deep reinforcement learning model;

[0071] S2, adding cold load impact constraints, power flow constraints, radial topology constraints and frequency response constraints during the entire restoration process of the distribution system;

[0072] S3, optimize the entire recovery process of the power distribution system;

[0073] S4, using a deep reinforcement learning model that integrates physical information neural network to perform parameter update optimization;

[0074] S5. Train the deep reinforcement learning model;

[0075] S6. Dynamically reconstruct and sequentially restore the power distribution system based on the trained deep reinforcement learning model.

[0076] Specifically, the Markov decision process consists of a state set, an action set, a transition probability, a reward function, and a discount factor. The decision maker selects an action, and the environment transfers to the next state according to the action and obtains corresponding rewards according to the transfer result. By solving the Markov decision process, the optimal strategy for restoring the distribution system under the current state is found;

[0077] The state set of a Markov decision process contains the following elements:

[0078] The system topology status at time t, the status of the line switch at time t, the load recovery status at time t, the active / reactive power of the restored load at time t, and the voltage of each node of the system at time t;

[0079] The action set of a Markov decision process consists of the following elements:

[0080] The circuit breaker behavior, whether it is closed at the current moment;

[0081] Load recovery decision behavior, whether to restore at the current moment;

[0082] The reward function considers multiple angles based on the given behavior. When the given behavior violates the physical operation constraints, a negative reward will be obtained. At the same time, a corresponding positive reward will be obtained based on the importance of restoring the load at the current moment. The elements considered by the reward function include:

[0083] Priority of restoring loads, whether diesel generators are overloaded, whether radial constraints are violated, and whether frequency response constraints are violated;

[0084] Due to the multiple uncertainties in the sequential restoration process of the distribution system, the state transition probability is difficult to determine in advance. The Markov decision process is solved using a deep reinforcement learning algorithm, and this state transition probability (matrix) can be implemented using a neural network. The neural network learns the state transition process by updating parameters. The deep learning algorithm uses a policy neural network to achieve the mapping of states to actions. The mapping process is as follows:

[0085] The agent observes the state of the environment and models it as a vector;

[0086] Transcode the vector into a tensor, input the tensor into the neural network, encode the information, and output the action vector;

[0087] The agent executes the action vector.

[0088] In addition, the discount factor is considered during the training process of the deep reinforcement learning algorithm. The discount factor is a hyperparameter that can be fine-tuned based on the performance of the model.

[0089] Specifically, the present invention considers the cold load shock phenomenon that occurs when the load is restored. That is, when the load faces a long power outage, it will lose its diversity. At this time, if the load is powered on, its power demand will far exceed the power during the power outage. This phenomenon will affect the power demand when the load is restored, that is, the load power demand will change dynamically according to the characteristics of this phenomenon. Figure 4 As shown, the present invention models the cold load impact process as the following mathematical expression, and considers these constraints when updating the state to update the power demand of the load.

[0090] The cooling load impulse constraints are as follows:

[0091]

[0092] in, and are the maximum deviation coefficient and the normal deviation coefficient of the constant temperature load under the influence of the cold load shock, N represents the total number of recovery operation periods, D represents the total number of loads, and a d,t is the decay rate of the cold load shock phenomenon, D d,t The time period from when the load starts to charge to when the diversity is obtained; C d,t is the time period from when the load starts to obtain diversity to time t, expressed as:

[0093] C d,t =(t-1)Δt-D d,t ;

[0094] The active power demand and reactive power demand of the load when it is restored are:

[0095]

[0096] in, is a binary variable, indicating the charging state of the load; ΔP d (k) and ΔQ d (k) According to the formula L d (t) is calculated to represent the difference between active power and reactive power at time t and time t-1, respectively. and They represent the rated active power and reactive power of load d respectively.

[0097] Specifically, the sequential restoration problem needs to consider the following types of constraints: power flow constraints, cold load impact constraints, radial topology constraints, and frequency response constraints. The optimization objective function is to maximize the load restoration amount. The objective function of the optimization problem is:

[0098] where ω d represents the weight of the load, o d,t Indicates the recovery status of the load, is the load demand of load d at time t.

[0099] The power flow constraints are as follows:

[0100]

[0101] in, They are active power flow, reactive power flow, active generator output, reactive generator output, remote control switch control state, and i-node voltage state. and They represent the maximum active and reactive power flows allowed to pass through the line, δ ij represents the connectivity status of the line, ε represents the set of connected edges, r ij With x ij The present invention takes into account the uncertainty of load during the recovery process. Therefore, the real load demand curve is Figure 4 The curve shown is the baseline, which fluctuates randomly up and down by 10%. Figure 5 shown.

[0102] The radial topology constraints are as follows:

[0103] σ∈Ω

[0104]

[0105] Among them, σ ij,t Describes all possible tree topologies of the system, δ swi,t Indicates the open or closed state of the switch. Indicates the number of nodes in the system. Indicates the number of root nodes in the tree structure. represents the flow rate of the virtual flow, σ ij,t Indicates whether the line has a switch that can be opened and closed, M is a sufficiently large real number;

[0106] The frequency response constraints are as follows:

[0107]

[0108] The frequency response constraint means that the load restored by the distribution system at each step is limited by the frequency response constraints of all units. FRR Represents the system frequency response control coefficient.

[0109] Specifically, the present invention adopts a physical information neural network to improve the learning effect of the intelligent agent on physical constraints. The present invention integrates the deep reinforcement learning model parameter update framework of the physical information neural network as follows: Figure 6 As shown, the load information includes: load recovery status and active / reactive power demand of the restored load, and the grid topology includes the closing status of the line switch and the number of microgrids in the system.

[0110] The load information and grid topology information are input in the form of a matrix for further tensor transformation. Then, a convolutional neural network is used to encode the system state information. After the state information is input, it is processed by the policy neural network of the deep reinforcement learning model to output the initial operation action of the intelligent agent. The action includes the load position to be restored in the current state and the line switch action;

[0111] At the same time, according to the current state information, the feasible solution of the recovery strategy under the current state is obtained by solving the single-period optimization problem. The objective function of the optimization problem is to maximize the number of load restoration. After linearizing the nonlinear constraints, the commercial solver is used to solve the mixed integer programming to obtain the solution of the problem.

[0112] The optimization problem is integrated into the parameter update of the neural network as knowledge, the behavior vector of the feasible solution obtained by the optimization problem is calculated, and the cosine similarity between it and the initial action vector given by the intelligent agent is calculated, and the cosine similarity is combined with the original policy gradient function for optimization.

[0113] Specifically, the present invention adopts a deep reinforcement learning algorithm model to solve the proposed sequential recovery problem. The power flow constraint is systematically simulated by Open DSS software to simulate the most realistic system state. At the same time, the deep reinforcement learning algorithm code is written in python. Through the interface, the deep reinforcement learning algorithm will interact with the simulation environment built by OpenDSS to realize the learning of the intelligent agent strategy.

[0114] In the training phase of the deep reinforcement learning algorithm model, first, the policy network of the deep reinforcement learning model makes a decision based on the observed environmental state, and the value network gives the evaluation value of the decision. At each time step, the agent outputs an action from the policy network based on the current state, and interacts with the environment to obtain rewards and the next state. Then, using the experience replay mechanism, the state, action, reward, and next state are stored in a buffer. Next, these experiences are randomly sampled to update the policy network and value network, the value network is optimized by minimizing the mean square error, and the policy network is updated using deterministic policy gradients. By continuously iterating this process, the agent can learn the optimal strategy.

[0115] During the deep reinforcement learning algorithm model testing phase, the intelligent agent will directly use the policy network to take actions based on the current system state without the need to update the gradient again.

[0116] The present invention uses the DQN (deep Q-network) algorithm results as a reference to compare the network reconstruction results. The network topology obtained by the DQN algorithm is as follows: Figure 7 shown.

[0117] The present invention provides sequential recovery results, as shown in Table 1.

[0118] Table 1 Comparison of sequential recovery results

[0119] Model No. Restore system power Recovery load quantity Load recovery rate 1 6124.02KW 128 100％ 2 5702.91KW 117 91.41％ 3 5908.77KW 125 97.66％ 4 5409.35KW 127 99.22％

[0120] In Table 1, model number 1 uses OU-Noise for exploration actions, and the model is combined with a physical information neural network. Model number 2 uses Gaussian noise for exploration actions, and the model is combined with a physical information neural network. Model number 3 uses OU-Noise for exploration actions, and the model is not combined with a physical information neural network. Model number 4 uses the Deep Q-learning algorithm.

[0121] It can be seen from the data in Table 1 that the recovery strategy provided by the deep reinforcement learning algorithm model of the present invention has a better load recovery rate than the compared method.

[0122] The process of sequential recovery of the present invention is as follows Figure 8 As shown, first the distribution system fault detection is performed, and after the fault is detected, the fault is isolated. Then the distribution system collects data, and then sends the system data to the central control end (control center). The strategy is derived based on the strategy network, and then the strategy is executed to update the system status and finally complete the recovery.

[0123] Specifically, when the distribution system detects a fault, data collection is carried out. The collected data includes load recovery status, power demand of the restored load, line connectivity status, system voltage parameters and system flow parameters. The collected information is then encoded and input into the deep reinforcement learning model. The strategy network of the deep reinforcement learning model will derive the optimal recovery decision for the system at the current moment based on the obtained system status information. The recovery decision will be transmitted to each sensor through communication equipment and executed by the distribution system operator. After that, the system status information will be updated. Through this iterative behavior, the distribution system will execute sequential recovery decisions step by step.

[0124] The above is only a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the concept described herein through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not deviate from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A dynamic reconstruction method of power distribution system based on deep reinforcement learning, characterized in that: include: S1. The entire restoration process of the distribution system is modeled as a Markov decision process, which is solved using a deep reinforcement learning model; S2, adding cold load impact constraints, power flow constraints, radial topology constraints and frequency response constraints during the entire restoration process of the distribution system; S3, optimize the entire recovery process of the power distribution system; S4, using a deep reinforcement learning model that integrates physical information neural network to perform parameter update optimization; S5. Train the deep reinforcement learning model; S6. Dynamically reconstruct and sequentially restore the power distribution system based on the trained deep reinforcement learning model.

2. The method for dynamic reconstruction of a power distribution system based on deep reinforcement learning according to claim 1, characterized in that: In step S1, the Markov decision process consists of a state set, an action set, a transition probability, a reward function and a discount factor. The decision maker selects an action, and the environment transfers to the next state according to the action, and obtains corresponding rewards according to the transfer result. By solving the Markov decision process, the optimal strategy for restoring the distribution system under the current state is found; The state set of a Markov decision process contains the following elements: The system topology status at time t, the status of the line switch at time t, the load recovery status at time t, the active / reactive power of the restored load at time t, and the voltage of each node of the system at time t; The action set of a Markov decision process consists of the following elements: The circuit breaker behavior, whether it is closed at the current moment; Load recovery decision behavior, whether to restore at the current moment; The reward function considers multiple angles based on the given behavior. When the given behavior violates the physical operation constraints, a negative reward will be obtained. At the same time, a corresponding positive reward will be given based on the importance of restoring the load at the current moment. The elements considered in the reward function include: Priority of restoring loads, whether diesel generators are overloaded, whether radial constraints are violated, and whether frequency response constraints are violated; The Markov decision process is solved using a deep reinforcement learning algorithm. This state transition probability (matrix) can be implemented using a neural network. The neural network updates parameters and learns the state transition process. The deep learning algorithm uses a strategy neural network to achieve the mapping of states to actions. The mapping process is as follows: The agent observes the state of the environment and models it as a vector; Transcode the vector into a tensor, input the tensor into the neural network, encode the information, and output the action vector; The agent executes the action vector.

3. The method for dynamic reconstruction of a power distribution system based on deep reinforcement learning according to claim 1, characterized in that: In step S2, the cold load impact constraint is as follows: in and are the maximum deviation coefficient and the normal deviation coefficient of the constant temperature load under the influence of the cold load shock, N represents the total number of recovery operation periods, D represents the total number of loads, and a d,t is the decay rate of the cold load shock phenomenon, D d,t The time period from when the load starts to charge to when the diversity is obtained; C d,t is the time period from when the load starts to obtain diversity to time t, expressed as: C d,t =(t-1)Δt-D d,t ; The active power demand and reactive power demand of the load when it is restored are: in, is a binary variable, indicating the charging state of the load, ΔP d (k) and ΔQ d (k) According to the formula L d (t) is calculated to represent the difference between active power and reactive power at time t and time t-1, respectively. and Respectively represent the rated active power and reactive power of load d; The power flow constraints are as follows: in, P g,t ,Q g,t ,δ ij,t ,u i,t They are active power flow, reactive power flow, active generator output, reactive generator output, remote switch control state, i node voltage state, and They represent the maximum active and reactive power flows allowed to pass through the line, δ ij represents the connectivity status of the line, ε represents the set of connected edges, r ij With x ij They represent the resistance and reactance of the line respectively; The radial topology constraints are as follows: σ∈Ω d swi,t ≤σ ij,t ,sωi on(i,j) Among them, σ ij,t Describes the set of all possible tree topologies of the distribution system, δ swi,t Indicates the open or closed state of the switch. represents the number of nodes in the power distribution system, Indicates the number of root nodes in the tree structure. represents the flow rate of the virtual flow, σ ij,t Indicates whether the line has a switch that can be opened and closed, M is a real number; The frequency response constraints are as follows: The frequency response constraint indicates that the load restored by the distribution system at each step is limited by the frequency response constraints of all units, γ FRR Represents the system frequency response control coefficient.

4. The method for dynamic reconstruction of a power distribution system based on deep reinforcement learning according to claim 1, characterized in that: In step S3, the objective function of optimization is to maximize the load recovery amount. The objective function of the optimization problem is: where ω d represents the weight of the load, o d,t Indicates the recovery status of the load, is the load demand of load d at time t.

5. The method for dynamic reconstruction of a power distribution system based on deep reinforcement learning according to claim 1, characterized in that: Step S4 specifically includes: The load information and grid topology information are input in the form of a matrix, and the system state information is encoded using a convolutional neural network. After the state information is input, it is processed by the policy neural network and the initial operation action of the intelligent agent is output. The action includes the load position to be restored in the current state and the line switch action; At the same time, according to the current state information, the feasible solution of the recovery strategy under the current state is obtained by solving the single-period optimization problem. The objective function of the optimization problem is to maximize the number of load restoration. After linearizing the nonlinear constraints, the commercial solver is used to solve the mixed integer programming to obtain the solution of the problem. The optimization problem is integrated into the parameter update of the neural network as knowledge, the behavior vector of the feasible solution obtained by the optimization problem is calculated, and the cosine similarity between it and the initial action vector given by the intelligent agent is calculated, and the cosine similarity is combined with the original policy gradient function for optimization.

6. The method for dynamic reconstruction of a power distribution system based on deep reinforcement learning according to claim 1, characterized in that: Step S5 specifically includes: The policy network of the deep reinforcement learning model makes decisions based on the observed environmental state, and the value network of the deep reinforcement learning model gives an evaluation value of the decision. At each time step, the agent outputs an action from the policy network based on the current state and interacts with the environment to obtain rewards and the next state. Then, the state, action, reward, and next state are stored in a buffer using the experience replay mechanism. Next, these experiences are randomly sampled to update the policy network and value network. The value network is optimized by minimizing the mean square error, and the deterministic policy gradient is used to update the policy network. By continuously iterating this process, the agent learns the optimal strategy.

7. The method for dynamic reconstruction of a power distribution system based on deep reinforcement learning according to claim 1, characterized in that: In step S6, performing sequential recovery specifically includes: When the distribution system detects a fault, data collection is carried out. The collected data includes load recovery status, power demand of restored load, line connectivity status, system voltage parameters and system flow parameters. The collected information is then encoded and input into the deep reinforcement learning model. The strategy network of the deep reinforcement learning model will derive the optimal recovery decision for the system at the current moment based on the obtained system status information. The recovery decision will be transmitted to each sensor through communication equipment and executed by the distribution system operator. After that, the system status information will be updated. Through iterative update behavior, the distribution system will execute sequential recovery decisions step by step.

8. A computer device, comprising a memory, wherein the memory stores program instructions, characterized in that: When the program instructions are executed, the dynamic reconstruction method of the distribution system based on deep reinforcement learning as described in any one of claims 1 to 7 is executed.

Images