Post-disaster power distribution network power supply restoration method based on reinforcement learning guided by expert demonstration

Abstract

The application discloses a post-disaster power distribution network power supply recovery method based on expert demonstration guidance reinforcement learning, first, an expert demonstration dataset is constructed, and multiple priority experience replay pools are established, including expert demonstration, normal operation and constraint violation experience replay pools, experiences are batch extracted from the expert demonstration experience replay pool, a reinforcement learning model is pre-trained, after pre-training is completed, an invalid action shielding module is constructed and integrated, invalid operations are shielded, then in a large number of fault scenarios, simulation operation is carried out, normal operation and constraint violation experiences are respectively stored into corresponding replay pools, experiences are extracted in proportion for priority replay, the model is trained, finally, the trained model is used to recursively solve in real-time fault scenarios, and an approximate optimal decision of post-disaster power supply recovery is output. The application significantly improves the accuracy and efficiency of post-disaster power distribution network power supply recovery, and solves the problems of high calculation complexity and poor real-time performance in the prior art.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power system distribution network power supply recovery, in particular to a post-disaster distribution network power supply recovery method based on expert demonstration guidance reinforcement learning. BACKGROUND

[0002] Extreme disasters often cause multiple failures of power systems, which may trigger large-scale power outages and cause huge economic and social losses. Therefore, it is urgent to develop a safe, reliable and fast distribution system restoration (DSR) method. Distribution system restoration refers to rebuilding the network topology by controlling the intelligent switches installed on the line after the failure of the distribution line and the automatic isolation of the fault by feeder automation, and then supplying power to the outage load under the premise of meeting the grid operation constraints.

[0003] The existing methods for solving the DSR problem mainly include mathematical analysis method and heuristic algorithm. The mathematical analysis method has the advantages of providing optimal guarantee and flexible modeling, but has obvious limitations: 1) it depends on accurate power grid model, and it is difficult to provide reliable solutions when the system parameters are inaccurate or incomplete; 2) as the system size increases, the computational complexity increases significantly, and the solution time increases greatly, so it is difficult to be applied online in large-scale systems. Heuristic algorithms such as genetic algorithm and particle swarm algorithm have the advantage of quickly solving to a better solution, but it is difficult to solve the global optimal solution, and it relies on exploration, and the effect is unstable. SUMMARY

[0004] The present application aims to overcome the shortcomings of the prior art and provides a post-disaster distribution network power supply recovery method based on expert demonstration guidance reinforcement learning. By combining expert demonstration data, multiple priority experience replay and invalid action shielding technology, the accuracy and efficiency of post-disaster distribution network power supply recovery are significantly improved, and the problems of high computational complexity and poor real-time performance in the prior art are solved.

[0005] To achieve the above-mentioned purpose, the technical solution provided by the present application is: a post-disaster distribution network power supply recovery method based on expert demonstration guidance reinforcement learning, comprising the following steps:

[0006] S1: obtaining the data recorded by the distribution network data acquisition and monitoring control system during historical failures, and constructing an expert demonstration data set;

[0007] S2: constructing a multiple priority experience replay pool, including an expert demonstration experience replay pool, a normal operation experience replay pool and a constraint violation experience replay pool, and importing the expert demonstration data set into the expert demonstration experience replay pool to initialize the expert demonstration experience replay pool;

[0008] S3: Batch extraction of experience from the expert demonstration experience replay pool, pre-training of the constructed reinforcement learning model;

[0009] S4: After pre-training, construct an invalid action shielding module to shield repeated closed switch actions, actions forming substation loops, actions forming de-energized load islands, and actions forming new energy distributed power supply islands, and integrate the invalid action shielding module into the reinforcement learning model;

[0010] S5: In a large number of fault scenarios, normal operation experience and constraint violation experience are stored in normal operation experience replay pool and constraint violation experience replay pool respectively, and according to the preset proportion, the priority experience replay mechanism is used to extract experience from the expert demonstration experience replay pool, the normal operation experience replay pool and the constraint violation experience replay pool to form a training batch, and the reinforcement learning model integrated with the invalid action shielding module is trained;

[0011] S6: Using the trained reinforcement learning model to recursively solve the power supply restoration problem under real-time fault scenarios of distribution network, outputting the approximate optimal decision of post-disaster power supply restoration of distribution network, i.e. obtaining the approximate optimal switch control decision instruction at each time step.

[0012] Further, in step S1, the constructed expert demonstration data set includes grid observation data and dispatcher operation data, wherein the grid observation data includes distribution network topology, load data, upper and lower limits of distributed power output and node voltage, and the dispatcher operation data refers to the switch operation sequence of the dispatcher.

[0013] Further, in step S2, the expert demonstration experience replay pool stores the experience of expert demonstration, which represents the high reward experience of post-disaster distribution network power supply restoration problem; the normal operation experience replay pool stores the normal operation experience collected when the agent interacts with the environment, which reflects the performance of the agent in the conventional state; the constraint violation experience replay pool stores the experience of the agent jumping out of the iteration after violating the grid constraints, which contains rare but important negative reward information of post-disaster distribution network power supply restoration problem.

[0014] Further, in step S3, the reinforcement learning model is a D3QN neural network, and the parameters of the reinforcement learning model are updated by the following loss function:

[0015] L(θ)=L Q (θ)+λ1L S (θ)+λ2L L2 (θ)

[0016]

[0017]

[0018] wherein: L(θ) is a loss function of the reinforcement learning model, L Q (θ), L S (θ) and L L2 (θ) are Q-learning loss, supervision loss and regularization loss respectively; θ and are model parameters; λ1 and λ2 are weights of the supervision loss and the regularization loss respectively; Q(s, a; θ) is a Q value output by the model with parameters θ when inputting state s and action a; Q(s', a'; θ) is a Q value output by the model with parameters θ when inputting state s' and action a'; γ is a discount factor; R(s, a) is an immediate reward value obtained when action a is taken in state s; I(a D ,a) is an indicator function, a D is an expert demonstration action, which takes the value 1 when a D =a, and takes the value 0 otherwise; Q(s, a D ; θ) is a Q value output by the model with parameters θ when inputting state s and expert action a D .

[0019] Further, at step S4, the invalid action masking module performs the following operations:

[0020] S41: generating an invalid action mask iam = [iam1, iam2, …, iam i , …, iam |ε| ], which is a Boolean vector with a length of |ε|, where |ε| is the number of branches; for the i-th action, if the value at the corresponding position of the invalid action mask satisfies iam i = 1, it indicates that the action is an invalid action; otherwise, if iam i = 0, it indicates that the action is a valid action;

[0021] S42: screening invalid actions according to the following rules:

[0022] a. If the action is selected to close a switch of a branch, the state of the switch is checked, and if the switch is already closed, the action is an invalid action of repeatedly closing the switch;

[0023] b. If the action is selected to close a switch of a branch, the following constraint is checked, and if the constraint is not satisfied, the action is an invalid action of forming a power substation loop, specifically:

[0024]

[0025] wherein: is a set of power substation power supply loops; M lthe number of branches in the power supply loop of the lth substation, the switch state of the mth branch in the power supply loop of the lth substation.

[0026] c. If the switch of a branch is selected to be closed, the power supply states of the nodes at both ends of the branch are judged. If both are load nodes and are in a power-off state, the action is an invalid action for forming a power-off load island;

[0027] d. If the switch of a branch is selected to be closed, and one end of the branch is a power-off load node and the other end is a new energy distributed power node, the action is an invalid action for forming a new energy distributed power island;

[0028] S43: According to the invalid action screened above, the value of the corresponding position of the invalid action mask is modified to 1. If the action a is an invalid action, the value of the action a in the corresponding position iam of the invalid action mask is set to 1; a ;

[0029] S44: Before the agent takes an action, the Q value output by the reinforcement learning model is updated using the following formula:

[0030] Q(s, a; θ)←Q(s, a; θ)-λ·iam a

[0031] In the formula: Q(s, a; θ) is the Q value output by the model with the input state s, action a, and the parameter θ; λ is a fixed positive number.

[0032] Further, in step S5, the preset proportions are 0.3:0.5:0.2.

[0033] Further, in step S6, the trained reinforcement learning model is used to recursively solve the power supply restoration problem under the real-time fault scenario of the distribution network, and an approximate optimal decision of post-disaster power supply restoration of the distribution network is output. The specific steps are as follows:

[0034] S61: Let the time step t=1;

[0035] S62: Update the grid observation data of the current time step, including the distribution network topology, load data, output upper and lower limits of distributed power, and node voltage;

[0036] S63: Use the trained reinforcement learning model to output the optimal switch control decision instruction of the time step t;

[0037] S64: Let t=t+1, if t≤T, T is the time step of the restoration process, return to step S61; if t>T, the loop is terminated, and the approximate optimal switch control decision instruction of each time step is obtained.

[0038] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0039] 1. Improved power supply recovery efficiency: The present application uses a reinforcement learning method based on expert demonstration guidance, through pre-training and invalid action shielding module, compared with traditional mathematical analysis method, can greatly shorten the calculation time of power distribution network power supply recovery, has higher real-time calculation efficiency.

[0040] 2. Enhance the accuracy of power supply recovery decision: The present application uses expert demonstration data for pre-training, and improves the learning effect of the model through multiple priority experience replay pool, compared with heuristic algorithm, can greatly improve the accuracy of power supply recovery decision, obtain the effect close to mathematical analysis method.

[0041] 3. Better deal with incomplete or inaccurate data: Compared with the mathematical analysis method which depends on accurate power grid model, the present application can better deal with the situation of inaccurate or incomplete distribution network parameters through expert demonstration data set and reinforcement learning model, so as to provide more reliable recovery scheme.

[0042] 4. Improve the universality and scalability of the method: Through the construction of multiple priority experience replay pool and invalid action shielding module, the present application can flexibly adapt to distribution network systems of different scales and complexities, and effectively apply in different post-disaster recovery scenarios, showing good universality and scalability.

[0043] 5. Enhance the security of the system: The present application effectively avoids invalid actions such as repeated closing of switches, forming substation loop, forming power-off load island and new energy distributed power island through invalid action shielding module, ensuring the safety of operation in power supply recovery process. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is the framework diagram of the method of the present application.

[0045] Figure 2 is the schematic diagram of multiple priority experience replay pool.

[0046] Figure 3 is the improved IEEE-37 example diagram.

[0047] Figure 4 is the reward curve diagram of the training process.

[0048] Figure 5 is the implementation flowchart of the method of the present application. DETAILED DESCRIPTION

[0049] The present application will be further described in detail below in combination with embodiments and drawings, but the implementation manner of the present application is not limited thereto.

[0050] As Figure 1 and Figure 5 shown, the embodiment discloses a post-disaster power distribution network power supply restoration method based on expert demonstration guided reinforcement learning, comprising the following steps:

[0051] S1: Obtain the data recorded by the power distribution network data acquisition and monitoring control system during the historical failure, and construct an expert demonstration data set.

[0052] In this embodiment, the constructed expert demonstration data set includes collected power grid observation data and dispatcher operation data. Among them, the power grid observation data includes power distribution network topology, load data, output upper and lower limits of distributed power, and node voltage; the dispatcher operation data refers to the switch operation sequence of the dispatcher.

[0053] S2: Construct a multiple priority experience replay pool combined with expert demonstration, including an expert demonstration experience replay pool, a normal operation experience replay pool, and a constraint violation experience replay pool, and import the expert demonstration data set into the expert demonstration experience replay pool to initialize the expert demonstration experience replay pool.

[0054] As Figure 2 shown, in this embodiment, the expert demonstration experience replay pool stores the experience of expert demonstration, which represents the high return experience of the post-disaster power distribution network power supply restoration problem; the normal operation experience replay pool stores the normal operation experience collected when the agent interacts with the environment, which reflects the performance of the agent in the conventional state; the constraint violation experience replay pool stores the experience of the agent jumping out of the iteration after violating the power grid constraints, which contains rare but important negative reward information of the post-disaster power distribution network power supply restoration problem.

[0055] S3: Batch extract experience from the expert demonstration experience replay pool, and pre-train the constructed reinforcement learning model, wherein the reinforcement learning model is a D3QN neural network, and the parameters of the reinforcement learning model are updated through the following loss function:

[0056] L(θ)=L Q (θ)+λ1L S (θ)+λ2L L2 (θ)

[0057]

[0058] In the formula: L(θ) is the loss function of the reinforcement learning model, L Q (θ), L S (θ) and L L2 (θ) are Q-learning loss, supervision loss and regularization loss, respectively; θ and are model parameters; λ1 and λ2 are weights of supervised loss and regularization loss respectively; Q(s, a; θ) is a Q value output by the model with parameters θ when inputting state s and action a; Q(s', a'; θ) is a Q value output by the model with parameters θ when inputting state s' and action a'; γ is a discount factor; R(s, a) is an immediate reward value obtained when action a is taken in state s; I(a D D D D D

[0059] S4: After the pre-training is completed, an invalid action shielding module is constructed, which is used to shield repeated closing switch actions, actions forming a substation loop, actions forming a load island after power failure, and actions forming a new energy distributed power source island, and the invalid action shielding module is integrated into the reinforcement learning model. In this embodiment, the invalid action shielding module specifically performs the following operations:

[0060] S41: An invalid action mask iam = [iam1, iam2, …, iam i , …, iam |ε| ] is generated, which is a Boolean vector with a length of |ε|, where |ε| is the number of branches; for the ith action, if the value of the action at the corresponding position of the invalid action mask satisfies iam i = 1, it indicates that the action is an invalid action; otherwise, if iam i = 0, it indicates that the action is a valid action.

[0061] S42: The invalid action is screened according to the following rules:

[0062] a. If a switch of a branch is selected to be closed, the state of the switch is checked, and if the switch has been closed, the action is an invalid action of repeatedly closing the switch.

[0063] b. If a switch of a branch is selected to be closed, the constraint of the following formula is checked, and if the constraint is not satisfied, the action is an invalid action of forming a substation loop, which is specifically:

[0064]

[0065] In the formula: is a set of substation power supply loops; M l is the number of branches in the lth substation power supply loop, is the state of the switch of the mth branch in the lth substation power supply loop. ​​​​​

[0066] c. If the switch of a branch is selected to be closed, the power supply states of the nodes at both ends of the branch are determined. If both are load nodes and are in a power-off state, the action is an invalid action for forming a power-off load island;

[0067] d. If the switch of a branch is selected to be closed, and one end of the branch is a power-off load node and the other end is a new energy distributed power supply node, the action is an invalid action for forming a new energy distributed power supply island;

[0068] S43: According to the invalid action screened above, the value of the position corresponding to the invalid action mask is modified to 1. If the action a is an invalid action, the value of the action a in the invalid action mask corresponding position iam a is set to 1.

[0069] S44: Before the agent takes an action, the Q value output by the reinforcement learning model is updated using the following formula:

[0070] Q(s, a; θ) ← Q(s, a; θ) - λ·iam a

[0071] In the formula: λ is a very large fixed positive number, and the value is 1×10 8 .

[0072] S5: In a large number of fault scenarios, normal operation experience and constraint violation experience are stored in normal operation experience replay pool and constraint violation experience replay pool respectively. Then, according to the ratio of 0.3:0.5:0.2, the priority experience replay mechanism is used to extract experience from the expert demonstration experience replay pool, the normal operation experience replay pool and the constraint violation experience replay pool to form a training batch, and the reinforcement learning model is trained.

[0073] S6: The trained reinforcement learning model is used to recursively solve the power supply recovery problem under real-time fault scenarios of the distribution network, and the approximate optimal decision of the post-disaster power supply recovery of the distribution network is output. Specifically, the following steps are included:

[0074] S61: Let the time step t = 1;

[0075] S62: Update the current time step of the power grid observation data, including the distribution network topology, load data, distributed power output upper and lower limits and node voltage;

[0076] S63: Use the trained reinforcement learning model to output the optimal switch control decision instruction of time step t;

[0077] S64: Let t = t + 1. If t ≤ T, where T is the time step of the recovery process, return to step S61. If t > T, the loop terminates, and the approximate optimal switching control decision instructions for each time step are obtained.

[0078] This embodiment uses the IEEE-37 node standard example as the illustrative example. To illustrate the specific application scheme, it is modified to connect two photovoltaic and two traditional distributed energy sources. Furthermore, the node numbered 0 is split into four nodes: 37, 38, 39, and 40, based on its four feeders. The improved example is as follows: Figure 3 As shown.

[0079] The improved algorithm used in this embodiment has 44 nodes and 43 branches. Its state includes the distribution network topology, load data, the upper and lower limits of distributed generation output, and node voltage, with a total of 278 dimensions. Its action space has a total of 43 dimensions.

[0080] The hyperparameters during training are configured as follows: pre-training steps are 2 × 10. 3 Step; the experience replay pool size is 1×10 4 The sample size for each training batch is 512; the number of formal training steps is 3 × 10. 4 Steps; training learning rate is 1×10 -3 The activation function is the ReLU function; the optimizer is the Adam optimizer.

[0081] like Figure 4 As shown, after training, the method of this invention successfully converged to a high reward value. After training, 100 fault scenarios were randomly generated, and tests were conducted using heuristic methods, mathematical analytical methods, and the proposed method, respectively. Optimality was calculated using the following formula:

[0082]

[0083] The solution time and optimality of each scenario were recorded, and the average solution time and average optimality of each method were calculated. The results are shown in Table 1 below.

[0084] Table 1 - Average solution time and optimality results of each method

[0085]

[0086] According to the analysis of the above table, the method has significant advantages in average solving time and optimality compared with the heuristic method and the mathematical analysis method. Specifically, the average solving time of the method is 0.19 seconds, which is significantly lower than 0.53 seconds of the heuristic method and 28.78 seconds of the mathematical analysis method; in terms of average optimality, the method reaches 99.38%, close to 100% of the mathematical analysis method, while the average optimality of the heuristic method is only 80.46%. Therefore, the method significantly improves the accuracy and efficiency of power supply restoration of the post-disaster distribution network, and is worth popularizing.

[0087] The above embodiments are preferred embodiments of the present application, but the embodiments of the present application are not limited to the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are included in the protection scope of the present application.

Claims

1. A method for power supply restoration of post-disaster distribution network based on reinforcement learning guided by expert demonstration, characterized in that, The method comprises the following steps: S1: obtaining data recorded by a power distribution network data acquisition and monitoring control system at a historical fault time, and constructing an expert demonstration data set; S2: constructing a multiple priority experience replay pool, including an expert demonstration experience replay pool, a normal operation experience replay pool, and a constraint violation experience replay pool, and importing the expert demonstration data set into the expert demonstration experience replay pool to initialize the expert demonstration experience replay pool; S3: batch extracting experience from the expert demonstration experience replay pool, and pre-training a constructed reinforcement learning model; S4: after the pre-training is completed, constructing an invalid action shielding module for shielding repeated closed switch actions, actions forming a substation loop, actions forming a power-off load island, and actions forming a new energy distributed power island, and integrating the invalid action shielding module into the reinforcement learning model; S5: simulating under a large number of fault scenarios, storing normal operation experience and constraint violation experience into the normal operation experience replay pool and the constraint violation experience replay pool respectively, extracting experience from the expert demonstration experience replay pool, the normal operation experience replay pool, and the constraint violation experience replay pool according to a preset proportion by using a priority experience replay mechanism to form a training batch, and training the reinforcement learning model integrated with the invalid action shielding module; S6: recursively solving a power supply restoration problem under a real-time fault scenario of a power distribution network by using the trained reinforcement learning model, and outputting an approximate optimal decision of post-disaster power supply restoration of the power distribution network, that is, obtaining an approximate optimal switch control decision instruction at each time step.

2. The method of claim 1, wherein the method is based on expert demonstration guided reinforcement learning for post-disaster distribution network restoration. In step S1, the constructed expert demonstration data set comprises grid observation data and dispatcher operation data, wherein the grid observation data comprises power distribution network topology, load data, upper and lower limits of distributed power output, and node voltage, and the dispatcher operation data refers to a switch operation sequence of a dispatcher.

3. The method of claim 1, wherein the method is based on reinforcement learning guided by expert demonstration. In step S2, the expert demonstration experience replay pool stores experience of experts, which represents high-reward experience of the post-disaster power supply restoration problem of the power distribution network; the normal operation experience replay pool stores experience collected when an agent interacts with an environment, which reflects the performance of the agent under a conventional state; and the constraint violation experience replay pool stores experience of the agent jumping out of iteration after performing an action that violates a grid constraint, which contains rare but important negative reward information of the post-disaster power supply restoration problem of the power distribution network.

4. The method of claim 1, wherein the method is based on reinforcement learning guided by expert demonstration. In step S3, the reinforcement learning model is a D3QN neural network, and the parameters of the reinforcement learning model are updated by using the following loss function: L(θ) = L Q (θ) + λ1L S (θ) + λ2L L2 (θ) In the formula, L(θ) is a loss function of the reinforcement learning model, L Q (θ), L S (θ) and L L2 (θ) are respectively a Q-learning loss, a supervision loss and a regularization loss; θ and are model parameters; λ1 and λ2 are respectively weights of the supervision loss and the regularization loss; Q(s, a; θ) is a Q value output by the model with parameters θ when inputting a state s and an action a; Q(s′, a′; θ) is a Q value output by the model with parameters θ when inputting a state s′ and an action a′; γ is a discount factor; R(s, a) is an immediate reward value obtained when an action a is taken in a state s; I(a D ,a) is an indicator function, a D is an expert demonstration action, and when a D =a, the value is 1, and otherwise, the value is 0; Q(s, a D ; θ) is a Q value output by the model with parameters θ when inputting a state s and an expert action a D .

5. The method for power supply restoration of post-disaster distribution network based on reinforcement learning guided by expert demonstration of claim 1, wherein, In step S4, the invalid action shielding module performs the following operations: S41: Generate an invalid action mask iam = [iam1, iam2, ..., iam] i ,…,iam |ε| The mask is a Boolean vector of length |ε|, where |ε| is the number of branches; for the i-th action, if its value at the corresponding position in the invalid action mask satisfies iam i =1, which indicates that the action is invalid; conversely, if iam i =0 indicates that the action is valid; S42: screening invalid actions according to the following rules: a. If a switch of a branch is selected to be closed, the state of the switch is checked, and if the switch has been closed, the action is an invalid action of repeatedly closing the switch; b. If a switch of a branch is selected to be closed, the following constraint is checked, and if the constraint is not satisfied, the action is an invalid action of forming a substation loop, and the constraint is: wherein: M is a set of power supply loops of the substation; M l N is the number of branches in the lth power supply loop of the substation, Slm is the switch state of the mth branch in the lth power supply loop of the substation; c. If the switch of a branch is selected to be closed, the power supply states of the nodes at both ends of the branch are determined. If both are load nodes and are in a power-off state, the action is an invalid action to form a power-off load island; d. If the switch of a branch is selected to be closed, and one end of the branch is a power-off load node and the other end is a new energy distributed power node, the action is an invalid action to form a new energy distributed power island; S43: according to the invalid action screened above, the value of the invalid action mask corresponding position is modified to 1, if the action a is an invalid action, the value of the action a in the invalid action mask corresponding position iam a is set to 1; S44: Before the agent takes an action, the Q value output by the reinforcement learning model is updated using the following formula: Q(s, a; 0) <— Q(s, a; 0) - lama a Wherein: Q(s, a; θ) is the Q value output by the model with input state s, action a and parameter θ; λ is a fixed positive number.

6. The method for power supply restoration of post-disaster distribution network based on reinforcement learning guided by expert demonstration of claim 1, wherein, In step S5, the preset proportions are 0.3:0.5:0.

2.

7. The method of claim 1, wherein the method further comprises: In step S6, the trained reinforcement learning model is used to recursively solve the power supply restoration problem under real-time fault scenarios of the power distribution network, and an approximate optimal decision of post-disaster power supply restoration of the power distribution network is output. The specific steps are as follows: S61: Let the time step t = 1; S62: Update the power grid observation data of the current time step, including the power distribution network topology, load data, output upper and lower limits of distributed power, and node voltage; S63: Use the trained reinforcement learning model to output the optimal switch control decision instruction of time step t; S64: Let t = t + 1, if t ≤ T, T is the time step of the restoration process, return to step S61; if t > T, the loop is terminated, and the approximate optimal switch control decision instruction of each time step is obtained.

Images