Active power distribution network health management method and system based on reinforcement learning

Abstract

The invention provides an active power distribution network health management method and system based on reinforcement learning, and relates to the technical field of intelligent operation and maintenance of power distribution networks. According to the method provided by the invention, a target function and a constraint condition of state maintenance of the power distribution network are established according to structural characteristics of the power distribution network of the distributed power supply, and then a health state set, a maintenance action set and a reward function corresponding to an equipment degradation process in the power distribution network are constructed by using a Markov decision simulation mode; and then utilizing a reinforcement learning algorithm to obtain a maintenance action with the maximum reward value of each health state in the power distribution network, and integrating the maintenance actions to obtain a state maintenance decision of the power distribution network, so that the degradation process of the equipment health state and the influence of the distributed power supply on the maintenance decision are obtained. And the lean performance of the maintenance decision of the active power distribution network is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation and maintenance of distribution networks, and in particular to a method and system for health management of active distribution networks based on reinforcement learning. Background Art

[0002] The distribution network maintenance decision-making method aims to obtain a scientific and efficient distribution network operation and maintenance strategy, which is an effective way to reduce costs and increase efficiency in distribution network operation and maintenance management and promote distribution network asset management. However, with the access of a large number of random and impactful distributed power sources, charging facilities, etc. to the distribution network, the degradation defect rate of distribution equipment has significantly accelerated, and the complexity of the distribution network and the difficulty of operation and maintenance management have increased significantly.

[0003] The existing distribution network maintenance decision-making methods are mainly based on the strategies of "scheduled maintenance" and "emergency repair instead of operation", that is, the operation status of the distribution network is managed by adopting scheduled maintenance for the normally operating distribution network and emergency repair for the faulty distribution network.

[0004] Therefore, the distribution network status maintenance method disclosed in the prior art makes the contradiction between the limited operation and maintenance resources of the distribution network and the heavy maintenance tasks more prominent, fails to fully consider the impact of distributed power sources on maintenance decisions and the degradation process of the health status of distribution equipment, and is difficult to adjust the maintenance plan of the distribution network according to the actual operating status of the distribution network, which greatly reduces the leanness of the operation and maintenance decisions of the active distribution network. Summary of the invention

[0005] In order to solve the problem that the distribution network status maintenance method disclosed in the prior art does not fully consider the influence of distributed power sources on maintenance decisions and the degradation process of the health status of distribution equipment, the present invention proposes an active distribution network health management method and system based on reinforcement learning. In the method provided by the present invention, the objective function and constraint conditions of the distribution network status maintenance are established according to the structural characteristics of the distribution network of distributed power sources, and then the health state set, maintenance action set and reward function corresponding to the equipment degradation process in the distribution network are constructed by Markov decision simulation. Then, the reinforcement learning algorithm is used to obtain the maintenance action with the maximum reward value for each health state in the distribution network, and the state maintenance decision of the distribution network is obtained by integrating the maintenance actions, thereby realizing the acquisition of the degradation process of the health status of the equipment and the influence of distributed power sources on the maintenance decision, and improving the leanness of the maintenance decision for the active distribution network.

[0006] On the one hand, the present invention provides an active distribution network health management method based on reinforcement learning, comprising:

[0007] Obtaining a network structure of a distribution network connected to a distributed power source, and constructing an objective function and constraint conditions for state maintenance of the distribution network based on the network structure;

[0008] Based on the objective function and constraint conditions of the distribution network status maintenance, a health state set, a maintenance action set and a reward function of the distribution network are constructed in a Markov decision process;

[0009] For each health state in the health state set, using a reinforcement learning algorithm method, determine, in the maintenance action set, an optimal action with the maximum reward value of the reward function in the health state;

[0010] The optimal action under each health state is comprehensively considered to determine the state maintenance strategy of the distribution network.

[0011] Optionally, the constructed objective function includes:

[0012] For the lines to be repaired in the distribution network, the maintenance costs of post-fault maintenance and predictive maintenance of the lines are constructed, and the obtained maintenance costs are used as the constructed objective function.

[0013] Optionally, for the line to be repaired in the distribution network, the maintenance cost of post-fault maintenance and predictive maintenance of the line is constructed, and the obtained maintenance cost is used as the objective function, using the following expression:

[0014]

[0015] Where i is the line to be repaired in the distribution network, where i∈{0,1,2,3,…,n}, n is the number of lines to be repaired; t represents the repair period (t,t+1]; R i,t is the maintenance cost incurred by performing maintenance actions on line i in the distribution network during time period t; The labor and material costs involved in performing predictive maintenance on line i in the distribution network during time period t; It is the labor and material costs involved in performing post-fault maintenance on line i in the distribution network within time period t.

[0016] Optionally, the objective function and constraint conditions of the distribution network state maintenance are used to construct a health state set, maintenance action set and reward function of the distribution network in a Markov decision process, including:

[0017] Modeling the equipment degradation process of the distribution network as a random process;

[0018] Acquire the health status of the distribution network according to the random process of the equipment of the distribution network, and construct a health status set of the distribution network by integrating the health status of the distribution network;

[0019] Using the constraint conditions, comprehensively performing maintenance actions on the lines of the distribution network to obtain a maintenance action set of the distribution network;

[0020] The objective function corresponding to the maintenance of the distribution network is inverted, and the inverted objective function is used as the reward function of the distribution network.

[0021] Optionally, for each health state in the health state set, determining, in the maintenance action set, an optimal action with the maximum reward value of the reward function in the health state by using a reinforcement learning algorithm method includes:

[0022] Randomly obtain any health state in the health state set of the distribution network, and use the obtained health state as the initial health state of the distribution network;

[0023] Initialize the agent and Q-value table of the reinforcement learning algorithm, wherein the Q-value table is obtained according to the action-value function;

[0024] Based on the initial health state of the distribution network, using the intelligent agent to perform maintenance actions on the distribution network, obtain a reward and a next health state, and update the Q value table according to the obtained reward;

[0025] According to the Q value table finally obtained, the maintenance action with the largest Q value in the initial healthy state of the distribution network is selected as the optimal action of the distribution network in the healthy state.

[0026] Optionally, based on the initial health state of the distribution network, the agent is used to perform maintenance actions on the distribution network, a reward and a next health state are obtained, and a Q value table is updated according to the obtained reward, which is implemented as the following expression:

[0027] q(S k ,A k )=q(S k ,A k )+α[U k -q(S k ,A k )],

[0028]

[0029] Where, k is the termination condition that is satisfied before the kth maintenance action is performed on the distribution network; q(S k , A k ) is the distribution network in a healthy state S k Perform maintenance action A k The corresponding Q value, α is the learning rate set according to the Q learning method; U k is the biased reward after the distribution network reaches the terminal state, R k+1is the reward corresponding to the k+1th maintenance action performed on the distribution network; γ is the discount factor set for the reward for the maintenance action performed on the distribution network at the current moment; The distribution network is in a healthy state S k+1 The maximum reward obtained by performing the maintenance action below.

[0030] Optionally, the network structure of the distribution network connected to the distributed power source includes: multiple distributed power sources, multiple power supply nodes, multiple segment switches and tie switches; the multiple distributed power sources are sequentially connected to the multiple power supply nodes through distribution network lines, each segment switch is connected between two adjacent power supply nodes, and the tie switch is connected to the feeder between two power supply lines in the distribution network;

[0031] The distributed power source is used to provide electric energy to multiple power supply nodes in the distribution network;

[0032] The section switch is used to isolate the power supply node in the fault area of ​​the distribution network by disconnecting itself;

[0033] The tie switch is used for providing a backup power supply path to the power supply nodes in the non-fault power failure area in the power distribution network by closing itself.

[0034] On the other hand, the present invention provides an active distribution network health management system based on reinforcement learning, comprising:

[0035] An environment acquisition module is used to acquire the network structure of the distribution network connected to the distributed power source, and to construct the objective function and constraint conditions of the distribution network status maintenance based on the network structure;

[0036] A decision process building module, used to build a health state set, a maintenance action set and a reward function of the distribution network in a Markov decision process based on the objective function and constraint conditions of the distribution network state maintenance;

[0037] An optimal action determination module, for determining, for each health state in the health state set, an optimal action with the maximum reward value of the reward function in the health state in the maintenance action set by using a reinforcement learning algorithm;

[0038] The maintenance strategy output module is used to determine the state maintenance strategy of the distribution network by comprehensively analyzing the optimal actions under each health state.

[0039] In another aspect, the present invention further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0040] The memory is used to store one or more programs;

[0041] When the one or more programs are executed by the at least one processor, the active distribution network status maintenance method based on reinforcement learning as described in the above technical solution is implemented.

[0042] On the other hand, the present invention also provides a readable storage medium having an execution program stored thereon, and when the execution program is executed, the active distribution network status maintenance method based on reinforcement learning as described in the above technical solution is implemented.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] In the method provided by the present invention, the objective function and constraint conditions of the distribution network status maintenance are established according to the structural characteristics of the distribution network of the distributed power source, and then the health state set, maintenance action set and reward function corresponding to the equipment degradation process in the distribution network are constructed by Markov decision simulation. Then, the reinforcement learning algorithm is used to obtain the maintenance action with the maximum reward value for each health state in the distribution network, and the state maintenance decision of the distribution network is obtained by integrating the maintenance actions, thereby realizing the degradation process of the equipment health state and the influence of the distributed power source on the maintenance decision, and improving the leanness of the maintenance decision for the active distribution network. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a method for health management of active distribution network based on reinforcement learning of the present invention;

[0046] Figure 2 A network structure diagram of a distribution network connected to a distributed power source in an active distribution network health management method based on reinforcement learning of the present invention;

[0047] Figure 3 A flowchart of obtaining a health state set, a maintenance action set and a reward function in a health management method for an active distribution network based on reinforcement learning of the present invention;

[0048] Figure 4 A flowchart for obtaining optimal actions in a method for health management of an active distribution network based on reinforcement learning of the present invention;

[0049] Figure 5 A structural diagram of an improved IEEE33-node power distribution system in an active power distribution network health management method based on reinforcement learning of the present invention;

[0050] Figure 6 The predicted power curves of active load, photovoltaic power generation and wind power generation in the active distribution network health management method based on reinforcement learning of the present invention;

[0051] Figure 7A framework diagram of active distribution network status maintenance in an active distribution network health management method based on reinforcement learning of the present invention;

[0052] Figure 8 A flowchart of obtaining a Q value table by using a reinforcement learning algorithm in a reinforcement learning-based active distribution network health management method of the present invention;

[0053] Fig. 9 It is a schematic diagram of executing the maintenance strategy on the segmented branch L6 by using scheme 1 and scheme 2 in a method for health management of an active distribution network based on reinforcement learning of the present invention;

[0054] Fig.10 It is a schematic diagram of executing a maintenance strategy on the segmented branch L6 using scheme three in a reinforcement learning-based active distribution network health management method of the present invention;

[0055] Fig.11 It is a schematic diagram of executing a maintenance strategy on the segmented branch L6 using scheme 4 in a reinforcement learning-based active distribution network health management method of the present invention;

[0056] Fig.12 The figure is a schematic diagram of the structure of an electronic device of the present invention. DETAILED DESCRIPTION

[0057] Embodiment 1:

[0058] like Figure 1 As shown, the present invention provides an active distribution network health management method based on reinforcement learning, comprising:

[0059] S101, obtaining a network structure of a distribution network connected to a distributed power source, and constructing an objective function and constraint conditions for state maintenance of the distribution network based on the network structure;

[0060] S102, constructing a health state set, maintenance action set and reward function of the distribution network in a Markov decision process based on the objective function and constraint conditions of the distribution network state maintenance;

[0061] S103, for each health state in the health state set, using a reinforcement learning algorithm method, determining, in the maintenance action set, an optimal action with the maximum reward value of the reward function under the health state;

[0062] S104, determining a state maintenance strategy for the distribution network by comprehensively analyzing the optimal actions under each health state.

[0063] By establishing the objective function and constraints of the distribution network status maintenance according to the structural characteristics of the distribution network of distributed power sources, and then using the Markov decision simulation method to construct the health state set, maintenance action set and reward function corresponding to the equipment degradation process in the distribution network, the reinforcement learning algorithm is then used to obtain the maintenance action with the maximum reward value for each health state in the distribution network, and the state maintenance decision of the distribution network is obtained by integrating the maintenance actions, thereby realizing the degradation process of the equipment health state and the influence of distributed power sources on the maintenance decision, and improving the leanness of the maintenance decision for the active distribution network.

[0064] The current distribution network maintenance decision-making method aims to solve the problems of what to repair, when to repair and how to repair. Scientific maintenance strategy is an effective way to reduce costs and increase efficiency in the operation and maintenance of distribution networks and promote asset management of distribution networks. However, with the access of a large number of random and impactful distributed power sources, charging facilities, etc. to the distribution network, the degradation defect rate of distribution equipment has significantly accelerated, and the complexity of the distribution network and the difficulty of operation and maintenance control have increased significantly. The existing distribution network maintenance decision-making method is mainly based on the "scheduled maintenance" and "rush instead of operation" strategies. There are problems of insufficient maintenance or excessive maintenance. The contradiction between the limited operation and maintenance resources of the distribution network and the heavy maintenance tasks is more prominent. If the threshold of distribution network operation and maintenance control can be moved forward, the extensive and inefficient maintenance strategies such as fault repair can be upgraded to lean and efficient maintenance strategies, thereby optimizing the maintenance process and reducing costs and increasing efficiency.

[0065] Therefore, in order to improve the reliability of power equipment and the benefits of lean management throughout its life cycle, it is urgently necessary to break through the traditional distribution operation and maintenance model that relies on manual inspections and post-accident repairs, and build a new operation and maintenance system that focuses on prevention, moves checkpoints forward, is proactive and efficient, and has system control, so as to promote the transformation of power equipment operation and maintenance from traditional passive post-accident repairs to proactive operation and maintenance control.

[0066] In some embodiments, the network structure of the distribution network provided by the present invention includes: a plurality of distributed power sources, a plurality of power supply nodes, a plurality of segment switches and a tie switch; the plurality of distributed power sources are sequentially connected to the plurality of power supply nodes through a distribution network line, each segment switch is connected between two adjacent power supply nodes, and the tie switch is connected to the feeder between two power supply lines in the distribution network;

[0067] The distributed power source is used to provide electric energy to multiple power supply nodes in the distribution network;

[0068] The section switch is used to isolate the power supply node in the fault area of ​​the distribution network by disconnecting itself;

[0069] The tie switch is used for providing a backup power supply path to the power supply nodes in the non-fault power failure area in the power distribution network by closing itself.

[0070] like Figure 2 As shown, the network structure of the distribution network shown in the figure is a distribution network network structure connected to distributed power sources. The distribution network network structure includes 11 power supply nodes and 12 power supply lines, among which L1, L2, ..., L11 are different power supply nodes, S1, S2, ..., S11 are different section switches, Q1 is a connecting switch, PV and WP are distributed power sources currently connected to the distribution network, PV is photovoltaic power generation, WP is wind power generation, and LD1 and LD2 are loads.

[0071] The interconnecting switch Q1 is connected between the power supply node L10 and the power supply node L5, the section switches S1, S2, ..., S11 are all in the normally closed state, and the interconnecting switch Q1 is in the normally open state. The photovoltaic power generation and wind power generation connected to the current distribution network are used as distributed power sources to establish a model of an active distribution network.

[0072] The sectionalizer is used to isolate the fault or restore the power outage area, and can disconnect itself to cut off the power supply to the fault area in the current area, while the tie switch is used to connect the adjacent feeders to provide a backup power supply path to other non-fault power outage areas. In the process of solving the problem of distribution network status maintenance, the sectionalizer and tie switch are used to achieve different functions in different operating conditions of the distribution network by adjusting their own operating status.

[0073] When predictive maintenance is performed on the distribution network, the scope of power outage and unnecessary load loss can be reduced by load transfer. Exemplarily, to perform predictive maintenance on the power supply node L5 in the distribution network provided in the above embodiment, it is necessary to disconnect the segment switch S5 first. At this time, the non-fault power outage area caused by the disconnection of the segment switch S5 is the downstream area of ​​the power supply node L5. Through power flow analysis, it is determined whether it is possible to transfer the power supply node in the downstream area of ​​the power supply node L5 from S5 to the Q1 contact switch to realize power supply to the non-fault power outage area. If the power flow analysis process shows that the remaining capacity of the line is insufficient or the voltage of the power supply node exceeds the limit, that is, the adjusted line is difficult to meet the electrical constraints, and only the load LD1 in the current line can be transferred, then some power supply nodes in the non-fault power outage area can be powered by cutting off part of the load. Exemplarily, the contact switch Q1 and the segment switch S6 are closed, and the segment switch S7 is disconnected.

[0074] By establishing the objective function and constraints of the distribution network status maintenance according to the structural characteristics of the distribution network of distributed power sources, and then using the Markov decision simulation method to construct the health state set, maintenance action set and reward function corresponding to the equipment degradation process in the distribution network, the reinforcement learning algorithm is then used to obtain the maintenance action with the maximum reward value for each health state in the distribution network, and the state maintenance decision of the distribution network is obtained by integrating the maintenance actions, thereby realizing the degradation process of the equipment health state and the influence of distributed power sources on the maintenance decision, and improving the leanness of the maintenance decision for the active distribution network.

[0075] The goal of condition maintenance of the distribution network is to optimize the maintenance budget and decision-making maintenance method based on the lines to be maintained in the area, and to minimize the maintenance cost of the lines to be maintained while meeting economic and technical constraints.

[0076] Therefore, the present invention constructs the maintenance cost of post-fault maintenance and predictive maintenance for the line to be repaired in the distribution network, and uses the obtained maintenance cost as the objective function to be constructed. The objective function constructed by the present invention adopts the following expression:

[0077]

[0078] Where i represents the line to be repaired in the distribution network, where i∈{0,1,2,3,…,n}, n represents the number of lines to be repaired; t represents the repair period (t,t+1]; R i,t represents the maintenance cost incurred by performing maintenance actions on line i in the distribution network during time period t; represents the labor and material costs involved in performing a predictive maintenance on line i in the distribution network within a time period t; It represents the labor and material costs involved in performing post-fault maintenance on line i in the distribution network within time period t.

[0079] The labor and material costs involved in performing post-fault maintenance on line i in the distribution network during time period t The following expression can be used.

[0080]

[0081] in, represents the labor cost per unit man-hour corresponding to the post-fault maintenance of line i in the distribution network; represents the total man-hours for post-fault maintenance of line i in the distribution network; Represents the material cost of performing post-fault repair on line i in the distribution network.

[0082] The labor and material costs involved in performing predictive maintenance on line i in the distribution network during time period t are The following expressions can be used:

[0083]

[0084] in, represents the load transfer cost corresponding to the predictive maintenance of line i in the distribution network; represents the labor cost per unit man-hour corresponding to the predictive maintenance of line i in the distribution network; represents the total man-hours for performing predictive maintenance on line i in the distribution network; It represents the material cost when performing predictive maintenance on line i in the distribution network.

[0085] The distribution network adopts a closed-loop design and open-loop operation mode. Therefore, when predictive maintenance is carried out on the lines in the distribution network, there will be non-fault power outage areas downstream of the power supply node where the predictive maintenance is performed. In this regard, it is necessary to use other areas to supply power to the loads in the non-fault power outage areas through load transfer, and the cost of load transfer is high. The expression is as follows:

[0086]

[0087] in, represents the power shortage cost of performing predictive maintenance on line i in the distribution network; represents the cost of increased active network loss caused by predictive maintenance of line i in the distribution network; represents the switch operation cost caused by predictive maintenance of line i in the distribution network; w1, w2 and w3 represent the weight parameters of power shortage cost, active network loss increase cost and switch operation cost, respectively.

[0088] In the process of predictive maintenance of the distribution network, the load transfer of the distribution network must first realize the power shortage cost. Minimum. The power shortage cost of load j caused by predictive maintenance of line i in the distribution network is expressed as follows:

[0089]

[0090] Wherein, j represents the number of loads in the distribution network, j = 1, 2, ..., M; M represents the total number of loads in the distribution network; It represents the power shortage of load j in the distribution network; It represents the power outage recovery time caused by predictive maintenance of line i in the distribution network; ζ represents the power outage loss cost per unit load.

[0091] Increase costs through active network losses It can fully evaluate the economic feasibility of load transfer and further reduce network losses during load transfer. Performing predictive maintenance on line i in the distribution network will increase the cost of active network loss. Use the following expression:

[0092]

[0093] in, It represents the increase in active network loss caused by predictive maintenance of line i in the distribution network, which is calculated according to the power flow of the distribution network; ψ represents the cost corresponding to the unit increase in network loss.

[0094] Calculate the increase in active network loss based on the power flow of the distribution network Use the following expression:

[0095]

[0096] Where, l represents the number of lines for load transfer after closing the tie switch in the distribution network, l = 1, 2, ..., N; N represents the total number of lines for load transfer after closing the tie switch in the distribution network; r l It represents the resistance of the line l through which the load is transferred after the tie switch is closed in the distribution network; It indicates the voltage of the first-end power supply node of line l to which the load is transferred after the tie switch is closed in the distribution network; It represents the active power flowing through line l transferred by the load after the tie switch in the distribution network is closed; It represents the reactive power flowing through line l transferred by the load after the tie switch in the distribution network is closed; Indicates the network loss of the line before closing switch 1 in the distribution network.

[0097] Switching operation cost It is calculated based on the frequency of switch operation. By obtaining the switch operation cost, it is possible to evaluate the loss of the switch action to the equipment in the distribution network. The expression is as follows:

[0098]

[0099] in, is the number of switch operations during the predictive maintenance process; d is the cost of a single operation.

[0100] In order to facilitate the constraints on the constructed distribution network status maintenance, the construction of the constraint conditions for the distribution network status maintenance provided by the present invention includes economic constraints and technical constraints:

[0101] First, the economic constraints of distribution network condition maintenance are as follows:

[0102] During the maintenance period t, the maintenance budget allocated for the distribution network status maintenance cannot exceed the maintenance budget upper limit B, which is expressed as follows:

[0103]

[0104] In the technical constraints of the distribution network condition maintenance, the power balance constraints of the power supply nodes of the distribution network need to be met, which can be expressed as follows:

[0105]

[0106] Among them, P e,t represents the active power of the power supply node e during period t; α ef represents the allocation weight of the branch ef between the power supply node e and the power supply node f; C(e) represents the set of all power supply nodes directly connected to the power supply node e; f represents the power supply node directly connected to the power supply node e; G ee represents the conductance of the power supply node e; V e,t represents the voltage of the power supply node e during the period t; V f,t G represents the voltage of the power supply node f during the period t; ef represents the conductance of the branch ef between the power supply node e and the power supply node f; B ef represents the susceptance of the branch ef between the power supply node e and the power supply node f; δ ef,t represents the voltage phase difference of branch ef between power supply node e and power supply node f during period t; Q e,t represents the reactive power of the power supply node e during period t; B ee Represents the susceptance of the power supply node e.

[0107] At the same time, the voltage of the power supply node of the distribution network in the technical constraints of the distribution network status maintenance needs to meet the normal voltage constraints, which are expressed as follows:

[0108]

[0109] in, Indicates that the power supply node e meets the voltage upper limit corresponding to normal use; Indicates that the power supply node e meets the voltage lower limit corresponding to normal use; V e,t represents the voltage of the power supply node e during period t.

[0110] Furthermore, the technical constraints of the distribution network condition maintenance must ensure that the branch transmission power and branch current cannot exceed the upper limit of the line transmission capacity at time t, as shown in the following expression:

[0111]

[0112] Among them, Sef,t represents the power flow through the branch ef between the power supply node e and the power supply node f during the period t; I represents the upper limit of the power flow allowed to pass through the branch ef between the power supply node e and the power supply node f; ef,t represents the current flowing through the branch ef between the power supply node e and the power supply node f during the period t; It represents the upper limit of the current allowed to flow through the branch ef between the power supply node e and the power supply node f.

[0113] In the technical constraints of the distribution network status maintenance constructed by the present invention, the connectivity constraints and radial constraints of the distribution network must also be met, and the expressions are as follows:

[0114]

[0115] Among them, l de represents the virtual load of the power supply node e in the distribution network; f represents the power supply node directly connected to the power supply node e in the distribution network; φ ef represents the weight coefficient between the power supply node e and the power supply node f; l ef,t represents the virtual flow rate flowing through the branch ef between the power supply node e and the power supply node f at time t; N b is the number of branches; N n Indicates the number of power supply nodes; Ns indicates the number of power supplies.

[0116] The power injected by the distributed power source must meet the upper and lower limit constraints of the distribution network connected to the distributed power source. The expression is as follows:

[0117]

[0118] in, Indicates the lower limit of active power injected by distributed generation into the distribution network; P represents the upper limit of active power injected by distributed generation into the distribution network; DG,e,t It represents the active power injected by the distributed generation into the distribution network during the period t; Indicates the lower limit of reactive power injected by distributed generation into the distribution network; Indicates the upper limit of reactive power injected by distributed generation into the distribution network; Q DG,e,t It represents the reactive power injected into the distribution network by the distributed generation during the period t.

[0119] The present invention provides a condition inspection and health management method under an active distribution network. By connecting distributed power sources to the distribution network, it can not only meet the power shortage demand during the load transfer process, but also reduce active network losses, effectively overcome the problems of over-maintenance and under-maintenance, and avoid unnecessary load power outages, thereby improving the leanness of distribution network maintenance decisions.

[0120] Reference Figure 3 In some embodiments, S102 includes:

[0121] S1021, modeling the equipment degradation process of the distribution network as a random process;

[0122] S1022, obtaining the health status of the distribution network according to the random process of the equipment of the distribution network, and constructing a health status set of the distribution network by integrating the health status of the distribution network;

[0123] S1023, using the constraint conditions, comprehensively performing maintenance actions on the lines of the distribution network to obtain a maintenance action set of the distribution network;

[0124] S1024, inverting the objective function corresponding to the maintenance of the distribution network, and using the inverted objective function as the reward function of the distribution network.

[0125] Since the Markov decision process can effectively describe the degradation process of the health state, as well as the predictive maintenance cost and post-fault maintenance cost, it is conducive to improving the leanness of the distribution network condition maintenance decision. Therefore, in the field of distribution network condition maintenance, the Markov decision process is composed of five key elements {S, A, P, R, γ}, which are used to model the equipment degradation process as a random process and describe the evolution of the health state of the distribution network through action state rewards and action state transfers.

[0126] The method provided by the present invention assumes that the distribution network state maintenance decision conforms to the Markov decision process, and the distribution network state maintenance decision includes five basic elements {S, A, P, R, γ}, wherein S is the health state space set, A is the maintenance action set, P is the action state transition probability, R is the reward, and γ is the discount coefficient.

[0127] Exemplarily, the action set A represents the set of actions performed by the circuit to transfer from one state to another state, the state transition probability P represents the probability of transferring from one state to another state after performing a certain action, and the reward R represents the reward for transferring from one state to another state after performing a certain action.

[0128] With respect to the reward for the status maintenance of the distribution network, the present invention defines the maintenance cost after executing a certain maintenance action as the corresponding reward.

[0129] When the line is in state S during the degradation process i If we take action a k , the line will be with probability P i,j From state S i Transfer to state S j At this time, the cost of the corresponding maintenance action is recorded as R k, and each time the maintenance action is completed a k Cost R k The negative number is denoted as R t , at this time, the reward R of the distribution network state maintenance decision t The expression is as follows:

[0130]

[0131] Among them, R t is the reward for distribution network condition maintenance decision, R k The cost of performing maintenance actions on the distribution network during time period t, The labor and material costs involved in performing predictive maintenance on the distribution network during time period t; The labor and material costs involved in performing post-fault maintenance on the distribution network for period t;

[0132] According to the action state transfer function corresponding to the state maintenance of the distribution network obtained by using the Markov decision process in the present invention, the health state of the line to be maintained in the distribution network is discretized.

[0133] For example, assuming that the health state space S consists of N+1 states, when the health state of the line belongs to the interval [X0, X1], (X1, X2], ..., (X N-1 ,X N ],(X N ,X N+1 ], the health status is recorded as S1, S2, S3, ..., S n+1 When no maintenance action is taken, the equipment in the line is in any of the health states during the degradation process, and the health state of the equipment will only deteriorate.

[0134] S1 means it is in the initial healthy state without any deterioration, while S n+1 Indicates that there is a serious defect, that is, a failure state. The remaining states S2, S3, .., S n Represents the health status of different degrees of degradation. The larger the subscript, the higher the degree of degradation of the corresponding equipment.

[0135] Therefore, since the power distribution equipment is easily affected by the complex internal mechanism and the changing external environment during operation, the health status of the equipment continues to deteriorate, and the change in the failure rate caused by it can be Function modeling, line i in state S k The failure rate under is expressed as follows:

[0136]

[0137] Among them, λi,t Indicates that line i is in healthy state S k Failure rate under k Indicates the health status of line i; Indicates health status S k The degradation status of the corresponding equipment; Indicates health status S k-1 The degradation state of the corresponding device; rand represents a uniformly distributed random number in the range [0,1]; φ represents the health state degradation factor.

[0138] In order to show the impact of maintenance actions on the health status of the line in the distribution network status maintenance, the health status of the equipment will change differently according to different maintenance methods. The health status will deteriorate over time, but through preventive maintenance actions, the health status of the equipment can be effectively restored. The maintenance action set constructed by the present invention includes two different maintenance methods, expressed as follows:

[0139]

[0140] Among them, a i represents the action performed on maintenance line i, Indicates that no maintenance action is taken. Indicates a predictive maintenance action.

[0141] Example, predictive maintenance action Indicates that the health status of the equipment is restored to 95% of the initial status without taking any maintenance action. It means that the health status will continue to deteriorate. If the equipment does not fail, the equipment will continue to operate. If the equipment fails at this time, post-failure maintenance is required to restore the health status of the corresponding equipment to its initial state. Due to the equipment failure, a higher post-failure maintenance cost is required.

[0142] Regarding the discount factor γ in the distribution network condition maintenance process, the discount factor is used to evaluate the long-term or short-term effects of maintenance actions on the health status of equipment.

[0143] For example, when γ is larger, it indicates that the maintenance action has a greater impact on the long-term reward of the equipment health status. Similarly, when γ is smaller, it indicates that the maintenance action has a greater impact on the short-term reward of the equipment health status.

[0144] The change in failure probability caused by the deterioration of the equipment health status over time during the condition maintenance of the distribution network can be modeled using the Weibull function, normal distribution function or gamma function.

[0145] Since the failure rate distribution function of the equipment presents a bathtub curve, the Weibull function can simulate different stages of the bathtub curve. Therefore, the present invention simulates the bathtub curve by adopting a piecewise construction method through a model based on the Weibull function.

[0146] The life cycle of equipment includes three stages: early failure, accidental failure and wear and tear failure. In the early failure stage, equipment failures are mainly caused by problems in the design, manufacturing and assembly process, and test methods can be used to eliminate the failures. In the accidental failure stage, the equipment is in a good use stage, the initial failures have been solved, the various components are well coordinated, and the failure rate is low. In the wear and tear failure stage, the failure rate increases rapidly over time due to factors such as equipment wear and fatigue, and the equipment performance decreases.

[0147] The distribution function f(t) of the model failure rate established using the Weibull function is expressed as follows:

[0148]

[0149] Correspondingly, the probability density function λ of the failure rate of the model established by the Weibull function t The expression is as follows:

[0150]

[0151] Among them, θ and η are both positive numbers, representing the scale parameter and shape parameter of the model established by the Weibull function, respectively. By adjusting the values ​​of the shape parameter θ and the scale parameter η of the model established by the Weibull function, the rise, constancy and decline of the curve can be described.

[0152] When the shape parameter η is less than 1, the failure rate of the equipment decreases; when η is equal to 1, the failure rate of the equipment remains constant; when η is greater than 1, the failure rate of the equipment increases.

[0153] Furthermore, the equipment is easily affected by the complex internal mechanism and the changing external environment during operation, and its health status continues to deteriorate. The degradation process of the state over time in the distribution network condition maintenance model and the change of the failure probability are expressed as follows:

[0154]

[0155] Among them, λ i,t represents the failure rate of line i in period t; θ represents the scale parameter of the model established by the Weibull function; η represents the shape parameter of the model established by the Weibull function; Indicates the degradation status of the equipment during period t.

[0156] For the distribution network connected to the distributed power source constructed by the above technical solution, the present invention uses reinforcement learning algorithms such as Q-learning algorithm (Q learning algorithm), SARSA algorithm (time difference online control algorithm, i.e. State-Action-Reward-State-Action), DQN algorithm (reinforcement Q learning algorithm, i.e. Deep Q-Learning), DDPG algorithm (deep deterministic policy gradient algorithm, i.e. Deep Deterministic Policy Gradient) to solve. Accordingly, the present invention adopts Q learning algorithm for solution.

[0157] Since the Q-learning algorithm achieves maximum reward through the continuous interaction of the constructed intelligent agent with the environment corresponding to the health state of the distribution network structure, the intelligent agent is used to observe the environment and perform maintenance actions, and obtain rewards and enter the next state after performing the maintenance actions, and at the same time learns and updates the Q-value table based on the rewards.

[0158] The key elements of the Q learning algorithm provided by the present invention in the solution process are rewards and actions, and the optimization method of distribution network status maintenance decision-making aims to minimize the comprehensive maintenance cost. The maintenance strategy represents the formation of the optimal maintenance plan based on different health states and maintenance actions of the distribution network.

[0159] Reference Figure 4 In some embodiments, S103 includes:

[0160] S1031, randomly obtaining any health state in the health state set of the distribution network, and using the obtained health state as the initial health state of the distribution network;

[0161] S1032, initializing an intelligent agent and a Q value table of a reinforcement learning algorithm, wherein the Q value table is obtained according to an action value function;

[0162] S1033, based on the initial health state of the distribution network, using the intelligent agent to perform maintenance actions on the distribution network, obtain a reward and a next health state, and update the Q value table according to the obtained reward;

[0163] S1034, according to the Q value table finally obtained, selecting the maintenance action with the largest Q value in the initial healthy state of the distribution network as the optimal action of the distribution network in the healthy state.

[0164] Since the distribution network condition maintenance model contains five basic elements {S, A, P, R, γ}, the intelligent agent constructed by the Q reinforcement learning algorithm observes the health status S of the distribution network in the environment k , and obtain the observed value O t , the health status of the distribution network Sk is obtained based on the health status set, expressed as: S k ∈S, where S represents the health state set; the observation value O t ∈O, where O represents the observation space.

[0165] Then, the agent decides to perform the maintenance action a according to the observation value O, where A represents the maintenance action set. The environment gives the agent a reward R according to the maintenance action. k+1 , R k+1 ∈R, where R represents the reward space and enters the health state S at the next moment k+1 , the above process can be expressed as the following expression:

[0166] S O ,O o ,A o ,R1,S1,O1,A1,R2,…,S r =S 终止 ,

[0167] Among them, S O Indicates the initial health state of the distribution network; O o Represents the agent's observed health state S O The observed value obtained; A o It means that the agent is based on the observation value O o Determine the maintenance action to be performed; R1 indicates that the environment is based on maintenance action A o The reward given to the agent; S1 indicates that the distribution network performs maintenance action A o The health state after the agent observes the health state S1; O1 represents the observation value obtained by the agent when observing the health state S1; A1 represents the maintenance action that the agent decides to perform based on the observation value O1; R2 represents the reward given to the agent by the environment based on the maintenance action A1; S r Corresponding to the final health status of the distribution network; S 终止 The corresponding termination state of the distribution network is the state where the equipment in the distribution network completely fails.

[0168] Similarly, assuming that the agent constructed by the Q-learning algorithm can fully observe the health status of the distribution network in the environment and is considered to be fully observable, the following expression can be obtained:

[0169] S O ,A o ,R1,S1,A1,R2,…,S r =S 终止 ,

[0170] Among them, S O Indicates the initial health state of the distribution network; A o It means that the intelligent agent is based on the health status of the distribution network So The maintenance action determined to be executed; R1 represents the environment according to the maintenance action A o The reward given to the agent; S1 represents the power distribution network executing the maintenance action A o The subsequent health state; A1 represents the maintenance action determined by the agent according to the health state S1 of the power distribution network; R2 represents the reward given to the agent by the environment according to the maintenance action A1; S r Corresponding to the final health state obtained by the power distribution network; S 终止 Corresponding to the termination state of the power distribution network.

[0171] Taking the reward given by the environment to the agent according to the maintenance action and entering the health state at the next moment as a round. In the round task of maintaining the health state of the power distribution network, if a certain round reaches the termination state of the power distribution network at the Tth step, then the total future reward after the kth step (k < T) is defined as the reward G k The reward G k The expression is as follows:

[0172] G k = R k+1 + R k+2 + R k+3 + … + R r-1 + R r ,

[0173] Among them, G k Represents the total future reward given to the agent by the environment after the maintenance action at the kth step; R k+1 Represents the reward given to the agent by the environment according to the maintenance action at the (k + 1)th step; R k+2 Represents the reward given to the agent by the environment according to the maintenance action at the (k + 2)th step; R k+3 Represents the reward given to the agent by the environment according to the maintenance action at the (k + 3)th step; R r-1 Represents the reward given to the agent by the environment according to the maintenance action at the (r - 1)th step; R r Represents the reward given to the agent by the environment according to the maintenance action at the rth step.

[0174] In order to reduce the uncertainty of future rewards, the present invention makes the model more focused on the rewards at the current moment by introducing a discount factor γ, thereby reducing the dependence on future rewards and improving the stability of learning; the reward G k The expression is as follows:

[0175] G k = R k+1 + γR k+2 + γ 2 R k+3 + … + γ r-2 R r-1 + γ r-1R r

[0176] Where γ represents the discount factor and r represents the number of maintenance actions performed.

[0177] Based on the maintenance action, the state maintenance strategy of the distribution network is obtained. For the state maintenance strategy π of the distribution network, the present invention will reward the definition of the state value function v π (s) and the action-value function q π (s,a).

[0178] State value function v π (s) represents the expected reward corresponding to the state maintenance strategy π in the health state s, and the expression is as follows:

[0179] v π (s) = E π [G k |S k =s],

[0180] Among them, v π (s) represents the reward of health state s; E π Indicates the reward G obtained under the health state s k The mathematical expectation of .

[0181] The action value function q π (s,a) represents the expected reward of taking strategy π after performing maintenance action a in health state s, and is expressed as follows:

[0182] q π (s,a)=E[G k |S k =s,A k =a],

[0183] Among them, q π (s) represents the reward for performing maintenance action a in health state s; E represents the reward G for performing maintenance action a in health state s k The mathematical expectation of .

[0184] Furthermore, according to the definition of action value under the distribution network state strategy π, the following expression is obtained:

[0185] q π (s,a)=E π [R k+1 +γG k+1 |S k =s,A k =a],

[0186] Among them, R k+1G represents the reward given by the environment to the agent after the k+1th maintenance action; k+1 It represents the reward given by the environment to the agent after the k+1th maintenance action.

[0187] Based on the above situation, the action value function is updated by the incremental method, and the action value function is converged. At this time, the update process of the action value function can be expressed as:

[0188] q(S k ,A k )←q(S k ,A k )+α[G k -q(S k ,A k )],

[0189] Among them, G k Stands for unbiased reward.

[0190] Since the calculation process of unbiased rewards is complicated, this application uses biased rewards U k Used to obtain the value G of the unbiased reward k , the updating process of the action value function can be expressed as:

[0191] q(S k ,A k )=q(S k ,A k )+α[U k -q(S k ,A k )],

[0192]

[0193] Wherein, k is the termination condition satisfied before the kth maintenance action is performed on the distribution network; q(S k , A k ) is the distribution network in a healthy state S k Perform maintenance action A k The corresponding Q value, α is the learning rate set according to the Q learning method; U k is the biased reward after the distribution network reaches the terminal state, R k+1 is the reward corresponding to the k+1th maintenance action performed on the distribution network; γ is the discount factor set for the reward for the maintenance action performed on the distribution network at the current moment; The distribution network is in a healthy state S k+1 The maximum reward obtained by performing the maintenance action below.

[0194] During the training process, the agent built according to the Q-learning algorithm interacts with the environment to learn and update the state-value function.

[0195] In order to explore the maintenance actions that may bring more rewards, the present invention also adopts the ε-greedy strategy, which is expressed as follows:

[0196]

[0197] Among them, π(a|s) represents the reward corresponding to the maintenance action after the greedy strategy correction; ε represents the probability of the agent choosing a non-greedy action; |A| represents the amount of data in the set of selectable maintenance actions; a represents the currently executed maintenance action; argmaxq π (s,a′) represents the value of the corresponding maintenance action when the reward is maximum.

[0198] Finally, the convergence condition of the Q-learning algorithm is δ Q Satisfy the continuous less than δ s The number of times reaches N δ , and δ Q The calculation formula is expressed as follows:

[0199]

[0200] Where m represents the number of Q values ​​in the Q value table; Q i,k (s,a) represents the health status S k The corresponding i-th Q value; Q i,k-1 (s,a) represents the health status S k-1 The corresponding i-th Q value.

[0201] In some embodiments, in view of the idea of ​​the above technical solution and for the effectiveness of the distribution network condition maintenance optimization method provided by the present invention, an improved IEEE33-node distribution system is used as a distribution network condition maintenance environment, and a case analysis is performed.

[0202] The network structure of the improved IEEE33-node power distribution system is as follows: Figure 5 As shown in the figure, the reference capacity of the improved IEEE33 node distribution system is 10MVA and the reference voltage is 12.66kV. At the same time, L1, L2, ..., L32 are all segmented branches, represented by solid lines, and each segmented branch is connected to a segmented switch; L33, L34, ..., L37 are interconnected branches, represented by dotted lines, and each interconnected branch is connected to a interconnected switch. In addition, photovoltaic power generation is connected at node 13, and wind power generation is connected at node 30 as distributed power sources connected to the distribution network.

[0203] In terms of maintenance decision-making, in order to obtain the maintenance decisions made at different times during the equipment degradation process in the distribution network, the maintenance decision of the segmented branch L6 in the improved IEEE33-node distribution system is analyzed as an example:

[0204] Assume that the health status of segment branch L6 deteriorates in time period t, and introduce the following Figure 6 The predicted power curves of active load, photovoltaic power generation and wind power generation are shown in Figure 2, where the set of t values ​​is t∈{t1,t2,t3,…t 12}, the maintenance duration is in days, and the maintenance decision needs to be completed within a planning period of 2t.

[0205] In the process of setting the Q learning algorithm, the number of epochs of the Q learning algorithm parameters is set to 10. 5 , the learning rate α is 0.1, the greedy strategy coefficient ε is 0.5, and the discount coefficient γ is set to 0.99. The learning rate α and the greedy strategy coefficient ε are set to gradually decrease with the iterative update, so as to satisfy the agent to explore more areas in the early stage of training to avoid falling into the local optimal solution. At the same time, it can also speed up the convergence speed of Q-value table training in the later stage of training and reduce the training time.

[0206] In some embodiments, for the distribution network state maintenance and health management optimization decision model constructed, the present invention uses the reward function, health state set and maintenance action set of the Markov decision process, and obtains the solution method of the distribution network state maintenance target through the Q learning algorithm. By using the Q learning algorithm to calculate the Q value of different actions under the same state, the action with the maximum reward is selected as the optimal action under the state, and then the optimal actions under all states are combined to form a state maintenance strategy, including:

[0207] Initialize the distribution network condition maintenance environment, use the Q learning algorithm to build an intelligent agent, and set the parameters of the Q learning algorithm, including the maximum number of training cycles, step size, discount factor, and Q value table;

[0208] Execute each round of the distribution network state maintenance process according to the set step size, and in each round, use the intelligent agent to perform maintenance actions on the distribution network, obtain rewards and the next health state, and update the Q value table according to the obtained rewards;

[0209] According to the Q value table finally obtained, the maintenance action with the largest Q value in the initial healthy state of the distribution network is selected as the optimal action of the distribution network in the healthy state.

[0210] like Figure 7 and Figure 8As shown, in each round, the intelligent agent is used to perform maintenance actions on the distribution network, obtain rewards and the next health state, and update the Q value table according to the obtained rewards, including:

[0211] Get the random initial health state S of the line to be repaired k ;

[0212] According to the ε-greedy strategy (0<ε<1), the maintenance action a (corresponding to Figure 7 Medium S t ), and obtain the reward R (corresponding to Figure 7 Medium S t+1 ), and transfer to the next state S k+1 , learn and update the Q value table;

[0213] Repeat the maintenance action on the next health state of the distribution network until the round ends;

[0214] Among them, the comprehensive reward corresponding to the maintenance strategy in the entire round is obtained, and the relative change of the Q value between the previous and next two rounds is calculated. Q ; If δ Q Satisfy the continuous less than δ s The number of times reaches N δ The convergence condition is met, the iteration is stopped, and the current Q index is used as the final Q value table.

[0215] According to the final Q value table, the maintenance action a with the largest Q value is selected based on the health status S of the line, thereby forming the optimal maintenance strategy.

[0216] like Fig. 9 As shown, the present invention performs condition maintenance on the segmented branch L6 based on two maintenance schemes: Scheme 1 and Scheme 2: Scheme 1 is a distribution network maintenance strategy that considers distributed power sources; Scheme 2 is a distribution network maintenance strategy that does not consider distributed power sources. The maintenance cost of performing maintenance on the segmented branch L6 according to Scheme 1 and Scheme 2 is shown in Table 1.

[0217] Table 1 Maintenance costs of Scheme 1 and Scheme 2 for maintenance of segmented branch L6

[0218]

[0219] Among them, for the maintenance decision optimization results of branch L6 when distributed power is connected, the monthly maintenance actions include predictive maintenance actions and no maintenance action is taken During the degradation period from t1 to t4, since the initial health status of branch L6 is good and the degradation degree is low, no maintenance action is taken on the line during this period, that is, In the degradation period t5, since the health status of branch L6 continues to deteriorate and the risk of failure is high, it is necessary to perform predictive maintenance on branch L6. During the maintenance period, the non-fault power-off load node 6-17 caused by the L6 predictive maintenance is transferred to the adjacent feeder through load transfer, and the power supply of other load points is not affected. After the maintenance is completed, the non-fault power-off load node will be transferred back to the original feeder to reduce the power loss. 13 and t 14 , although the health status of branch L6 has been seriously deteriorated and the failure risk has increased, the power demand of the load node is high at this time, and the load transfer cost and corresponding maintenance cost are high at this time. Therefore, it is considered to postpone the predictive maintenance to time period t 15 During the degradation period t 23 , since the health status of branch L6 is constantly deteriorating and the power demand of the load is low at this time, the load transfer cost caused by maintenance is low, and the corresponding maintenance cost is also low. Therefore, it is considered to make the predictive maintenance decision in time period t 23 conduct.

[0220] In addition, by comparing the optimization results of the condition maintenance decision of Scheme 1 and Scheme 2, the maintenance time of Scheme 1 is delayed compared with Scheme 2, and the corresponding distribution network predictive maintenance cost is reduced by 30.4%, the post-fault maintenance cost is reduced by 35.5%, and the comprehensive maintenance cost is reduced by 33.2%. The results show that the access of distributed generation to the distribution network not only effectively reduces the predictive maintenance cost, but also improves the anti-fault risk ability of the distribution network. The reason is that the access of distributed generation to the distribution network can reduce the load transfer cost, meet the power shortage demand during the load transfer process, and reduce the active network loss. Therefore, the proposed method can effectively reduce the comprehensive cost of the distribution network condition maintenance and improve the leanness of the distribution network condition maintenance decision.

[0221] In order to further verify the leanness of the distribution network condition maintenance and health management optimization decision model and method proposed in the present invention, the present invention also sets schemes 3 and 4 for branch L6: Fig.10 As shown, the condition maintenance strategy of scheme three is carried out on the segmented branch L6. Scheme three adopts fixed threshold maintenance, in which the maintenance thresholds are set to S40 and S50 respectively; Fig.11 As shown, the condition maintenance strategy of Scheme 4 is carried out on the segmented branch L6; Scheme 4 adopts fixed-cycle maintenance, in which the maintenance cycles are set to six months and eight months respectively. The maintenance cost of performing maintenance on the segmented branch L6 according to Schemes 3 and 4 is shown in Table 2.

[0222] Table 2 Maintenance costs of Scheme 3 and Scheme 4 for maintenance of segmented branch L6

[0223]

[0224] For branch L6, the fixed threshold maintenance strategy of scheme 3 is adopted. Compared with scheme 1, when the maintenance threshold is set to a lower S 40 When the maintenance threshold is set to a higher S, there is an over-maintenance problem, the maintenance frequency increases, the average predictive maintenance cost increases by 26.3%, the average post-failure maintenance cost increases by 9.3%, and the comprehensive maintenance cost increases by 19,449.8 yuan. 50 When the maintenance threshold is set, there is a problem of insufficient maintenance, the average predictive maintenance cost increases by 17.3%, the average post-failure maintenance cost increases by 17.7%, and the comprehensive maintenance cost increases by 6022 yuan. The results show that the fixed threshold maintenance strategy of Scheme 3 has many shortcomings. First, the fixed threshold maintenance is judged only based on the preset threshold. When the maintenance threshold is set low, the equipment may not actually need maintenance, resulting in unnecessary maintenance costs and time waste. In addition, when the maintenance threshold is set high, this may cause the equipment to fail to take timely action when maintenance is required, thereby increasing the risk of equipment failure and damage. Finally, when the equipment state deterioration reaches the set maintenance threshold, but because the maintenance cost is high at this time, if the predictive maintenance action is performed, although the post-failure maintenance cost is effectively reduced, the comprehensive maintenance cost will still be high.

[0225] For branch L6, the fixed-cycle maintenance strategy of Scheme 4 is adopted. When the maintenance cycle is set to half a year, compared with Scheme 1, there is a problem of over-maintenance, the maintenance frequency increases, the average predictive maintenance cost increases by 31.3%, the average post-fault maintenance cost increases by 12.1%, and the comprehensive maintenance cost increases by 21,213.4 yuan. When the maintenance cycle is set to August, compared with Scheme 1, there is a problem of insufficient maintenance, the average predictive maintenance cost increases by 17.5%, the average post-fault maintenance cost increases by 23.3%, and the comprehensive maintenance cost increases by 7,069.9 yuan. The results show that the fixed-cycle maintenance strategy may cause the equipment to take predictive maintenance in a healthy state, resulting in unnecessary maintenance and waste of resources. In addition, it may also cause the equipment to not perform predictive maintenance when it is severely deteriorated, resulting in an increase in the post-fault maintenance cost, thereby significantly increasing the comprehensive maintenance cost.

[0226] According to the above verification results, the present invention constructs a Markov decision process for the state maintenance of the distribution network. By training and learning to update the Q value table to formulate the optimal maintenance strategy, it not only takes into account the current health status and degradation process of the equipment, but also takes into account the comprehensive maintenance cost. By comprehensively weighing and optimizing the post-fault maintenance cost and the predictive maintenance cost, a corresponding maintenance strategy is formed, which effectively overcomes the problems of over-maintenance and under-maintenance, and can effectively reduce the comprehensive maintenance cost of the distribution network. In addition, by considering the access of distributed power sources, the load transfer cost is effectively reduced, the power supply reliability of the distribution network is improved, and the leanness of the distribution network state maintenance decision-making and the asset management level are further improved.

[0227] Embodiment 2:

[0228] The present invention based on the same inventive concept also provides an active distribution network health management system based on reinforcement learning, comprising:

[0229] An environment acquisition module is used to acquire the network structure of the distribution network connected to the distributed power source, and to construct the objective function and constraint conditions of the distribution network status maintenance based on the network structure;

[0230] A decision process building module, used to build a health state set, a maintenance action set and a reward function of the distribution network in a Markov decision process based on the objective function and constraint conditions of the distribution network state maintenance;

[0231] An optimal action determination module, for determining, for each health state in the health state set, an optimal action with the maximum reward value of the reward function in the health state in the maintenance action set by using a reinforcement learning algorithm;

[0232] The maintenance strategy output module is used to determine the state maintenance strategy of the distribution network by comprehensively analyzing the optimal actions under each health state.

[0233] In some embodiments, the network structure of the distribution network connected to the distributed power source obtained by the environment acquisition module is:

[0234] A plurality of distributed power sources, a plurality of power supply nodes, a plurality of segment switches and a tie switch; the plurality of distributed power sources are sequentially connected to the plurality of power supply nodes through a distribution network line, each segment switch is connected between two adjacent power supply nodes, and the tie switch is connected to a feeder between two power supply lines in the distribution network;

[0235] The distributed power source is used to provide electric energy to multiple power supply nodes in the distribution network;

[0236] The section switch is used to isolate the power supply node in the fault area of ​​the distribution network by disconnecting itself;

[0237] The tie switch is used for providing a backup power supply path to the power supply nodes in the non-fault power failure area in the power distribution network by closing itself.

[0238] In some embodiments, the environment acquisition module includes:

[0239] The objective function construction unit is used to construct the maintenance cost of post-fault maintenance and predictive maintenance of the line to be repaired in the distribution network, and use the obtained maintenance cost as the objective function.

[0240] In some embodiments, the objective function construction unit constructs the maintenance cost of post-fault maintenance and predictive maintenance for the line to be repaired in the distribution network, and uses the obtained maintenance cost as the objective function, using the following expression:

[0241]

[0242] Where i is the line to be repaired in the distribution network, where i∈{0,1,2,3,…,n}, n is the number of lines to be repaired; t represents the repair period (t,t+1]; R i,t is the maintenance cost incurred by performing maintenance actions on line i in the distribution network during time period t; The labor and material costs involved in performing post-fault maintenance on line i in the distribution network within time period t; It is the labor and material costs involved in performing post-fault maintenance on line i in the distribution network within time period t.

[0243] In some embodiments, the decision process building block includes:

[0244] An equipment degradation unit, used for modeling the equipment degradation process of the distribution network as a random process;

[0245] A health status set construction unit, used to obtain the health status of the distribution network according to the random process of the equipment of the distribution network, and to construct a health status set of the distribution network by integrating the health status of the distribution network;

[0246] A maintenance action set acquisition unit, used to use the constraint conditions to comprehensively perform maintenance actions on the lines of the distribution network to obtain a maintenance action set of the distribution network;

[0247] The maintenance action set acquisition unit is used to invert the objective function corresponding to the maintenance performed on the distribution network, and use the inverted objective function as the reward function of the distribution network.

[0248] In some embodiments, the optimal action determination module includes:

[0249] A health state acquisition unit, used for randomly acquiring any health state in the health state set of the active power distribution network, and using the acquired health state as the initial health state of the active power distribution network;

[0250] A parameter initialization unit, used to initialize the agent and Q value table of the reinforcement learning algorithm, wherein the Q value table is obtained according to the action value function;

[0251] A Q value table updating unit, configured to use the intelligent agent to perform maintenance actions on the active distribution network based on the initial health state of the active distribution network, obtain a reward and a next health state, and update the Q value table according to the obtained reward;

[0252] The optimal action determination unit is used to select the maintenance action with the largest Q value in the initial healthy state of the active distribution network according to the Q value table finally obtained as the optimal action of the active distribution network in the healthy state.

[0253] In some embodiments, the Q value table updating unit performs maintenance actions on the active distribution network based on the initial health state of the active distribution network using the intelligent agent, obtains a reward and a next health state, and updates the Q value table according to the obtained reward, which is specifically implemented as the following expression:

[0254] q(S k ,A k )=q(S k ,A k )+α[U k -q(S k ,A k )],

[0255]

[0256] Where, k is the termination condition that is satisfied before the kth maintenance action is performed on the active distribution network; q(S k , A k ) is the active distribution network in a healthy state S k Perform maintenance action A k The corresponding Q value, α is the learning rate set according to the Q learning method; U k is the biased reward after the active distribution network reaches the terminal state, R k+1 is the reward corresponding to the active distribution network performing the k+1th maintenance action; γ is the discount factor set for the reward for performing maintenance actions on the active distribution network at the current moment; For the active distribution network in a healthy state S k+1 The maximum reward obtained by performing the maintenance action below.

[0257] Embodiment 3:

[0258] like Fig.12 As shown, the present invention also provides an electronic device, which may be a computer device, a single-chip device, an intelligent mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, the processor, and the transceiver component are connected via a bus; the memory may be used to store an execution program, and an exemplary execution program may include instructions; the processor is used to execute the instructions stored in the memory. The memory may also be used to store data, which may be called and / or modified when the instructions are executed.

[0259] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits.

[0260] (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of an active distribution network health management method based on reinforcement learning in the above-mentioned embodiment.

[0261] Embodiment 4:

[0262] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in an electronic device for storing programs and data. It can be understood that the storage medium here can include both built-in storage media in electronic devices and, of course, extended storage media supported by electronic devices. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a method for health management of an active distribution network based on reinforcement learning in the above embodiment.

[0263] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0264] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0265] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0266] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0267] The above are merely embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are included in the scope of the claims of the present invention to be approved.

Claims

1. A method for health management of active distribution network based on reinforcement learning, characterized in that: include: Obtaining a network structure of a distribution network connected to a distributed power source, and constructing an objective function and constraint conditions for state maintenance of the distribution network based on the network structure; Based on the objective function and constraint conditions of the distribution network status maintenance, a health state set, a maintenance action set and a reward function of the distribution network are constructed in a Markov decision process; For each health state in the health state set, using a reinforcement learning algorithm method, determine, in the maintenance action set, an optimal action with the maximum reward value of the reward function in the health state; The optimal action under each health state is comprehensively considered to determine the state maintenance strategy of the distribution network.

2. The method according to claim 1, characterized in that The objective function constructed includes: For the lines to be repaired in the distribution network, the maintenance costs of post-fault maintenance and predictive maintenance of the lines are constructed, and the obtained maintenance costs are used as the constructed objective function.

3. The method according to claim 2, characterized in that For the line to be repaired in the distribution network, the maintenance cost of the line for post-fault maintenance and predictive maintenance is constructed, and the obtained maintenance cost is used as the objective function, using the following expression: Where i is the line to be repaired in the distribution network, where i∈{0,1,2,3,…,n}, and n is the number of lines to be repaired; t represents the maintenance period (t, t+1]; R i,t is the maintenance cost incurred by performing maintenance actions on line i in the distribution network during time period t; The labor and material costs involved in performing predictive maintenance on line i in the distribution network during time period t; It is the labor and material costs involved in performing post-fault maintenance on line i in the distribution network within time period t.

4. The method according to claim 1, characterized in that The objective function and constraint conditions of the distribution network state maintenance are used to construct the health state set, maintenance action set and reward function of the distribution network in the Markov decision process, including: Modeling the equipment degradation process of the distribution network as a random process; Acquire the health status of the distribution network according to the random process of the equipment of the distribution network, and construct a health status set of the distribution network by integrating the health status of the distribution network; Using the constraint conditions, comprehensively performing maintenance actions on the lines of the distribution network to obtain a maintenance action set of the distribution network; The objective function corresponding to the maintenance of the distribution network is inverted, and the inverted objective function is used as the reward function of the distribution network.

5. The method according to claim 1, characterized in that For each health state of the health state set, using a reinforcement learning algorithm method, determining the optimal action with the maximum reward value of the reward function in the health state in the maintenance action set includes: Randomly obtain any health state in the health state set of the distribution network, and use the obtained health state as the initial health state of the distribution network; Initialize the agent and Q-value table of the reinforcement learning algorithm, wherein the Q-value table is obtained according to the action-value function; Based on the initial health state of the distribution network, using the intelligent agent to perform maintenance actions on the distribution network, obtain a reward and a next health state, and update the Q value table according to the obtained reward; According to the Q value table finally obtained, the maintenance action with the largest Q value in the initial healthy state of the distribution network is selected as the optimal action of the distribution network in the healthy state.

6. The method according to claim 5, characterized in that Based on the initial health state of the distribution network, the agent is used to perform maintenance actions on the distribution network, obtain rewards and the next health state, and update the Q value table according to the obtained rewards, which is implemented as the following expression: q(S k ,A k )=q(S k ,A k )+α[U k -q(S k ,A k )], Where, k is the termination condition that is satisfied before the kth maintenance action is performed on the distribution network; q(S k , A k ) is the distribution network in a healthy state S k Perform maintenance action A k The corresponding Q value, α is the learning rate set according to the Q learning method; U k is the biased reward after the distribution network reaches the terminal state, R k+1 is the reward corresponding to the k+1th maintenance action performed on the distribution network; γ is the discount factor set for the reward for the maintenance action performed on the distribution network at the current moment; The distribution network is in a healthy state S k+1 The maximum reward obtained by performing the maintenance action below.

7. The method according to claim 1, characterized in that The network structure of the distribution network connected to the distributed power source includes: multiple distributed power sources, multiple power supply nodes, multiple segment switches and tie switches; the multiple distributed power sources are sequentially connected to the multiple power supply nodes through distribution network lines, each segment switch is connected between two adjacent power supply nodes, and the tie switch is connected to the feeder between two power supply lines in the distribution network; The distributed power source is used to provide electric energy to multiple power supply nodes in the distribution network; The section switch is used to isolate the power supply node in the fault area of ​​the distribution network by disconnecting itself; The tie switch is used for providing a backup power supply path to the power supply nodes in the non-fault power failure area in the power distribution network by closing itself.

8. An active distribution network health management system based on reinforcement learning, characterized in that: include: An environment acquisition module is used to acquire the network structure of the distribution network connected to the distributed power source, and to construct the objective function and constraint conditions of the distribution network status maintenance based on the network structure; A decision process building module, used to build a health state set, a maintenance action set and a reward function of the distribution network in a Markov decision process based on the objective function and constraint conditions of the distribution network state maintenance; An optimal action determination module, for determining, for each health state in the health state set, an optimal action with the maximum reward value of the reward function in the health state in the maintenance action set by using a reinforcement learning algorithm; The maintenance strategy output module is used to determine the state maintenance strategy of the distribution network by comprehensively analyzing the optimal actions under each health state.

9. An electronic device, characterized in that: include: at least one processor and memory; The memory and the processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the active distribution network condition maintenance method based on reinforcement learning as described in any one of claims 1 to 7 is implemented.

10. A readable storage medium, characterized in that: An execution program is stored thereon, and when the execution program is executed, the active distribution network status inspection method based on reinforcement learning as described in any one of claims 1 to 7 is implemented.

Images