A post-disaster restorative maintenance decision-making method based on deep reinforcement learning

Abstract

The present invention relates to the field of deep learning artificial intelligence technology, and in particular to a post-disaster restorative maintenance decision-making method based on deep reinforcement learning. A post-disaster restorative maintenance decision-making method based on deep reinforcement learning provides an efficient and intelligent solution for post-disaster restoration of large-scale distributed cluster systems through the combination of complex network models and deep reinforcement learning algorithms. After an extreme event, the method can dynamically adjust the repair strategies of multiple maintenance teams to minimize the elasticity loss in the system recovery process, thereby achieving rapid and efficient recovery of the system and ensuring that the cluster system can resume normal operation as soon as possible after a disaster.

Description

Technical Field

[0001] The present invention relates to the field of deep learning artificial intelligence technology, and in particular to a post-disaster restorative maintenance decision-making method based on deep reinforcement learning. Background Art

[0002] With the rapid development of modern society, cluster systems have been widely used in large-scale distributed infrastructures such as power systems, communication networks, and transportation networks. These cluster systems are usually composed of multiple key nodes and links to ensure the stable operation and efficient service of the system. However, the occurrence of extreme events such as earthquakes, typhoons and other natural disasters may cause serious damage to some nodes and links in the cluster system, thereby significantly reducing the overall performance of the system and even causing partial or complete failure of the system.

[0003] Post-disaster restorative maintenance is a key issue in cluster system operation and maintenance. In particular, how to quickly restore the normal operation of the system after an extreme event is extremely important for social and economic impacts. Traditional post-disaster repair strategies often rely on manual experience or simple rule algorithms. These methods show obvious deficiencies in dealing with the complexity and large-scale distributed characteristics of modern cluster systems. Specifically, existing methods are difficult to handle a large number of damaged nodes and links within a limited time, and lack effective optimization methods to coordinate the collaborative work of multiple maintenance teams, resulting in a time-consuming and inefficient repair process, and the system's recovery speed and effect are difficult to be satisfactory.

[0004] In order to overcome the above technical difficulties, post-disaster restoration strategies based on artificial intelligence and deep learning have received more and more attention and research in recent years. Among them, Deep Reinforcement Learning (DRL) has become an important research direction in the field of post-disaster restoration because it can obtain the optimal decision-making strategy through continuous learning in a dynamic and uncertain environment. By introducing deep reinforcement learning algorithms, after an extreme event occurs, it is possible to automatically generate an optimized scheduling strategy for multiple maintenance teams, adjust the repair sequence and path planning in real time, and thus significantly improve the recovery efficiency and effectiveness of the system.

[0005] In addition, as an effective tool for analyzing and optimizing cluster systems, complex network models can be used to simulate and evaluate the extent of damage to the system caused by extreme events and identify key nodes and links that are critical to system recovery. By combining complex network modeling and deep reinforcement learning algorithms, dynamic optimization of post-disaster restorative maintenance strategies can be achieved to ensure the rapid recovery of cluster systems after extreme events.

[0006] Aiming at typical problems such as corrective maintenance in cluster system operation and maintenance, this paper focuses on the representative frontier problem of this type of problem - the "post-disaster repair" strategy of cluster system, which is of great significance for the operation and maintenance of cluster systems for rapid repair and performance recovery under the influence of extreme events such as earthquakes and typhoons. The difficulty of the problem lies in the fact that large-scale distributed cluster systems in geographic space need to pay attention to the damage state of the super-large-scale node set of the cluster system and the cluster connectivity in their post-disaster repair work, and abstract the cluster system through complex network modeling methods to analyze the damage state of the links between nodes. On this basis, a comprehensive analysis is made of the temporal dynamic characteristics of the cluster system under the influence of extreme events. At the same time, for the support system responsible for post-disaster repair work, due to certain differences in factors such as maintenance personnel, spare parts and support equipment, each maintenance team in the entire post-disaster repair process shows different spatiotemporal dynamic characteristics in its maintenance support business level. Therefore, for the post-disaster repair plan, it is necessary to consider the damage state of large-scale distributed nodes and their links in the cluster system under extreme event scenarios, the maintenance sequence of nodes and their links, and the impact of key hub nodes and their links on the cluster performance recovery, and on this basis, reasonably plan the maintenance guarantee tasks of multiple maintenance teams, resulting in the cluster system post-disaster repair strategy problem having the coupling characteristics of large-scale node link maintenance sequence and multi-maintenance team path planning. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a post-disaster restorative maintenance decision-making method based on deep reinforcement learning, which aims to provide an efficient and intelligent solution for post-disaster restoration of large-scale distributed cluster systems through the combination of complex network models and deep reinforcement learning algorithms. This method can minimize the elasticity loss in the system recovery process by dynamically adjusting the repair strategies of multiple maintenance teams after extreme events, thereby achieving rapid and efficient recovery of the system and ensuring that the cluster system can resume normal operation as soon as possible after the disaster.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0009] A post-disaster restorative maintenance decision-making method based on deep reinforcement learning, the method comprising the following steps:

[0010] 1) Build a complex network model for the nodes and links in the cluster system, simulate the local damage caused by extreme events to the cluster system, and generate the damage adjacency matrix and node state vector;

[0011] 2) Using a deep reinforcement learning algorithm based on the Actor-Critic architecture, we design and train a policy-value neural network (AC-Net) to predict the policy function and value function of maintenance actions under the current state of the cluster system;

[0012] 3) Based on the Monte Carlo Tree Search algorithm (MCTS), combined with the output of the policy-value neural network, the search and optimization of the corrective maintenance actions of multiple maintenance teams are performed to generate the optimal repair path and timing for each maintenance team;

[0013] 4) During the repair process, dynamically adjust the maintenance strategies of each maintenance team according to the real-time recovery status and network characteristics of the cluster system, minimize the elasticity loss of system recovery, and improve the recovery speed of the overall system performance.

[0014] Preferably, the complex network model comprises:

[0015] Construct a network topology diagram for the nodes of the cluster system and the link relationships between them, where the nodes represent the key units in the cluster system and the links represent the communication or energy transmission paths between the nodes;

[0016] By simulating extreme events, the damage center and damage radius are randomly generated, the damage conditions of nodes and links in the damaged area are determined, and the damage adjacency matrix is ​​formed.

[0017] Preferably, before the local attack, the cluster system has an adjacency matrix A , the local damage is randomly initialized to the damage center and damage radius at time τd; then the damage adjacency matrix is ​​generated A d and the node vector S d nodes ;

[0018] (1.1)

[0019] When node n i When destroyed i node =0, otherwise S i node =1.

[0020] Preferably, the policy-value neural network (AC-Net) comprises:

[0021] An input layer for processing node and link state information;

[0022] A deep residual network (ResNet) module to extract deep features of the current state of the cluster system;

[0023] A policy output module to generate a priori probability matrix for the next maintenance action;

[0024] A value output module is used to evaluate the effectiveness of the repair strategy under the current state.

[0025] Preferably, the Monte Carlo tree search algorithm comprises:

[0026] Under the current cluster system state, the search process of the maintenance action is performed through Monte Carlo tree search based on the prior parameters output by the policy-value neural network;

[0027] The upper confidence interval (UCB) strategy is used to balance the exploration and exploitation of maintenance actions during the tree search process to generate a globally optimal sequence of maintenance strategies.

[0028] Preferably, the repair sequence and path planning of the multiple maintenance teams includes:

[0029] For different maintenance teams, the path planning and repair sequence of multiple maintenance teams are generated according to their maintenance capabilities, node locations and link damage conditions;

[0030] During the planning process, the collaborative actions of multiple maintenance teams are optimized through the Actor-Critic architecture to minimize the elasticity loss of system recovery.

[0031] As a preferred option, cluster efficiency (Figure of Merit, FOM) is selected as the performance quantitative indicator, and the average efficiency of the cluster system is E ( G ) is normalized as follows to define the FOM over time τ:

[0032] (1.2)

[0033] Where E*(G) is the average efficiency before the local attack event, and Eτ(G) is the average efficiency at time τ;

[0034] Assume that the local attack event occurs at time τ d , which causes a sharp drop in cluster functional performance; then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and at time τ d Reaching FOM threshold FOM thr Lower elastic loss RL To quantify the loss suffered by the cluster due to a local attack:

[0035] (1.3).

[0036] As a preferred approach, cluster post-disaster repair under extreme events is regarded as a Corrective Maintenance Decision (CMD) problem for multiple maintenance teams. In order to achieve optimal recovery, the CMD problem aims to minimize the recovery time under a limited recovery time and a certain recovery level. RL,The model representation of the CMD problem of multiple maintenance teams is as follows:

[0037]

[0038] In constraint (1.5), c is the maintenance team number, F={1,2,…, n cre} is composed of n cre A maintenance team consisting of a support system cluster. t c It's the maintenance team c The maintenance time step, T c {1,2,…, T c} is the time step set in its own maintenance process, a * tc Indicates the time step t c Maintenance actions, Ω c It's the maintenance team c A set of action sequences;

[0039] In constraint (1.6), t is the maintenance time step of the maintenance team and T={1,2,…,T} is the maintenance team time step set, a * t Indicates the time step t Maintenance actions, Ω It is a set of maintenance team action sequences;

[0040] Constraint (1.7) represents the set of action sequences of the cluster of the guarantee system Ω and each maintenance team action sequence set Ω the relationship between;

[0041] In the constraint (1.8) τ(a * tc ) indicates execution of action a * tc The time interval, τ thr is the recovery time threshold;

[0042] Constraint (1.8) forces the total repair time of the maintenance team to be below the threshold recovery time, while constraint (1.9) forces the cluster performance to be within time τ r Reached FOM threshold.

[0043] Preferably, the method is suitable for corrective maintenance in a distributed power system cluster. In the system damage repair under extreme events, the method can effectively improve the system recovery speed and performance.

[0044] Furthermore, a computer-readable storage medium stores a computer program or instruction thereon, which implements the method when executed by a processor.

[0045] The present invention adopts the above technical solution and proposes a post-disaster restorative maintenance decision-making method based on deep reinforcement learning. By combining the complex network model and the deep reinforcement learning algorithm, the present invention effectively solves the problems of low efficiency and slow response of the traditional post-disaster repair method when dealing with large-scale damage of the cluster system, and achieves significant technical effects, which are specifically manifested in the following aspects:

[0046] 1. Improve system recovery efficiency: This invention optimizes the scheduling and path planning of multiple maintenance teams through the deep reinforcement learning algorithm (DRL), and can quickly generate the optimal repair strategy after an extreme event. Compared with traditional manual experience or simple rule algorithms, this invention can more effectively coordinate the collaborative work of multiple maintenance teams, ensuring that key nodes and links are repaired in the shortest time, thereby significantly improving the system's recovery speed.

[0047] 2. Dynamically adjust the repair strategy: This invention introduces a deep reinforcement learning model based on the Actor-Critic architecture, which can dynamically adjust the task allocation and repair order of the maintenance team according to the real-time recovery status and damage of the cluster system. This dynamic adjustment mechanism makes the repair process more flexible, can quickly respond to emergencies or secondary disasters, and ensure the optimality and adaptability of the repair strategy.

[0048] 3. Minimize the elasticity loss during system recovery: By evaluating the criticality of nodes and links in the system through complex network models, the present invention can identify nodes and links that are critical to system performance recovery and repair them first. Combined with the optimization decision of deep reinforcement learning algorithm, the present invention can minimize the elasticity loss of the system during post-disaster recovery and ensure that the performance of the cluster system is restored to a level close to the pre-disaster level in the shortest time.

[0049] 4. Adapt to post-disaster repair of various distributed cluster systems: The method of the present invention is not only applicable to power systems, but can also be widely used in other types of large-scale distributed cluster systems such as communication networks and transportation networks. Through the combination of complex network modeling and deep reinforcement learning algorithms, the present invention can achieve efficient and rapid post-disaster repair in a variety of application scenarios, with good versatility and scalability.

[0050] 5. Enhance the resilience of the system to deal with extreme events: The present invention significantly enhances the response and recovery capabilities of the cluster system after extreme events through continuous optimization and iterative repair strategies. This method can not only cope with known disaster situations, but also gradually improve the resilience of the system by learning new damage patterns, and enhance its recovery capabilities in the face of extreme events that may occur in the future.

[0051] In summary, the present invention combines complex network models with deep reinforcement learning to achieve intelligent, dynamic and optimized post-disaster restorative maintenance decisions, which not only improves the recovery speed and efficiency of the cluster system after extreme events, but also minimizes the elasticity loss in the recovery process, providing an advanced technical means for post-disaster restoration of large-scale distributed cluster systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 shows the complex network model of a cluster system under large-scale destruction.

[0053] Figure 2 is a schematic diagram of the residual module structure.

[0054] Figure 3 is a diagram showing the construction of a complex network model for a distributed power system cluster.

[0055] Figure 4 shows an example of large-scale localized damage to a power system cluster in a regional power grid.

[0056] Figure 5 is a diagram of the corrective maintenance process of multiple maintenance teams.

[0057] Figure 6 is an analysis diagram of AC-Net learning and training.

[0058] Figure 7 shows the elastic loss analysis of different methods for various post-disaster damage situations.

[0059] Figure 8 is a maintenance strategy elasticity analysis chart for different maintenance team sizes.

[0060] Figure 9 is a maintenance strategy elasticity analysis diagram for maintenance teams with different business capabilities. DETAILED DESCRIPTION

[0061] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] The present invention aims at the post-disaster repair strategy problem of the cluster system, considers the two-level decision of maintenance sequence and path planning, and studies the restorative maintenance decision method based on Actor-Critic Monte Carlo Tree Search (AC-MCTS). The designed restorative maintenance decision method needs to consider the spatial position of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, and comprehensively analyzes the two-level decisions of node link maintenance sequence and multi-team path planning, which includes two core ideas: a deep reinforcement learning Actor-Critic architecture considering the two-level decision of multi-team maintenance sequence and path planning, and an AC-MCTS algorithm for restorative maintenance of cluster systems. For the two-level decision of the repair sequence of the cluster system and the path planning of the maintenance team, solving the global optimal decision under the condition of uneven benefits of multiple maintenance teams is the core feature that needs to be focused on. Therefore, it is necessary to focus on the restorative maintenance strategy considering multiple maintenance teams, and be able to plan the optimal path for multiple maintenance teams to repair damaged systems and components, as well as the optimal sequence for each maintenance team to repair damaged systems and links between systems. In order to solve the two-level decision-making characteristics of the maintenance strategy between multiple maintenance teams and a single maintenance team in this problem, the AC-MCTS method designed by the present invention uses the damage information of cluster system nodes and links to determine the optimal maintenance actions and paths of multiple maintenance teams. In addition, the above-mentioned Actor-Critic architecture needs to design a strategy-value neural network to estimate the value function and the strategy function, and then perform the maintenance action search process based on AC-MCTS, and finally obtain the maintenance action search probability and the optimal maintenance action. Based on the proposed AC-MCTS corrective maintenance decision algorithm, the present invention conducts case verification in the case analysis part for post-disaster repair of multiple maintenance teams in extreme event scenarios for distributed power system clusters. The case analysis results show that the proposed corrective maintenance decision algorithm can provide an efficient corrective maintenance solution in the post-disaster repair work of distributed power system clusters, and realizes the rapid recovery of power system cluster performance under large-scale damage.

[0063] 1.1 Description of cluster system post-disaster repair problems

[0064] 1.1.1 Large-scale destruction and repair of cluster systems

[0065] First, we analyze the cluster system and its large-scale destruction, consider the interconnection relationship between nodes in the distributed cluster system, and build its complex network model, such as Figure 1 shown.

[0066] (1) Destruction

[0067] superior Figure 1 In the above example, based on the cluster, the destruction center and destruction radius are randomly generated. Figure 1 The red area in the figure is the local damage area. Assume that all nodes and links in the red area are invalid. Figure 1 Then, the state of the links between nodes can be described as the current adjacency matrix after the local attack: A damage ,The state vector of a node in a cluster system can be described as:

[0068] (1.1)

[0069] When node n i When destroyed i node =0, otherwise S i node =1

[0070] (2) Repair

[0071] This paper will propose a rich and executable strategy to support maintenance activities, considering maintenance teams, maintenance time series and maintenance objects. The proposed strategy aims to make up for the limitations of other current studies, which mainly consider the repair object. The maintenance activities are performed in parallel by multiple maintenance teams. Repairing damaged nodes i The average time of MTTR i The time to repair each edge can be expressed as follows:

[0072] (1.2)

[0073] in v edge Indicates how quickly broken links can be repaired. i Move to j The time can be expressed as follows:

[0074] (1.3)

[0075] in v move Indicates the movement speed of the repair team.

[0076] (3) Recovery

[0077] Elasticity is currently widely used to characterize the recovery performance of a cluster after a local attack. Therefore, the present invention introduces cluster elasticity to evaluate the effectiveness of the recovery strategy, with the goal of improving cluster performance and shortening recovery time. In order to describe the quantitative measurement of cluster elasticity, it is necessary to define a quantity that represents the functional performance of the cluster. The present invention selects cluster effectiveness (Figure of Merit, FOM) as a quantitative performance indicator. On this basis, the average efficiency of the cluster system isE ( G ) is normalized as follows to define the FOM over time τ:

[0078] (1.4)

[0079] Where E*(G) is the average efficiency before the local attack event, and Eτ(G) is the average efficiency at time τ. In addition, other performance parameters of the power grid can also be used in the definition of FOM.

[0080] Assume that the local attack event occurs at time τ d , and the cluster functional performance drops sharply. Then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and at time τ d Reaching FOM threshold FOM the The following resilience indicators are used to quantify the loss suffered by the cluster due to local attacks:

[0081] (1.5)

[0082] 1.1.2 Corrective maintenance decision-making problem for post-disaster emergency repair

[0083] Cluster post-disaster repair under extreme events is regarded as a corrective maintenance decision (CMD) problem for multiple maintenance teams. The constraints are the recovery time and recovery level, and the optimization goal is the elastic loss of the cluster system. RL , the basic decision variable is the restorative maintenance action. From the nature of the decision variables, this problem is an integer programming problem using binary variables. For each maintenance team, considering the three actions related to post-disaster repair, namely, repairing nodes, repairing links, and moving, the number of strategies is 2 N+K ; For multiple maintenance teams, their strategy spaces are coupled to each other, so this problem is an NP-hard problem. In order to achieve optimal recovery, the CMD problem aims to minimize under limited recovery time and a certain recovery level. RL Therefore, the CMD problem for multiple maintenance teams can be expressed as follows:

[0084] In the constraint (1.62), c is the maintenance team number, F={1,2,…, n cre} is composed of n cre A maintenance team consisting of a support system cluster, t c It's the maintenance team c The maintenance time step is T c ={1,2,…,T c} is the time step set in its own maintenance process, a * tc represents the time step t c Maintenance action, Ω c It's the maintenance team c In the constraint (1.63), t is the maintenance time step of the maintenance team and T ={1,2,…, T c} is the maintenance team time step set, a * t Indicates the time step t The maintenance action is, Ω is the maintenance team action sequence set. Constraint (1.64) represents the maintenance system cluster action sequence set Ω and each maintenance team action sequence set Ω c In the constraint condition (1.65), τ( a * tc ) indicates execution of an action a * tc The time interval, τ thr is the recovery time threshold. Constraint (1.65) forces the total repair time of the maintenance team to be lower than the threshold recovery time, while constraint (1.66) forces the cluster performance to be within time τ r Reached FOM threshold.

[0085] 1.2 Corrective maintenance decision-making algorithm considering timing and path planning

[0086] The corrective maintenance decision-making for post-disaster repair of cluster systems needs to consider the spatial position of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, comprehensively analyze the node link maintenance timing and multi-team path planning, and design a corrective maintenance decision-making method based on the deep reinforcement learning Actor-Critic architecture. Considering the two-level decision-making of multi-team maintenance timing and path planning, it supports corrective maintenance work such as the generation of corrective maintenance plans for cluster systems with network characteristics. This method contains two core ideas: a deep Actor-Critic architecture that considers the two-level decision-making of multi-team maintenance timing and path planning, and an AC-MCTS algorithm for corrective maintenance of cluster systems.

[0087] 1.2.1 Actor-Critic architecture considering two-level decision-making of multi-team maintenance timing and path planning

[0088] Regarding the research content of corrective maintenance considering timing and path planning, especially for the corrective maintenance work of network cluster systems such as infrastructure networks, two basic challenging problems are the two-level decisions of the repair timing and path planning of the network cluster system. Therefore, the present invention focuses on the study of corrective maintenance strategies considering multiple maintenance teams, which can plan the optimal paths for multiple maintenance teams to repair damaged systems and components, as well as the optimal timing for each maintenance team to repair damaged systems and inter-system links. In order to solve the two-level decision-making characteristics of the maintenance strategies between multiple maintenance teams and a single maintenance team in this problem, the present invention intends to design a deep Actor-Critic architecture that considers the two-level decisions of multi-team maintenance timing and path planning. The idea of ​​this decision-making architecture uses the damage information of the network cluster system to determine the optimal maintenance actions and paths for multiple maintenance teams.

[0089] The main idea of ​​the deep Actor-Critic architecture considering two-level decisions of multi-team maintenance timing and path planning is to iteratively execute the multi-repair team CMDP process in the policy iteration process. First, the simulation environment generates local damage and network cluster system status. Then, the restorative maintenance agent executes the multi-repair team CMDP process and generates a series of maintenance actions based on the AC-MCTS algorithm. When generating each maintenance action, the restorative maintenance agent uses AC-Net to extract the current recovery state of the network cluster system and the multi-repair team timing characteristics, and estimates the value function and policy function of the maintenance action to support the AC-MCTS algorithm to search for maintenance actions. Subsequently, the simulation environment rewards the maintenance strategy based on evaluation methods such as elasticity analysis. The maintenance strategy consists of a series of maintenance actions generated by the CMDP process. Finally, the AC-Net parameters are adjusted based on the training sample data set; the new AC-Net parameters are used in the next policy iteration process to provide MCTS with more effective maintenance action search support. For the CMD problem, the deep reinforcement learning framework proposed in the present invention. The framework shows four parts: multiple crews repair process (MCRP), multi-repair team repair timing, elasticity analysis and reward, and neural network training, as described below.

[0090] (1) Maintenance process of multiple maintenance teams

[0091] Before the local attack, the cluster system has an adjacency matrix A Then, the local damage occurs at time τ d The damage center and damage radius are randomly initialized. Then the damage adjacency matrix is ​​generated A d and the node vector S d nodes . FOM(τ d) can be calculated according to formula (1.4).

[0092] To restore cluster performance, a cre The maintenance team's support system cluster performs a series of maintenance actions during the MCRP period. This maintenance action sequence set is as follows:

[0093] (1.7)

[0094] Maintenance Team c The action sequence set can be described as follows:

[0095] (1.8)

[0096] Corrective maintenance actions are defined as a * t∣tc = a * tc∣t =( i , j ), which means that the maintenance team c Repair Node i With Node j After repairing the link, if the node j If the node is damaged, the maintenance team will continue to repair the node, otherwise continue to perform the next maintenance action. t The corrective maintenance agent uses the integrated AC-Net search algorithm to find the maintenance team that has just completed the last maintenance action. c Select maintenance action a * t∣tc .

[0097] The decision of the corrective maintenance agent can be described as follows:

[0098] (1.9)

[0099] in S=[S f ,S c ] The current maintenance status. S f represents the status of repairs performed by the maintenance team, and S c Representative maintenance team c The status of the repairs performed.

[0100] (2) Time sequence of multiple maintenance teams

[0101] For havingn cre The maintenance team's support system cluster has a time sequence and n cre The timing of each maintenance team.

[0102] Maintenance Team c The timing can be described as follows:

[0103] (1.10)

[0104] where τ tc Indicates that the maintenance team c Complete its maintenance tasks a * tc∣t =( i , j ) time, τ tc The calculation method of is as follows:

[0105] (1.11)

[0106] Where τ( a * tc∣t ) indicates execution of an action a * tc∣t The maintenance time consists of four situations. In the first situation, the maintenance team is at the node i The location of the node j Normal. In the second case, the maintenance team at the node i The location of the node j In the third case, the maintenance team is at the node i ' position, and the node j Normal. In the fourth case, the maintenance team is at the node i ' position, and the node j is in a damaged state. According to (1.2) and (1.3), τ( a * tc∣t ) is calculated as follows:

[0107] (1.12)

[0108] The time series of the security system cluster can be described as follows:

[0109] (1.13)

[0110] Where τ t =0=τd , τ t Complete actions on behalf of the maintenance team a * t∣tc = a * tc∣t =( i , j ) maintenance time.

[0111] (3) Elasticity analysis and rewards

[0112] In the above MCRP process, for maintenance actions a * t∣tc = a * tc∣t =( i , j ), if the node j is normal, then repair the link ( i , j ) after which the link is assumed to be operational again; if the node j is damaged, then repair the link ( i , j ) and nodes j Afterwards, it is assumed that the link and the node are back in operation. In the MCRP process, the FOM (τ t ) can be calculated according to formula (1.4). When the maintenance teams perform their respective maintenance actions, the FOM (τ t ) can be considered unchanged; whenever a maintenance team completes a maintenance action, the repaired links and nodes resume operation, and the FOM (τ t ) can be considered as a step increase. Therefore, FOM(τ t ) is a step function. After completing MCRP, FOM(τ t ) Integrate over the duration of MCRP to calculate the elastic loss of the entire repair process RL .

[0113] The repair sequence of damaged links and damaged nodes in the cluster, as well as the path planning of each maintenance team, will affect the step function FOM (τ t ) performance recovery curve, thus affecting elasticity loss RL Therefore, the optimized maintenance strategy needs to obtain a smaller elastic loss. RL , then the reward of the maintenance strategy executed by the entire MCRP process can be defined as the elastic loss RLThe inverse proportional function of the inverse proportional function is the integral of the cluster FOM without elastic loss in the specified time interval, as shown below:

[0114] (1.14)

[0115] (4) Actor-critic network training

[0116] For every moment t , Corrective Maintenance Agent uses AC-Net f θ ( S, A ) = ( p , v ) is a variant of the MCTS algorithm f ts ( S,p,v ) = ( π , a * ) provides search guidance, where p is the prior repair probability matrix, which is used to guide the tree search to select corrective repair actions, and v is a normalization parameter used to evaluate the reward of the current corrective repair action selection. The search algorithm uses the parameter p and v To output the search probability π , to select high-quality repair actions a * .

[0117] Every moment t The AC-Net training data is stored as (S t ,A, π t ,Z t ),in It's the action a * t∣tc α and β are empirical parameters. When the corrective maintenance agent completes the MCRP, an episode is completed and the data ( S , A , π , z ) is sampled from all time steps of this MCRP. AC-Net then uses the empirical data ( S , A , π , z ) to adjust its parameters θ The training goal is to maximize the prior repair probabilityp With tree search probability π similarity and minimize the prior value v and rewards z Therefore, by using gradient descent to adjust the parameters θ , the loss function includes cross-entropy losses (CELs) and mean squared errors (MSEs), as shown below:

[0118] (1.15)

[0119] in c L2 is a constant used to control the level of L2 regularization to prevent overfitting.

[0120] For each episode, a small batch of training samples is randomly selected from the dataset consisting of all episode data, and then batch training is used to adjust the parameters θ . Adjust the parameters θ After that, AC-Net can improve the decision-making ability of the corrective maintenance agent in the next episode. In addition, the initialized AC-Net parameters θ is randomly generated. As the number of episodes increases, the trained AC-Net can provide stronger search guidance for tree search. After enough episodes, AC-Net has been well trained, and MSE and CEL tend to be stable. The restorative maintenance agent can provide effective decision support for cluster systems to solve CMD problems under large-scale local damage. In addition, when generating maintenance strategies for large-scale local damage during the operation phase of the cluster system, there is no need to train AC-Net again.

[0121] 1.2.2 AC-MCTS algorithm for corrective maintenance of cluster systems

[0122] When the multi-maintenance team CMDP process generates each maintenance action, that is, any maintenance action of the CMDP process t At this moment, the cluster system feature tensor X t Under the condition of, the result of estimating the value function and strategy function by AC-Net ( P t , V t 2) As input, the maintenance action search process is performed based on AC-MCTS, such as Figure 2 As shown in the figure, it mainly includes four steps: selection, expansion and evaluation, backtracking and execution, and finally the maintenance action search probability can be obtained. πand best maintenance actions .

[0123] 1.2.2.1 AC-MCTS Algorithm

[0124] In CMDP implementation t At this moment, for the decision feature tensor X t , using the prior parameters obtained by the AC-Net prediction model ( P t , V t ) as input, and the maintenance action search operation is performed based on the MCTS method, and finally the improved maintenance state transition probability can be obtained. π and best maintenance actions a t * , whose expression is:

[0125] (1.16)

[0126] MCTS search tree each tree node X All branches of X , a ), corresponding to all the next maintenance actions of this tree node a ∈ Action ( X ), each branch ( X , a ) stores a set of statistics as follows:

[0127] (1.17)

[0128] in N ( X , a ) indicates the number of visits; W ( X , a ) represents the sum of total action values; Q ( X , a ) represents the average action value; P ( X , a ) indicates selecting a branch ( X , a )’s prior probability.

[0129] When the multi-maintenance team CMDP process generates each maintenance action, that is, at any time in the CMDP process t At this moment, the AC-MCTS algorithm performs the following four steps.

[0130] 1) Select

[0131] First, t Characteristic tensor of the moment cluster system X t As the root node of the Monte Carlo tree X 0 (i.e. the cluster system status at each moment), and from the root node X 0 Start searching until L The leaf node at the end of the search tree is reached at any time X L Ends at l time( l < L ), based on the Upper Confidence Bound Apply to Tree (UCT) and the Maximum Upper Confidence Bound (UCB) algorithm, the best action under the current iteration conditions is selected according to the statistical data stored in each branch of the search tree. , whose expression is:

[0132] (1.18)

[0133] (1.19)

[0134] where c puct is a constant determined by the degree of MCTS search; a∈Action(X), Action(X) represents the action space;

[0135] In the early stage, this search strategy tends to select maintenance actions that usually have higher prior probabilities and lower visit times, but as the search proceeds, it tends to select "unexpanded" nodes, so that the MCTS search takes into account both breadth and depth.

[0136] 2) Extension and evaluation

[0137] The leaf node X L Add to the queue as the input of ResNet, and then the leaf node X L Expand, each edge (X L ,a) Initialize storage data:

[0138] (1.20)

[0139] All positions in the queue are evaluated by the neural network and the leaf nodes are obtained. XL The strategic value of v , the search thread is locked until the policy evaluation is completed.

[0140] 3) Backtracking

[0141] After expansion and evaluation, the leaf nodes X L The strategic value of v Backtrack along the original path to the parent node. During the backtracking process, each branch of the search path The stored access count parameter is updated once, and its expression is:

[0142] (1.21)

[0143] At the same time, search for the branch The total action value and average action value parameters of the maintenance action are also updated once, and their expressions are:

[0144] (1.22)

[0145] (1.23)

[0146] 4) Execution

[0147] By iterating the above three steps 1000 times, the maintenance status probability is obtained π , and select the feature tensor of the cluster system at the current moment X t Global optimal maintenance action under the condition a * t Among them, the improved cluster system maintenance status probability π It is exponentially related to the number of visits, and its expression is:

[0148] (1.24)

[0149] where τ is an empirical parameter that controls the search process.

[0150] Execute optimal maintenance actions a * t Afterwards, the feature tensor of the cluster system maintenance status is transformed from t time X t Transfer to t+ 1 moment X t+1 , the search tree will be X t+1 The new root node X0 , and keep the nodes in the original search tree X t+1 All child nodes below it and their statistics, while discarding the branches of the remaining nodes at the same level as this node.

[0151] In summary, the AC-MCTS search algorithm can be expressed as f ts ( S,p,v ) = ( π , a * ).

[0152] 1.2.2.2 Actor-Critic Neural Network Prediction Model

[0153] AC-Net f θ ( S, A ) = ( p , v ) is a dual-output residual neural network (ResNet), which is used as a maintenance strategy prediction model. The network parameters are randomly initialized. θ 0.

[0154] At every moment t , AC-Net f θ The input is the current repair state before damage S t = [ S f t , S c tc ] and the adjacency matrix A . S f t It can be written as follows:

[0155] (1.25)

[0156] Among them A f t is the time step of MCRP t The adjacency matrix at , if t<1, then A f t = A f 1 =A damage Damage. The historical adjacency matrix is ​​necessary because the restorative maintenance agent cannot fully observe all state information from the current adjacency matrix, because MCRP is a sequential decision process. At the same time, S ctc It can be written as follows:

[0157] (1.26)

[0158] Among them A c tc It's the maintenance team c Time step of the repair process t c The adjacency matrix of c <1, then A c tc = A 1 c =A d .

[0159] The output of the Actor module and the Critic module is the prior repair probability matrix P t Sum V t .matrix P t is a N × N The adjacency matrix, whose elements P ij Yes Select Repair Action a = ( i , j ) probability. V t is a standardized parameter used to evaluate the reward of the current corrective repair action selection.

[0160] The AC-Net in the restorative repair agent uses a residual module consisting of residual blocks of convolutional layers with ReLU functions to process the input feature S t , and the input features A is a constant matrix and is processed by a multi-layer module consisting of fully connected layers. The residual module and the multi-layer module include shared layers and are merged into two independent modules, namely policy and value. In addition, the policy and value modules take their inputs from the shared layers and output the prior repair probability using fully connected layers and sigmoid functions. P t Sum V t .

[0161] The ResNet network structure includes convolution module, residual module, batch normalization module, ReLU function module, etc. Input feature tensor X It is processed by a deeply stacked module, which consists of a single convolution module and an intermediate layer module that deeply stacks multiple residual modules:

[0162] a) Separate convolutional modules:

[0163] ① A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0164] ② ReLU function layer;

[0165] b) Intermediate layer module: The intermediate layer of the deep neural network is constructed by stacking residual modules (taking 9 layers of stacked residual modules as an example). The structure of the residual module is as follows: Figure 5 As shown, it includes the following structures:

[0166] ① A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0167] ② Batch normalization layer;

[0168] ③ ReLU function layer;

[0169] ④ A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0170] ⑤ Batch normalization layer;

[0171] ⑥ Addition module: Add the input feature tensor of the residual module to the output feature tensor of the previous layer;

[0172] ⑦ ReLU function layer;

[0173] The output of the deeply stacked residual module is divided into two parts: the policy module and the value module:

[0174] c) Strategy Module:

[0175] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0176] ② ReLU function layer;

[0177] ③ Fully connected output layer: outputs a feature vector of size m×n, corresponding to the logit of the prior maintenance state transition probability p of the cluster;

[0178] d) Value Module:

[0179] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0180] ② A linear fully connected layer with a size of 256;

[0181] ③ ReLU function layer;

[0182] ④ Linear fully connected layer;

[0183] ⑤ Sigmoid output layer: outputs a scalar value in the interval (0,1).

[0184] The expression of the above ResNet prediction model is f θ ( X ) = ( p , v ), θ is the network parameter of ResNet, the initial network parameter θ 0 is obtained by random initialization.

[0185] 1.3 Analysis of post-disaster emergency repair cases

[0186] In the case analysis part, the present invention selected a large-scale distributed power system cluster for case study, and took the power system cluster in a certain regional power grid as the case object, and analyzed the performance of the power system cluster in a certain regional power grid under extreme events through Python simulation, and generated a corrective maintenance strategy based on the proposed AC-MCTS algorithm, and then analyzed the recovery effect. In addition, the power system cluster in the regional power grid consists of 228 nodes and 612 links. For each maintenance team, considering the three types of actions related to post-disaster repair, namely repairing nodes, repairing links, and moving, based on the basic assumptions of the corrective maintenance decision problem, it can be known that the number of strategies is 2 228+612 =7.33×10 252 ; For multiple maintenance teams, their strategy spaces are coupled with each other, and the total number of strategies will increase further.

[0187] 1.3.1 Power system cluster analysis

[0188] Taking the distributed power system cluster in the power network as an example, the power system is abstracted into nodes, focusing on the interconnection between nodes, and building its complex network model. On this basis, the temporal dynamic characteristics of nodes and links in the power system cluster under large-scale damage scenarios are considered, including various operation and maintenance situations such as normal, damage, and repair. Figure 3 shown.

[0189] Taking a regional power grid as an example, we first comprehensively analyze the basic information of the nodes in the power system cluster in a regional power grid and the basic information of the links between nodes. As shown in Table 1, the node numbers and geographical location information of the power system cluster in a regional power grid are given. Table 2 gives the link information of the power system cluster in a regional power grid, that is, the adjacency matrix information of the complex network model of the power system cluster. Based on the above information, the complex network modeling method can be used to establish a complex network model of the power system cluster in a regional power grid. G .

[0190] Table 1 Basic information of power system cluster nodes in a certain area power grid

[0191]

[0192] Table 2 Basic information of power system cluster links in a certain area power grid

[0193]

[0194] 1.3.2 Post-disaster recovery analysis

[0195] 1.3.2.1 Analysis of sabotage cases

[0196] Based on the complex network model of the power system cluster in a certain area of ​​the power grid in the previous section G Modeling, continue the operation and maintenance status of the power system cluster under extreme events, consider using a circular area to represent an extreme event such as an earthquake, randomly generate the damage center and damage radius of the circular area, and then the power system cluster in a certain area of ​​the power grid under extreme events G The destructive situation Figure 4 As shown, pink nodes represent failed nodes.

[0197] The maintenance task is performed in parallel by multiple maintenance teams with different maintenance capabilities. In the present invention, five maintenance teams with different V edge and V move , and divide the nodes into three levels: the node weight of the first level node is w ∈(0, 2]; the node weight of the second-level node is w ∈(2, 4]; the node weight of the third-level node is w ∈(4, +∞). Each maintenance team has different MTTR : MTTR 1st , MTTR 2nd and MTTR 3rd ,The maintenance capability information of each maintenance team is shown in Table 3. In addition, the maintenance capability of the ,maintenance team can also be defined according to its actual ,situation.

[0198] Table 3 Maintenance team information

[0199]

[0200] Figure 5 Four different scale damage cases of the power system cluster are analyzed, and the damage information of each damage case is shown in Table 4. Figure 5 The four MCRPs in are performed by the same maintenance team group, which consists of maintenance team 1, maintenance team 2 and maintenance team 3.

[0201] Table 4 Relevant information of different damage cases

[0202]

[0203] 1.3.2.2 Maintenance strategy and flexibility analysis

[0204] for Figure 5 For each failure case in , assume that at time τ damage =200h, a large-scale local damage event occurs, which causes a sharp decline in the functional performance of the power system cluster. Then corrective maintenance work is performed on the power system cluster to recover from the large-scale local damage, after which the functional performance is improved and restored at time τ restoration Reaching FOM threshold FOM thr = 0.96, set τ thr The parameter can be set according to the actual situation. Generally speaking, different MCRPs have different τ restoration The elasticity analysis of the above four MCRPs is as follows: Figure 5 As shown, its elastic loss RL As shown in Table 5.

[0205] Table 5 Elasticity loss of corrective maintenance by multiple maintenance teams

[0206]

[0207] 1.3.2.3 AC-Net Learning and Training Analysis

[0208] The proposed AC-MCTS method is applied to train the DRL agent using a server with an Intel i9 7980XE CPU and four RTX2080 TI-A11G GPUs. The training starts with a completely random AC-Net and lasts for about 30 hours without intervention. During the training process, each MCTS uses 800 simulations to perform 10,000 MCRPs, which is equivalent to nearly 0.2 seconds of thinking time for each repair action selection. AC-Net uses the dataset generated by MCRP to train parameters and applies min-batches for update.

[0209] Figure 6 The mean square error and cross entropy parameter training of AC-Net after each MCRP simulation are analyzed. MSE is between the actual results z and AC-Net predicted value v The goal is to predict the elastic loss RLCEL represents the AC-Net prediction prior probability p and MCTS strategy π The goal is to predict the strategy π Although the mathematical models of MSE and CEL are different, AC-Net is trained using the same training dataset and the same loss function, Figure 6 It can be seen that in some training time periods, the two curves have the same changing trend. In addition, some empirical parameters: α=2, β=4, c L2 = 10 -4 , c puct = 5. After about 8000 MCRPs, MSE and CEL tend to be stable, and then AC-Net can effectively guide MCTS to select maintenance actions for multiple teams.

[0210] 1.3.3 Discussion and Analysis

[0211] The main goal of post-disaster recovery of power system clusters is to generate feasible multi-team corrective maintenance strategies. The present invention conducts an extended analysis of multi-team corrective maintenance strategies to determine the impact of different factors on the optimal strategy.

[0212] 1.3.3.1 Analysis of decision-making methods

[0213] This paper proposes an efficient AC-MCTS method for post-disaster recovery of cluster systems. In addition, the heuristic hybrid game (HHG) method and the DQN method are also used to solve the problem. The three methods are analyzed separately by running Python programs for each method to solve the above four post-disaster damage situations. AC-MCTS gives the optimal solution with better RL than other methods, such as Figure 7 As shown in Figure 2. In the early MCRP stage under each damage situation, the DQN method and the HHG method can achieve better FOM than AC-MCTS because these two methods select each repair action with the goal of achieving the best FOM improvement in the current repair stage. However, they may also lead to local optimality, such as in the middle and late stages of MCRP when multiple repair team members have to spend more time moving. According to Figure 7 As shown in the results, AC-MCTS achieves better performance in a shorter time than other methods. RL , because it uses the value predicted by DNN, aiming to achieve better RL Therefore, the results show that AC-MCTS generates the optimal strategy for post-disaster recovery.

[0214] The efficiency of these methods is analyzed in Table 6. According to these results, the solving speeds of AC-MCTS, DQN, and HHG are stable, and they do not decrease exponentially with the increase in the number of destroyed nodes and links. However, the solving speed of the HHG method decreases significantly with the increase in the degree of destruction.

[0215] Table 6 Running time of different methods (seconds)

[0216]

[0217] 1.3.3.2 Maintenance teams of different sizes

[0218] Keeping the same post-disaster damage situation ( Figure 5 In the case of post-disaster damage scenario 1), four methods are used to consider three, four, and five different repair team sizes. The corrective repair results of the three repair teams (repair teams 1, 2, and 3) are shown in Figure 1. Figure 7 As shown in (a), the corrective maintenance of four maintenance teams (teams 1, 2, 3 and 4) and five maintenance teams (members 1, 2, 3, 4 and 5) are as follows: Figure 8 The elasticity loss under different numbers of maintenance teams is shown in Table 7. As the number of maintenance teams increases, the elasticity loss RL and total repair time is reduced, the corrective maintenance agent also obtains better RL Therefore, these results show that the Restorative Repair Agent exhibits stable performance for the disaster recovery problem with different numbers of repair teams, because the number of repair teams is randomly generated during the Restorative Repair Agent training process.

[0219] Table 7 Analysis of elasticity loss for different maintenance team sizes

[0220]

[0221] 1.3.3.3 Maintenance teams at different levels of expertise

[0222] Maintaining the same post-disaster damage status ( Figure 5 In the post-disaster damage case 1), based on the maintenance strategy generated by the restorative maintenance agent, the difference in cluster system performance recovery after the maintenance strategy is executed by the maintenance teams with different business levels is analyzed. The elasticity analysis of the cluster system after the maintenance team 1, 2 and 3 performs the restorative maintenance task is shown in the figure. Figure 5 As shown in (MCRP and resilience analysis of post-disaster damage case 1), the relevant analysis of the support system cluster composed of maintenance teams 2, 3 and 4 and maintenance teams 3, 4 and 5 are shown in Fig. 9 As shown in (a) and (b). Considering that the maintenance teams in the guarantee system cluster have different business capabilities, the elasticity loss of the cluster system recovery process is shown in Table 8. For the same corrective maintenance agent, the generated maintenance strategy achieves similar elasticity loss in almost the same total repair time. Therefore, these results show that the proposed method shows stable performance for maintenance teams with different business capabilities.

[0223] Table 8 Analysis of elasticity loss for maintenance teams with different business capabilities

[0224]

Claims

1. A post-disaster restorative maintenance decision-making method based on deep reinforcement learning, which is applicable to restorative maintenance in a distributed power cluster system. In the damage repair of the cluster system under extreme events, the method can effectively improve the recovery speed and performance of the cluster system, and is characterized in that: The method comprises the following steps: 1) Build a complex network model for the nodes and links in the cluster system, simulate the local damage caused by extreme events to the cluster system, and generate the damage adjacency matrix and node state vector; 2) Using a deep reinforcement learning algorithm based on the Actor-Critic architecture, we design and train a policy-value neural network (AC-Net) to predict the policy function and value function of maintenance actions under the current state of the cluster system. 3) Based on the Monte Carlo tree search algorithm and combined with the output of the policy-value neural network, the search and optimization of the corrective maintenance actions of multiple maintenance teams are performed to generate the optimal repair path and timing for each maintenance team; 4) During the repair process, dynamically adjust the maintenance strategies of each maintenance team to minimize the elasticity loss of system recovery and improve the recovery speed of the overall system performance based on the real-time recovery status and network characteristics of the cluster system; The corrective maintenance decision method needs to consider the spatial location of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, and comprehensively analyze the two-level decisions of node link maintenance sequence and multi-team path planning, including: a deep reinforcement learning Actor-Critic architecture that considers the two-level decisions of multi-team maintenance sequence and path planning, and an AC-MCTS algorithm for corrective maintenance of cluster systems; The deep reinforcement learning Actor-Critic architecture includes four parts: multi-maintenance team maintenance process, multi-maintenance team maintenance sequence, elasticity analysis and rewards, and neural network training; a) The maintenance process of multiple maintenance teams is as follows: The maintenance action sequence set is as follows: The action sequence set of maintenance team c is described as follows: Corrective maintenance actions are defined as , which means that the maintenance team c repairs the link between node i and node j; after repairing the link, if node j is a damaged node, the maintenance team continues to repair the node, otherwise it continues to perform the next maintenance action; at each time t, the restorative maintenance team uses the search algorithm integrated with AC-Net to select a maintenance action for the maintenance team c that has just completed the last maintenance action ; The corrective maintenance team's decisions are described below: in The current maintenance status. represents the status of repairs performed by the maintenance team, and The status of maintenance performed on behalf of maintenance team c; b) The maintenance sequence of multiple maintenance teams is as follows: Maintenance Team c The timing description is as follows: in Indicates that the maintenance team c Complete its maintenance tasks time, The calculation method of is as follows: in Indicates execution of an action The maintenance time consists of four situations: In the first case, the maintenance team is at the node i The location of the node j Normal; in the second case, the maintenance team is at the node i The location of the node j in a state of destruction; in the third case, the repair team is at a node i ' position, and the node j Normal; in the fourth case, the maintenance team is at a node i ' position, and the node j in a state of destruction; The calculation method is as follows: The time to repair each edge is expressed as follows: ， in v edge Indicates the speed at which broken links are repaired; from i Move to j The time is expressed as follows: ， in v move Indicates the movement speed of the maintenance team; The time series of the security cluster system is described as follows: In the formula , Complete actions on behalf of the maintenance team Maintenance time; c) Elasticity analysis and rewards are as follows: During the maintenance process of multiple maintenance teams, , if the node j is normal, then repair the link ( i , j ) after which the link is assumed to be operational again; if the node j is damaged, then repair the link ( i , j ) and nodes j Afterwards, assuming that the link and the node are back up and running; the cluster performance of the cluster system during the maintenance process of multiple maintenance teams Calculated according to the following method: The cluster efficiency FOM is selected as the quantitative performance indicator, and the average efficiency of the cluster system is E ( G ) is normalized as follows to define time FOM on: in is the average efficiency before the local attack event, yes Average efficiency at each moment; When the maintenance teams perform their respective maintenance actions, the cluster system Each time a maintenance team completes a maintenance action, the repaired links and nodes resume operation, and the cluster system Consider the step-up. is a step function; after completing the multi-repair team maintenance process, the following method is used to Integrate over the duration of the multi-repair team repair process to calculate the elasticity loss of the entire repair process RL : Assume that the local attack event occurs at time , and the cluster's functional performance drops sharply; then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and Reaching FOM threshold FOM thr , using the following elastic loss RL To quantify the loss suffered by the cluster due to a local attack: ； The repair sequence of broken links and broken nodes in the cluster, as well as the path planning of each repair team, will affect the step function The performance recovery curve, which in turn affects the elasticity loss RL The calculation results; the optimized maintenance strategy needs to obtain smaller elastic loss RL , then the reward of the maintenance strategy executed by the entire multi-maintenance team maintenance process can be defined as the elastic loss RL The inverse proportional function of the inverse proportional function is the integral of the cluster FOM without elastic loss in the specified time interval, as shown below: d) Neural network training For every moment t , the corrective maintenance team uses AC-Net A variant of the MCTS algorithm Provides search guidance, including p is the prior repair probability matrix, which is used to guide the tree search to select corrective repair actions, and v is a normalization parameter used to evaluate the return of the current corrective repair action selection; the search algorithm uses the parameter p and v To output the search probability π , to select high-quality repair actions ; Every moment t The AC-Net training data is stored as , in It's the action The reward; a and β are empirical parameters, ; When the corrective repair team completes the multi-repair team repair process, a round is also completed, and the data ( S , A , π , z ) are sampled from all time steps of the multi-maintenance team maintenance process; AC-Net then uses the empirical data ( S , A , π , z ) to adjust its parameters θ ; The training goal is to maximize the prior repair probability p With tree search probability π similarity and minimize the prior value v and rewards z The error between ; by using gradient descent to adjust the parameters θ , the loss function includes cross entropy loss and mean square error, as shown below: where c L2 is a constant used to control the L2 regularization level to prevent overfitting; For each round, a small batch of training samples is randomly selected from the dataset consisting of all event data, and batch training is used to adjust the parameters θ ; Adjust parameters θ After that, AC-Net improves the decision-making ability of the corrective maintenance team in the next round; in addition, the initialized AC-Net parameters θ is randomly generated; as the number of rounds increases, the trained AC-Net can provide stronger search guidance for tree search; after enough rounds, AC-Net has been well trained, and the cross entropy loss and mean square error tend to be stable; the restorative maintenance team provides effective decision support for the cluster system to solve the restorative maintenance decision problem under large-scale local damage; The AC-MCTS algorithm for cluster system corrective maintenance generates each maintenance action for the corrective maintenance decision process of multiple maintenance teams, that is, any t At this moment, the cluster system feature tensor X t Under the condition of AC-Net estimation of value function and policy function, the maintenance action search process is performed based on AC-MCTS, including selection, expansion and evaluation, backtracking and execution. Finally, the maintenance action search probability can be obtained. π and the best maintenance action; where the AC-MCTS algorithm is as follows: In the corrective maintenance decision execution t At this moment, for the decision feature tensor X t , taking the prior parameters obtained by the AC-Net prediction model as input, the maintenance action search operation is performed based on the MCTS method, and finally the improved maintenance state transition probability can be obtained π and best maintenance actions a t * , whose expression is: MCTS search tree each tree node X All branches of X , a ), corresponding to all the next maintenance actions of this tree node a ∊ Action ( X ), each branch ( X , a ) stores a set of statistics as follows: in N ( X , a ) indicates the number of visits; W ( X , a ) represents the sum of total action values; Q ( X , a ) represents the average action value; P ( X , a ) indicates selecting a branch ( X , a )’s prior probability.

2. The method according to claim 1, characterized in that The complex network model includes: Construct a network topology diagram for the nodes of the cluster system and the link relationships between them, where the nodes represent the key units in the cluster system and the links represent the communication or energy transmission paths between the nodes; By simulating extreme events, the damage center and damage radius are randomly generated, the damage conditions of nodes and links in the damaged area are determined, and the damage adjacency matrix is ​​formed.

3. The method according to claim 2, characterized in that Before the local attack, the cluster system has an adjacency matrix A , local damage in time Randomly initialized to the damage center and damage radius; then generate the damage adjacency matrix A d and the node vector S d nodes ; When the node n i When destroyed d nodes =0, otherwise S d nodes =1.

4. The method according to claim 1, characterized in that: The policy-value neural network AC-Net includes: An input layer for processing node and link state information; A deep residual network module to extract deep features of the current state of the cluster system; A policy output module to generate a priori probability matrix for the next maintenance action; A value output module is used to evaluate the effectiveness of the repair strategy under the current state.

5. The method according to claim 1, characterized in that The Monte Carlo tree search algorithm includes: Under the current cluster system state, the search process of the maintenance action is performed through Monte Carlo tree search based on the prior parameters output by the policy-value neural network; The upper confidence interval strategy is used to balance the exploration and exploitation of maintenance actions during the tree search process to generate a globally optimal sequence of maintenance strategies.

6. The method according to claim 1, characterized in that Cluster post-disaster repair under extreme events is regarded as a corrective maintenance decision problem for multiple maintenance teams. In order to achieve optimal recovery, the corrective maintenance decision problem aims to minimize the recovery time under the condition of limited recovery time and certain recovery level. RL, The model of the corrective maintenance decision problem for multiple maintenance teams is expressed as follows: (1.1) (1.2) (1.3) (1.4) (1.5) (1.6)； In constraint (1.2), c is the maintenance team number, Is n cre A maintenance cluster system consisting of 3 maintenance teams. t c It's the maintenance team c The maintenance time step is is the time step set in its own maintenance process, Indicates the time step t c Maintenance actions, Ω c It's the maintenance team c A set of action sequences; In constraint (1.3), t is the maintenance time step of the maintenance team and is the set of maintenance team time steps, Indicates the time step t Maintenance actions, Ω It is a set of maintenance team action sequences; Constraint (1.4) represents the set of action sequences that guarantee the cluster system Ω and each maintenance team action sequence set Ω the relationship between; In constraint (1.5), Indicates execution of an action The time interval is the recovery time threshold; Constraint (1.5) forces the total repair time of the maintenance team to be below the threshold recovery time, while constraint (1.6) forces the cluster performance to be within the time Reached FOM threshold.

7. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method described in any one of claims 1 to 6 is implemented.

Images