A method for post-disaster restorative maintenance of cluster systems based on AC-MCTS algorithm

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a method for post-disaster restorative maintenance of cluster systems based on the AC-MCTS algorithm. The present invention improves the reliability and maintainability of cluster systems by real-time prediction of system status and dynamic selection of optimal maintenance strategies. The system can not only effectively solve various faults in cluster systems, but also improve the intelligence level of maintenance strategies, reduce system downtime, and improve the overall operating efficiency of the system through continuous learning and optimization.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for post-disaster restorative maintenance of a cluster system based on an AC-MCTS algorithm. Background Art

[0002] With the rapid development of modern society, cluster systems have been widely used in large-scale distributed infrastructures such as power systems, communication networks, and transportation networks. These cluster systems are usually composed of multiple key nodes and links to ensure the stable operation and efficient service of the system. However, natural disasters such as earthquakes and typhoons may cause serious damage to some nodes and links in the cluster system, significantly reducing the overall performance of the system, or even causing partial or complete failure of the system.

[0003] Post-disaster restorative maintenance is a key issue in cluster system operation and maintenance, especially how to quickly restore the normal operation of the system after a disaster. Traditional post-disaster repair strategies often rely on manual experience or simple rule algorithms, which show obvious shortcomings in dealing with the complexity and large-scale distributed characteristics of modern cluster systems. Specifically, existing methods are difficult to handle a large number of damaged nodes and links within a limited time, and lack effective optimization methods to coordinate the collaborative work of multiple maintenance teams, resulting in a time-consuming and inefficient repair process, and the system's recovery speed and effect are difficult to be satisfactory.

[0004] With the widespread application of cluster systems, the reliability and maintainability of cluster systems have attracted more and more attention. Cluster systems are usually composed of multiple nodes, which may cause hardware failures or software errors due to various factors during long-term operation, thus affecting the stability and performance of the entire system. In order to ensure the normal operation of the system, timely and effective corrective maintenance of the cluster system is required.

[0005] Traditional cluster system maintenance methods mainly rely on manual experience and preset maintenance strategies, which have great limitations: first, manual experience is easily limited by personal knowledge and judgment ability, and it is difficult to comprehensively and accurately handle various types of faults in complex systems; second, preset maintenance strategies are usually based on fixed rules, lack flexibility and adaptability, and are difficult to cope with the ever-changing fault conditions in the system.

[0006] In recent years, with the development of artificial intelligence technology, maintenance strategies based on deep learning and reinforcement learning have gradually become a research hotspot. Among them, the Monte Carlo Tree Search (MCTS) algorithm has become a potential tool for solving cluster system maintenance problems due to its excellent performance in complex decision-making problems. The MCTS algorithm effectively improves the quality and efficiency of decision-making by simulating possible future states and selecting the optimal action path. However, the traditional MCTS algorithm still has problems of low search efficiency and poor adaptability when facing complex and dynamic cluster systems. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a method for post-disaster restorative maintenance of cluster systems based on the AC-MCTS algorithm, which aims to combine the advantages of deep learning and the MCTS algorithm, and improve the reliability and maintainability of the cluster system through real-time prediction of system status and dynamic selection of the optimal maintenance strategy. The system can not only effectively solve various faults in the cluster system, but also improve the intelligence level of the maintenance strategy through continuous learning and optimization, reduce system downtime, and improve the overall operation efficiency of the system.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0009] A method for post-disaster restorative maintenance of a cluster system based on an AC-MCTS algorithm, the method comprising the following steps:

[0010] a) Construct a multi-repair team post-disaster corrective maintenance (CMDP) process and generate a decision feature tensor of the cluster system at any time point t X t ;

[0011] b) Using Actor-Critic Neural Network (AC-Net) X t Evaluate and output the prior parameters, which are used as input to the Monte Carlo Tree Search (MCTS) algorithm.

[0012] c) Based on the MCTS algorithm X t The search operation of the maintenance action is performed under the conditions, including:

[0013] i) Select: X t As the root node of MCTS, it selects the maintenance action with the best action value based on the upper confidence interval (UCB) algorithm;

[0014] ii) Expansion and evaluation: Add leaf nodes in the search tree to the queue, evaluate their strategic value using the ResNet neural network, and update the statistics of related nodes in the search tree;

[0015] iii) Backtracking: Based on the evaluation results, backtrack along the search path and update the number of visits and action value of each branch on the search path;

[0016] iv) Execution: By iterating the above steps, the improved maintenance state transition probability π is obtained and the global optimal maintenance action is selected a t * , transfer the cluster system from time point t to t+1;

[0017] d) Repeat steps a) to c) until the corrective maintenance task of the cluster system is completed.

[0018] Preferably, the CMDP is executed until t At this moment, for the decision feature tensor X t , using the prior parameters obtained by the AC-Net prediction model ( P t , V t ) as input, and the maintenance action search operation is performed based on the MCTS method, and finally the improved maintenance state transition probability can be obtained. π and best maintenance actions a t * , whose expression is:

[0019]

[0020] MCTS search tree each tree node X All branches of X , a ), corresponding to all the next maintenance actions of this tree node a ∈ Action ( X ), each branch ( X , a ) stores a set of statistics as follows:

[0021]

[0022] in N ( X , a ) indicates the number of visits; W ( X , a ) represents the sum of total action values; Q (X , a ) represents the average action value; P ( X , a ) indicates selecting a branch ( X , a )’s prior probability.

[0023] As a preferred method, based on the MCTS algorithm X t The search operation of the maintenance action is performed under the condition of:

[0024] i) Select:

[0025] First, t The decision feature tensor X of the moment cluster system t As the root node X of the Monte Carlo tree 0 , that is, the cluster system status at each moment, and from the root node X 0 Start searching until L Time reaches the leaf node X at the end of the search tree L Ends at l time, l < L , based on the upper confidence interval tree search algorithm (UCT) and the maximum upper confidence interval (UCB) algorithm, the best action a under the current iteration conditions is selected according to the statistical data stored in each branch of the search tree l , whose expression is:

[0026] ;

[0027] where c puct is a constant determined by the degree of MCTS search; a∈Action(X), Action(X) represents the action space;

[0028] ii) Extension and Evaluation

[0029] The leaf node X L Add to the queue as the input of ResNet, and then the leaf node X L To expand, each edge of the leaf node expansion (X L ,a) Initialize storage data:

[0030]

[0031] All positions in the queue are evaluated by the neural network, resulting in leaf nodes X L The strategic value of v , the search thread is locked until the strategy evaluation is completed;

[0032] iii) Backtracking

[0033] After expansion and evaluation, the leaf node X L The strategic value of v Backtrack along the original path to the parent node; during the backtracking process, each branch (X l ,a l ) The access count parameter stored in the memory is updated once, and its expression is:

[0034]

[0035] At the same time, search for the branch (X l ,a l ) maintenance action a l The total action value and average action value parameters are also updated once, and their expressions are:

[0036] ;

[0037] iv) Execution

[0038] By iterating the above three steps 1000 times, the maintenance status probability is obtained π , and select the feature tensor of the cluster system at the current moment X t Global optimal maintenance action under the condition a t * ; Among them, the improved cluster system maintenance status probability π It is exponentially related to the number of visits, and its expression is:

[0039]

[0040] Where τ is an empirical parameter that controls the search process;

[0041] Execute optimal maintenance actions a t * Afterwards, the feature tensor of the cluster system maintenance status is transformed from t Time X t Transfer to t+ 1 Moment X t+1 , the search tree will be X t+1 The new root node X 0 , and keep the node X in the original search tree t+1 All child nodes below it and their statistics, while discarding the branches of the remaining nodes at the same level as this node.

[0042] Preferably, the Actor-Critic neural network (AC-Net) is a dual-output residual neural network, including a strategy module and a value module; wherein:

[0043] a) The strategy module outputs the prior repair probability matrix through multiple convolutional layers and fully connected layers P t , the elements of this matrix represent the probability of selecting a repair action;

[0044] b) The value module outputs a scalar value V through the residual network structure t , used to evaluate the selection reward of the current corrective repair action.

[0045] Preferably, the Actor-Critic neural network (AC-Net) f θ ( S, A ) = ( p , v ) is a dual-output residual neural network (ResNet), which is used as a maintenance strategy prediction model. The network parameters are randomly initialized. θ 0;

[0046] At every moment t , AC-Net f θ The input is the current repair state S before damage t =[ S f t , S c tc ] and the adjacency matrix A ; S f t It can be written as follows:

[0047]

[0048] Among them A f t is the time step of MCRP t The adjacency matrix at , if , then A f t = A f 1 =A damage Damage; at the same time, S c tc Written as follows:

[0049]

[0050] Among them A c tc It's the maintenance teamc The time step t of the repair process c If the adjacency matrix of , then A c tc = A 1 c =A d ;

[0051] The output of the Actor module and the Critic module is the prior repair probability matrix P t Sum value V t ; Matrix P t is a N × N The adjacency matrix, whose element P ij Yes Select Repair Action a = ( i , j ) probability; value V t is a standardized parameter used to evaluate the reward of the current corrective repair action selection;

[0052] The AC-Net in the restorative repair agent uses a residual module consisting of residual blocks of convolutional layers with ReLU functions to process the input feature S t , and the input features A is a constant matrix, which is processed by a multi-layer module composed of fully connected layers; the residual module and the multi-layer module include shared layers and are merged into two independent modules, namely policy and value; in addition, the policy and value modules take their inputs from the shared layer and output the prior repair probability P using a fully connected layer and a sigmoid function. t Sum value V t .

[0053] Preferably, the selection basis of the upper confidence interval (UCB) algorithm includes:

[0054] a) maximize the upper confidence interval;

[0055] b) Select maintenance actions with lower visit counts and higher prior probabilities.

[0056] Preferably, the ResNet neural network structure comprises:

[0057] a) A convolutional module consisting of 256 3×3 filters;

[0058] b) a deeply stacked residual module, wherein the residual module includes a structure consisting of a plurality of convolutional layers, batch normalization layers, ReLU function layers, and Add modules;

[0059] c) Strategy module and value module, which are used to output the prior maintenance probability matrix p and the standardized value v respectively.

[0060] As a preference, the expression of the ResNet prediction model is f θ ( X ) = ( p , v ), θ is the network parameter of ResNet, the initial network parameter θ 0 is obtained by random initialization and is optimized and updated as the algorithm iterates.

[0061] Furthermore, the present invention also discloses a system for implementing the method, the system comprising:

[0062] a) Cluster system feature extraction module, used to generate the decision feature tensor of the cluster system at any time point t X t ;

[0063] b) Actor-Critic neural network module, communicating with the cluster system feature extraction module, for X t Perform evaluation and output prior parameters;

[0064] c) a Monte Carlo tree search module, communicating with the Actor-Critic neural network module, for performing a Monte Carlo tree search based on the MCTS algorithm. X t The Monte Carlo tree search module is used to perform the search operation of the maintenance action under the conditions of:

[0065] i) Select the module to X t As the root node of MCTS, it selects the maintenance action with the best action value based on the upper confidence interval (UCB) algorithm;

[0066] ii) Expansion and evaluation module, which is used to add leaf nodes in the search tree to the queue, evaluate their strategic value using the ResNet neural network, and update the statistics of related nodes in the search tree;

[0067] iii) A backtracking module, which is used to backtrack along the search path according to the evaluation results and update the visit count and action value of each branch on the search path;

[0068] iv) Execution module, which is used to obtain the improved maintenance state transition probability π by iterating the above steps and select the global optimal maintenance action a t * , transfer the cluster system from time point t to t+1.

[0069] Furthermore, the present invention also discloses a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the method is implemented.

[0070] The present invention adopts the above technical solution to provide an AC-MCTS algorithm system for corrective maintenance of cluster systems, which combines the advantages of deep learning and Monte Carlo Tree Search (MCTS) algorithm, significantly improves the intelligence and efficiency of cluster system maintenance, and is specifically embodied in the following technical effects:

[0071] 1. Improve the accuracy and dynamic adaptability of maintenance decisions: By introducing the Actor-Critic neural network (AC-Net), the system can accurately evaluate the characteristic tensor Xt of the cluster system at every moment, predict the current state of the system and the optimal maintenance action. Compared with traditional maintenance methods that rely on fixed rules or manual experience, this system can dynamically adapt to the complex and changing operating environment of the cluster system to ensure the accuracy of maintenance decisions.

[0072] 2. Optimize the search efficiency of maintenance actions: This system uses the MCTS algorithm to perform maintenance action search operations based on the prior parameters output by AC-Net, combined with the upper confidence interval (UCB) algorithm, taking into account the breadth and depth of the search. Through iterative optimization and extended node selection strategies, the system can efficiently find the global optimal maintenance action, significantly reducing the computational complexity and time cost in the maintenance action selection process.

[0073] 3. Enhance the system's learning ability and the intelligence of maintenance strategies: The system evaluates maintenance strategies and values ​​through a deep residual network (ResNet), and continuously learns and optimizes maintenance strategies. As the system runs and data accumulates, AC-Net will gradually optimize its network parameters, further improve prediction accuracy and the effectiveness of maintenance strategies, and realize intelligent and adaptive optimization of maintenance strategies.

[0074] 4. Reduce system downtime and improve overall operating efficiency: Through efficient and accurate maintenance decisions, this system can quickly repair faults in the cluster system and reduce system downtime caused by faults. At the same time, by optimizing the selection of maintenance actions, this system can maximize the overall operating efficiency of the system without increasing hardware resources, ensuring the stability and continuous operation of the cluster system.

[0075] In summary, the present invention provides an intelligent and efficient corrective maintenance solution for cluster systems by combining deep learning with Monte Carlo tree search, which not only improves the accuracy and efficiency of maintenance decisions, but also significantly improves the overall operating performance and reliability of the cluster system. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Fig. 1 Complex network model of cluster system under large-scale destruction.

[0077] Fig. 2 Schematic diagram of the residual module structure.

[0078] Fig. 3. Construction diagram of complex network model of distributed power system cluster.

[0079] Figure 4 Case of large-scale localized damage to power system clusters in a regional power grid.

[0080] Figure 5 Diagram of the corrective maintenance process of multiple maintenance teams.

[0081] Fig. 6 AC-Net learning and training analysis diagram.

[0082] Figure 7 Resilience loss analysis plots for different approaches under various post-disaster damage scenarios.

[0083] Figure 8 shows the elasticity analysis of maintenance strategies for different maintenance team sizes.

[0084] Figure 9 is a maintenance strategy elasticity analysis diagram for maintenance teams with different business capabilities. DETAILED DESCRIPTION

[0085] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0086] Aiming at the post-disaster repair strategy problem of cluster system, considering the two-level decision of repair sequence and path planning, this paper studies the restorative maintenance decision method based on Actor-Critic Monte Carlo Tree Search (AC-MCTS). The designed restorative maintenance decision method needs to consider the spatial position of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, and comprehensively analyze the two-level decisions of node link repair sequence and multi-team path planning, which includes two core ideas: the deep reinforcement learning Actor-Critic architecture considering the two-level decision of multi-team repair sequence and path planning, and the AC-MCTS algorithm for restorative maintenance of cluster systems. For the two-level decision of the repair sequence of the cluster system and the path planning of the maintenance team, solving the global optimal decision under the condition of uneven benefits of multiple maintenance teams is the core feature that needs to be focused on. Therefore, it is necessary to focus on the restorative maintenance strategy considering multiple maintenance teams, and be able to plan the optimal path for multiple maintenance teams to repair damaged systems and components, as well as the optimal timing for each maintenance team to repair damaged systems and links between systems. In order to solve the two-level decision-making characteristics of the maintenance strategy between multiple maintenance teams and a single maintenance team in this problem, the AC-MCTS method designed by the present invention uses the damage information of cluster system nodes and links to determine the optimal maintenance actions and paths of multiple maintenance teams. In addition, the above-mentioned Actor-Critic architecture needs to design a strategy-value neural network to estimate the value function and the strategy function, and then perform the maintenance action search process based on AC-MCTS, and finally obtain the maintenance action search probability and the optimal maintenance action. Based on the proposed AC-MCTS restorative maintenance decision algorithm, the present invention conducts case verification in the case analysis part for post-disaster repair of multiple maintenance teams in the post-disaster scenario of distributed power system clusters. The case analysis results show that the proposed restorative maintenance decision algorithm can provide an efficient restorative maintenance solution in the post-disaster repair work of distributed power system clusters, and realizes the rapid recovery of power system cluster performance under large-scale damage.

[0087] 1.1 Description of cluster system post-disaster repair problems

[0088] 1.1.1 Large-scale destruction and repair of cluster systems

[0089] First, we analyze the cluster system and its large-scale destruction, consider the interconnection relationship between nodes in the distributed cluster system, and build its complex network model, such as Figure 1 shown.

[0090] (1) Destruction

[0091] Based on the cluster, the destruction center and destruction radius are randomly generated. The red area in the figure is the local destruction area. Assume that all nodes and links in the red area are invalid. When these links are removed from the graph, the nodes are isolated. Then, the state of the links between nodes can be described as the current adjacency matrix after the local attack. A damage ,The state vector of a node in a cluster system can be described as:

[0092] (1.1)

[0093] When node n i When destroyed i node =0, otherwise S i node =1.

[0094] (2) Repair

[0095] This paper will propose a rich and executable strategy to support maintenance activities, considering maintenance teams, maintenance time series and maintenance objects. The proposed strategy aims to make up for the limitations of other current studies, which mainly consider the repair object. The maintenance activities are performed in parallel by multiple maintenance teams. Repairing damaged nodes i The average time of MTTR i The time to repair each edge can be expressed as follows:

[0096] (1.2)

[0097] in v edge Indicates how quickly broken links are repaired. i Move to j The time can be expressed as follows:

[0098] (1.3)

[0099] in v move Indicates the movement speed of the repair team.

[0100] (3) Recovery

[0101] Elasticity is currently widely used to characterize the recovery performance of a cluster after a local attack. Therefore, the present invention introduces cluster elasticity to evaluate the effectiveness of the recovery strategy, with the goal of improving cluster performance and shortening recovery time. In order to describe the quantitative measurement of cluster elasticity, it is necessary to define a quantity that represents the functional performance of the cluster. The present invention selects cluster effectiveness (Figure of Merit, FOM) as a quantitative performance indicator. On this basis, the average efficiency of the cluster system isE ( G ) is normalized as follows to define the FOM over time τ:

[0102] (1.4)

[0103] Where E*(G) is the average efficiency before the local attack event occurs, E τ (G) is the average efficiency at time τ. In addition, other performance parameters of the power grid can also be used in the definition of FOM.

[0104] Assume that the local attack event occurs at time τ d , and the cluster functional performance drops sharply. Then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and at time τ d Reaching FOM threshold FOM the The following resilience indicators are used to quantify the loss suffered by the cluster due to local attacks:

[0105] (1.5)

[0106] 1.1.2 Corrective maintenance decision-making problem for post-disaster emergency repair

[0107] Cluster post-disaster repair is considered as a corrective maintenance decision (CMD) problem for multiple maintenance teams. The constraints are the recovery time and recovery level, and the optimization goal is the elasticity loss of the cluster system. RL , the basic decision variable is the restorative maintenance action. From the nature of the decision variables, this problem is integer programming using binary variables. For each maintenance team, considering the three actions related to post-disaster repair, namely, repairing nodes, repairing links, and moving, the number of strategies is 2 N+K ; For multiple maintenance teams, their strategy spaces are coupled to each other, so this problem is an NP-hard problem. In order to achieve optimal recovery, the CMD problem aims to minimize under limited recovery time and a certain recovery level. RL Therefore, the CMD problem for multiple maintenance teams can be expressed as follows:

[0108] In the constraint (1.62), c is the maintenance team number, F={1,2,…, n cre} is composed of n cre A maintenance team consisting of a support system cluster, t c It's the maintenance team c The maintenance time step is Tc ={1,2,…, T c} is the time step set in its own maintenance process, a * tc represents the time step t c Maintenance action, Ω c It's the maintenance team c In the constraint (1.63), t is the maintenance time step of the maintenance team and T ={1,2,…, T c} is the maintenance team time step set, a * t Indicates the time step t The maintenance action is, Ω is the maintenance team action sequence set. Constraint (1.64) represents the maintenance system cluster action sequence set Ω and each maintenance team action sequence set Ω c In the constraint condition (1.65), τ( a * tc ) indicates execution of an action a * tc The time interval, τ thr is the recovery time threshold. Constraint (1.65) forces the total repair time of the maintenance team to be lower than the threshold recovery time, while constraint (1.66) forces the cluster performance to be within time τ r Reached FOM threshold.

[0109] 1.2 Corrective maintenance decision-making algorithm considering timing and path planning

[0110] The corrective maintenance decision-making for post-disaster repair of cluster systems needs to consider the spatial position of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, comprehensively analyze the node link maintenance timing and multi-team path planning, and design a corrective maintenance decision-making method based on the deep reinforcement learning Actor-Critic architecture. Considering the two-level decision-making of multi-team maintenance timing and path planning, it supports corrective maintenance work such as the generation of corrective maintenance plans for cluster systems with network characteristics. This method contains two core ideas: a deep Actor-Critic architecture that considers the two-level decision-making of multi-team maintenance timing and path planning, and an AC-MCTS algorithm for corrective maintenance of cluster systems.

[0111] 1.2.1 Actor-Critic architecture considering two-level decision-making of multi-team maintenance timing and path planning

[0112] Regarding the research content of corrective maintenance considering timing and path planning, especially for the corrective maintenance work of network cluster systems such as infrastructure networks, two basic challenging problems are the two-level decisions of the repair timing and path planning of the network cluster system. Therefore, the present invention focuses on the corrective maintenance strategy considering multiple maintenance teams, which can plan the optimal path for multiple maintenance teams to repair damaged systems and components, as well as the optimal timing for each maintenance team to repair damaged systems and inter-system links. In order to solve the two-level decision-making characteristics of the maintenance strategy between multiple maintenance teams and a single maintenance team in this problem, the present invention intends to design a deep Actor-Critic architecture that considers the two-level decision-making of multi-team maintenance timing and path planning. The idea of ​​this decision-making architecture uses the damage information of the network cluster system to determine the optimal maintenance actions and paths of multiple maintenance teams.

[0113] The main idea of ​​the deep Actor-Critic architecture considering two-level decisions of multi-team maintenance timing and path planning is to iteratively execute the multi-repair team CMDP process in the policy iteration process. First, the simulation environment generates local damage and network cluster system status. Then, the restorative maintenance agent executes the multi-repair team CMDP process and generates a series of maintenance actions based on the AC-MCTS algorithm. When generating each maintenance action, the restorative maintenance agent uses AC-Net to extract the current recovery state of the network cluster system and the multi-repair team timing characteristics, and estimates the value function and policy function of the maintenance action to support the AC-MCTS algorithm to search for maintenance actions. Subsequently, the simulation environment rewards the maintenance strategy based on evaluation methods such as elasticity analysis. The maintenance strategy consists of a series of maintenance actions generated by the CMDP process. Finally, the AC-Net parameters are adjusted based on the training sample data set; the new AC-Net parameters are used in the next policy iteration process to provide MCTS with more effective maintenance action search support. For the CMD problem, the deep reinforcement learning framework proposed in the present invention. The framework shows four parts: multiple crews repair process (MCRP), multi-repair team repair timing, elasticity analysis and reward, and neural network training, as described below.

[0114] (1) Maintenance process of multiple maintenance teams

[0115] Before the local attack, the cluster system has an adjacency matrix A Then, the local damage occurs at time τ d The damage center and damage radius are randomly initialized. Then the damage adjacency matrix is ​​generated A d and the node vector S d nodes . FOM(τ d) can be calculated according to formula (1.4).

[0116] To restore cluster performance, a cre The maintenance team's support system cluster performs a series of maintenance actions during the MCRP period. This maintenance action sequence set is as follows:

[0117] (1.7)

[0118] Maintenance Team c The action sequence set can be described as follows:

[0119] (1.8)

[0120] Corrective maintenance actions are defined as a * t∣tc = a * tc∣t =( i , j ), which means that the maintenance team c Repair Node i With Node j After repairing the link, if the node j If the node is damaged, the maintenance team will continue to repair the node, otherwise continue to perform the next maintenance action. t The corrective maintenance agent uses the integrated AC-Net search algorithm to find the maintenance team that has just completed the last maintenance action. c Select maintenance action a * t∣tc .

[0121] The decision of the corrective maintenance agent can be described as follows:

[0122] (1.9)

[0123] in S=[S f ,S c ] The current maintenance status. S f represents the status of repairs performed by the maintenance team, and S c Representative maintenance team c The status of the repairs performed.

[0124] (2) Time sequence of multiple maintenance teams

[0125] For havingn cre The maintenance team's support system cluster has a time sequence and n cre The timing of each maintenance team.

[0126] Maintenance Team c The timing can be described as follows:

[0127] (1.10)

[0128] where τ tc Indicates that the maintenance team c Complete its maintenance tasks a * tc∣t =( i , j ) time, τ tc The calculation method of is as follows:

[0129] (1.11)

[0130] Where τ( a * tc∣t ) indicates execution of an action a * tc∣t The maintenance time consists of four situations. In the first situation, the maintenance team is at the node i The location of the node j Normal. In the second case, the maintenance team at the node i The location of the node j In the third case, the maintenance team is at the node i ' position, and the node j Normal. In the fourth case, the maintenance team is at the node i ' position, and the node j is in a damaged state. According to (1.2) and (1.3), τ( a * tc∣t ) is calculated as follows:

[0131] (1.12)

[0132] The time series of the security system cluster can be described as follows:

[0133] (1.13)

[0134] Where τ t =0=τd , τ t Complete actions on behalf of the maintenance team a * t∣tc = a * tc∣t =( i , j ) maintenance time.

[0135] (3) Elasticity analysis and rewards

[0136] In the above MCRP process, for maintenance actions a * t∣tc = a * tc∣t =( i , j ), if the node j is normal, then repair the link ( i , j ) after which the link is assumed to be operational again; if the node j is damaged, then repair the link ( i , j ) and nodes j Afterwards, it is assumed that the link and the node are back up and running. τt ) can be calculated according to formula (1.4). When the maintenance teams perform their respective maintenance actions, the FOM of the cluster system ( τt ) can be considered unchanged; whenever a maintenance team completes a maintenance action, the repaired links and nodes resume operation, and the FOM ( τt ) can be considered as a step improvement. Therefore, FOM( τt ) is a step function. After completing MCRP, the FOM( τt ) Integrate over the duration of MCRP to calculate the elastic loss of the entire repair process RL .

[0137] The repair sequence of damaged links and damaged nodes in the cluster, as well as the path planning of each maintenance team, will affect the step function FOM ( τt ) performance recovery curve, thus affecting elasticity loss RL Therefore, the optimized maintenance strategy needs to obtain a smaller elastic loss. RL , then the reward of the maintenance strategy executed by the entire MCRP process can be defined as the elastic loss RLThe inverse proportional function of the inverse proportional function is the integral of the cluster FOM without elastic loss in the specified time interval, as shown below:

[0138] (1.14)

[0139] (4) Actor-critic network training

[0140] For every moment t , Corrective Maintenance Agent uses AC-Net f θ ( S, A ) = ( p , v ) is a variant of the MCTS algorithm f ts ( S,p,v ) = ( π , a * ) provides search guidance, where p is the prior repair probability matrix, which is used to guide the tree search to select corrective repair actions, and v is a normalization parameter used to evaluate the reward of the current corrective repair action selection. The search algorithm uses the parameter p and v To output the search probability π , to select high-quality repair actions a * .

[0141] Every moment t The AC-Net training data is stored as (S t ,A, π t ,Z t ),in It's action a * t∣tc α and β are empirical parameters. When the corrective maintenance agent completes the MCRP, an episode is completed and the data ( S , A , π , z ) is sampled from all time steps of this MCRP. AC-Net then uses the empirical data ( S , A , π , z ) to adjust its parameters θ The training goal is to maximize the prior repair probability pWith tree search probability π similarity and minimize the prior value v and rewards z Therefore, by using gradient descent to adjust the parameters θ , the loss function includes cross-entropy losses (CELs) and mean squared errors (MSEs), as shown below:

[0142] (1.15)

[0143] in c L2 is a constant used to control the level of L2 regularization to prevent overfitting.

[0144] For each episode, a small batch of training samples is randomly selected from the dataset consisting of all episode data, and then batch training is used to adjust the parameters θ . Adjust the parameters θ After that, AC-Net can improve the decision-making ability of the corrective maintenance agent in the next episode. In addition, the initialized AC-Net parameters θ is randomly generated. As the number of episodes increases, the trained AC-Net can provide stronger search guidance for tree search. After enough episodes, AC-Net has been well trained, and MSE and CEL tend to be stable. The restorative maintenance agent can provide effective decision support for cluster systems to solve CMD problems under large-scale local damage. In addition, when generating maintenance strategies for large-scale local damage during the operation phase of the cluster system, there is no need to train AC-Net again.

[0145] 1.2.2 AC-MCTS algorithm for corrective maintenance of cluster systems

[0146] When the multi-maintenance team CMDP process generates each maintenance action, that is, any maintenance action of the CMDP process t At this moment, the cluster system feature tensor X t Under the condition of, the result of estimating the value function and strategy function by AC-Net ( P t , V t ) as input, and the maintenance action search process is performed based on AC-MCTS, which mainly includes four steps: selection, expansion and evaluation, backtracking, and execution. Finally, the maintenance action search probability can be obtained. π and best maintenance actions .

[0147] 1.2.2.1 AC-MCTS Algorithm

[0148] In CMDP implementation t At this moment, for the decision feature tensor X t , using the prior parameters obtained by the AC-Net prediction model ( P t , V t ) as input, and the maintenance action search operation is performed based on the MCTS method, and finally the improved maintenance state transition probability can be obtained. π and best maintenance actions a t * , whose expression is:

[0149] (1.16)

[0150] MCTS search tree each tree node X All branches of X , a ), corresponding to all the next maintenance actions of this tree node a ∈ Action ( X ), each branch ( X , a ) stores a set of statistics as follows:

[0151] (1.17)

[0152] in N ( X , a ) indicates the number of visits; W ( X , a ) represents the sum of total action values; Q ( X , a ) represents the average action value; P ( X , a ) indicates selecting a branch ( X , a )’s prior probability.

[0153] When the multi-maintenance team CMDP process generates each maintenance action, that is, at any time in the CMDP process t At this moment, the AC-MCTS algorithm performs the following four steps.

[0154] 1) Select

[0155] First, t Characteristic tensor of the moment cluster system X t As the root node of the Monte Carlo tree X 0 (i.e. the cluster system status at each moment), and from the root node X 0 Start searching until L The leaf node at the end of the search tree is reached at any time X L Ends at l time( l < L ), based on the Upper Confidence Bound Apply to Tree (UCT) and the Maximum Upper Confidence Bound (UCB) algorithm, the best action a is selected under the current iteration conditions according to the statistical data stored in each branch of the search tree. l , whose expression is:

[0156] (1.18)

[0157] (1.19)

[0158] where c puct is a constant determined by the degree of MCTS search; a∈Action(X), Action(X) represents the action space;

[0159] In the early stage, this search strategy tends to select maintenance actions that usually have higher prior probabilities and lower visit times, but as the search proceeds, it tends to select "unexpanded" nodes, so that the MCTS search takes into account both breadth and depth.

[0160] 2) Extension and evaluation

[0161] The leaf node X L Add to the queue as the input of ResNet, and then the leaf node X L Expand, each edge (X L ,a) Initialize storage data:

[0162] (1.20)

[0163] All positions in the queue are evaluated by the neural network and the leaf nodes are obtained. X LThe strategic value of v , the search thread is locked until the policy evaluation is completed.

[0164] 3) Backtracking

[0165] After expansion and evaluation, the leaf nodes X L The strategic value of v Backtrack along the original path to the parent node. During the backtracking process, each branch (X l ,a l ) The access count parameter stored in the memory is updated once, and its expression is:

[0166] (1.21)

[0167] At the same time, search for the branch (X l ,a l ) maintenance action a l The total action value and average action value parameters are also updated once, and their expressions are:

[0168] (1.22)

[0169] (1.23)

[0170] 4) Execution

[0171] By iterating the above three steps 1000 times, the maintenance status probability is obtained π , and select the feature tensor of the cluster system at the current moment X t Global optimal maintenance action under the condition a * t Among them, the improved cluster system maintenance status probability π It is exponentially related to the number of visits, and its expression is:

[0172] (1.24)

[0173] where τ is an empirical parameter that controls the search process.

[0174] Execute optimal maintenance actions a * t Afterwards, the feature tensor of the cluster system maintenance status is transformed from t time X t Transfer to t+ 1 moment X t+1 , the search tree will be Xt+1 The new root node X 0 , and keep the nodes in the original search tree X t+1 All child nodes below it and their statistics, while discarding the branches of the remaining nodes at the same level as this node.

[0175] In summary, the AC-MCTS search algorithm can be expressed as f ts ( S,p,v ) = ( π , a * ).

[0176] 1.2.2.2 Actor-Critic Neural Network Prediction Model

[0177] AC-Net f θ ( S, A ) = ( p , v ) is a dual-output residual neural network (ResNet), which is used as a maintenance strategy prediction model. The network parameters are randomly initialized. θ 0.

[0178] At every moment t , AC-Net f θ The input is the current repair state before damage S t = [ S f t , S c tc ] and the adjacency matrix A . S f t It can be written as follows:

[0179] (1.25)

[0180] Among them A f t is the time step of MCRP t The adjacency matrix at , if , then A f t = A f 1 =A damageDamage. The historical adjacency matrix is ​​necessary because the restorative maintenance agent cannot fully observe all state information from the current adjacency matrix, because MCRP is a sequential decision process. At the same time, S c tc It can be written as follows:

[0181] (1.26)

[0182] Among them A c tc It's the maintenance team c Time step of the repair process t c If the adjacency matrix of , then A c tc = A 1 c =A d .

[0183] The output of the Actor module and the Critic module is the prior repair probability matrix P t Sum V t .matrix P t is a N × N The adjacency matrix, whose elements P ij Yes Select Repair Action a = ( i , j ) probability. V t is a standardized parameter used to evaluate the reward of the current corrective repair action selection.

[0184] This AC-Net in the restorative repair agent uses a residual module consisting of residual blocks of convolutional layers with ReLU functions to process input features. S t , and the input features A is a constant matrix, which is processed by a multi-layer module consisting of fully connected layers. The residual module and the multi-layer module include shared layers and are merged into two independent modules, namely policy and value. In addition, the policy and value modules take their inputs from the shared layers and output the prior repair probability using fully connected layers and sigmoid functions. P t Sum V t .

[0185] The ResNet network structure includes convolution module, residual module, batch normalization module, ReLU function module, etc. Input feature tensor X It is processed by a deeply stacked module, which consists of a single convolution module and an intermediate layer module that deeply stacks multiple residual modules:

[0186] a) Separate convolutional modules:

[0187] ① A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0188] ② ReLU function layer;

[0189] b) Intermediate layer module: The intermediate layer of the deep neural network is constructed by stacking residual modules (taking 9 layers of stacked residual modules as an example). The structure of the residual module is as follows: Figure 2 As shown, it includes the following structures:

[0190] ① A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0191] ② Batch normalization layer;

[0192] ③ ReLU function layer;

[0193] ④ A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0194] ⑤ Batch normalization layer;

[0195] ⑥ Addition module: Add the input feature tensor of the residual module to the output feature tensor of the previous layer;

[0196] ⑦ ReLU function layer;

[0197] The output of the deeply stacked residual module is divided into two parts: the policy module and the value module:

[0198] c) Policy Module:

[0199] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0200] ② ReLU function layer;

[0201] ③ Fully connected output layer: outputs a feature vector of size m×n, corresponding to the logit of the prior maintenance state transition probability p of the cluster;

[0202] d) Value Module:

[0203] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0204] ② A linear fully connected layer with a size of 256;

[0205] ③ ReLU function layer;

[0206] ④ Linear fully connected layer;

[0207] ⑤ Sigmoid output layer: outputs a scalar value in the interval (0,1).

[0208] The expression of the above ResNet prediction model is f θ ( X ) = ( p , v ), θ is the network parameter of ResNet, the initial network parameter θ 0 is obtained by random initialization.

[0209] 1.3 Analysis of post-disaster emergency repair cases

[0210] In the case analysis part, the present invention selected a large-scale distributed power system cluster for case study, and took the power system cluster in a certain regional power grid as the case object, and analyzed the performance of the power system cluster in a certain regional power grid under disasters through Python simulation, and generated a corrective maintenance strategy based on the proposed AC-MCTS algorithm, and then analyzed the recovery effect. In addition, the power system cluster in the regional power grid consists of 228 nodes and 612 links. For each maintenance team, considering the three types of actions related to post-disaster repair, namely repairing nodes, repairing links, and moving, based on the basic assumptions of the corrective maintenance decision problem, it can be known that the number of strategies is 2 228+612 =7.33×10 252 ; For multiple maintenance teams, their strategy spaces are coupled with each other, and the total number of strategies will increase further.

[0211] 1.3.1 Power system cluster analysis

[0212] Taking the distributed power system cluster in the power network as an example, the power system is abstracted into nodes, focusing on the interconnection between nodes, and building its complex network model. On this basis, the temporal dynamic characteristics of nodes and links in the power system cluster under large-scale damage scenarios are considered, including various operation and maintenance situations such as normal, damage, and repair. Figure 3 shown.

[0213] Taking a regional power grid as an example, we first comprehensively analyze the basic information of the nodes in the power system cluster in a regional power grid and the basic information of the links between nodes. As shown in Table 1, the node numbers and geographical location information of the power system cluster in a regional power grid are given. Table 2 gives the link information of the power system cluster in a regional power grid, that is, the adjacency matrix information of the complex network model of the power system cluster. Based on the above information, the complex network modeling method can be used to establish a complex network model of the power system cluster in a regional power grid. G .

[0214] Table 1 Basic information of power system cluster nodes in a certain area power grid

[0215]

[0216] Table 2 Basic information of power system cluster links in a certain area power grid

[0217]

[0218] 1.3.2 Post-disaster recovery analysis

[0219] 1.3.2.1 Analysis of sabotage cases

[0220] Based on the complex network model of power system cluster in a certain area power grid G Modeling, continue the operation and maintenance situation of the power system cluster under disasters, consider using a circular area to represent the disaster of earthquake as an example, after randomly generating the damage center and damage radius of the circular area, the power system cluster in a certain area of ​​the power grid G The destructive situation Figure 5 As shown, pink indicates failed nodes.

[0221] The maintenance task is performed in parallel by multiple maintenance teams with different maintenance capabilities. In the present invention, five maintenance teams with different V edge and V move , and divide the nodes into three levels: the node weight of the first level node is w∈ (0, 2]; the node weight of the second-level node is w∈ (2, 4]; the node weight of the third-level node is w∈ (4, +∞). Each maintenance team has different requirements for the third-level node. MTTR : MTTR 1st , MTTR 2nd and MTTR 3rd ,The maintenance capability information of each maintenance team is shown in Table 3. In addition, the maintenance capability of the ,maintenance team can also be defined according to its actual ,situation.

[0222] Table 3 Maintenance team information

[0223]

[0224] Figure 5 Four different scale damage cases of the power system cluster are analyzed, and the damage information of each damage case is shown in Table 4. Figure 5 The four MCRPs in are performed by the same maintenance team group, which consists of maintenance team 1, maintenance team 2 and maintenance team 3.

[0225] Table 4 Related information of different damage cases

[0226]

[0227] 1.3.2.2 Maintenance strategy and flexibility analysis

[0228] for Figure 5 For each failure case in damage =200h, a large-scale local damage event occurs, which causes a sharp decline in the functional performance of the power system cluster. Then corrective maintenance work is performed on the power system cluster to recover from the large-scale local damage, after which the functional performance is improved and restored at time τ restoration Reaching FOM threshold FOM thr = 0.96, set τ thr The parameter can be set according to the actual situation. Generally speaking, different MCRPs have different τ restoration , the elasticity analysis of the above four MCRPs is as follows Figure 5 As shown, its elastic loss RL As shown in Table 5.

[0229] Table 5 Elasticity loss of corrective maintenance by multiple maintenance teams

[0230]

[0231] 1.3.2.3 AC-Net Learning and Training Analysis

[0232] The proposed AC-MCTS method is applied to train the DRL agent using a server with an Intel i9 7980XE CPU and four RTX2080 TI-A11G GPUs. The training starts with a completely random AC-Net and lasts for about 30 hours without intervention. During the training process, each MCTS uses 800 simulations to perform 10,000 MCRPs, which is equivalent to nearly 0.2 seconds of thinking time for each repair action selection. AC-Net uses the dataset generated by MCRP to train parameters and applies min-batches for update.

[0233] Figure 6 The mean square error and cross entropy parameter training of AC-Net after each MCRP simulation are analyzed. MSE is between the actual results z and AC-Net predicted value v The goal is to predict the elastic loss RL CEL represents the AC-Net prediction prior probability p and MCTS strategy π The goal is to predict the strategy π Although the mathematical models of MSE and CEL are different, AC-Net is trained using the same training dataset and the same loss function, Figure 6 It can be seen that in some training time periods, the two curves have the same trend. In addition, some empirical parameters: α=2, β=4, c L2 = 10 -4 , c puct = 5. After about 8000 MCRPs, MSE and CEL tend to be stable, and then AC-Net can effectively guide MCTS to select maintenance actions for multiple teams.

[0234] 1.3.3 Discussion and Analysis

[0235] The main goal of post-disaster recovery of power system clusters is to generate feasible multi-team corrective maintenance strategies. The present invention conducts an extended analysis of multi-team corrective maintenance strategies to determine the impact of different factors on the optimal strategy.

[0236] 1.3.3.1 Analysis of decision-making methods

[0237] This paper proposes an efficient AC-MCTS method for post-disaster recovery of cluster systems. In addition, the heuristic hybrid game (HHG) method and the DQN method are also used to solve the problem. The three methods are analyzed separately by running Python programs for each method to solve the above four post-disaster damage situations. AC-MCTS gives the optimal solution with better RL than other methods, such as Figure 7 As shown in Figure 2. In the early MCRP stage under each damage situation, the DQN method and the HHG method can achieve better FOM than AC-MCTS because these two methods select each repair action with the goal of achieving the best FOM improvement in the current repair stage. However, they may also lead to local optimality, such as in the middle and late stages of MCRP when multiple repair team members have to spend more time moving. According to Figure 7 As shown in the results, AC-MCTS achieves better performance in a shorter time than other methods. RL , because it uses the value predicted by DNN, aiming to achieve better RL Therefore, the results show that AC-MCTS generates the optimal strategy for post-disaster recovery.

[0238] The efficiency of these methods is analyzed in Table 6. According to these results, the solving speeds of AC-MCTS, DQN, and HHG are stable, and they do not decrease exponentially with the increase in the number of destroyed nodes and links. However, the solving speed of the HHG method decreases significantly with the increase in the degree of destruction.

[0239] Table 6 Running time of different methods (seconds)

[0240]

[0241] 1.3.3.2 Maintenance teams of different sizes

[0242] Keeping the same post-disaster damage situation ( Figure 5 In the case of post-disaster damage scenario 1), four methods are used to consider three, four, and five different repair team sizes. The corrective repair results of the three repair teams (repair teams 1, 2, and 3) are shown in Figure 1. Figure 7 As shown in (a), the corrective maintenance of four maintenance teams (teams 1, 2, 3 and 4) and five maintenance teams (members 1, 2, 3, 4 and 5) are as follows: Figure 8 The elasticity loss under different numbers of maintenance teams is shown in Table 7. As the number of maintenance teams increases, the elasticity loss RL and total repair time is reduced, the corrective maintenance agent also obtains better RLTherefore, these results show that the Restorative Repair Agent exhibits stable performance for the disaster recovery problem with different numbers of repair teams, because the number of repair teams is randomly generated during the Restorative Repair Agent training process.

[0243] Table 7 Analysis of elasticity loss for different maintenance team sizes

[0244]

[0245] 1.3.3.3 Maintenance teams at different levels of expertise

[0246] Maintaining the same post-disaster damage status ( Figure 5 In the post-disaster damage case 1), based on the maintenance strategy generated by the restorative maintenance agent, the difference in cluster system performance recovery after the maintenance strategy is executed by the maintenance teams with different business levels is analyzed. The elasticity analysis of the cluster system after the maintenance team 1, 2 and 3 performs the restorative maintenance task is shown in the figure. Figure 5 As shown in (MCRP and resilience analysis of post-disaster damage case 1), the relevant analysis of the support system cluster composed of maintenance teams 2, 3 and 4 and maintenance teams 3, 4 and 5 are shown in Fig. 9 As shown in (a) and (b). Considering that the maintenance teams in the guarantee system cluster have different business capabilities, the elasticity loss of the cluster system recovery process is shown in Table 8. For the same corrective maintenance agent, the generated maintenance strategy achieves similar elasticity loss in almost the same total repair time. Therefore, these results show that the proposed method shows stable performance for maintenance teams with different business capabilities.

[0247] Table 8 Analysis of elasticity loss for maintenance teams with different business capabilities

[0248]

Claims

1. A method for post-disaster restorative maintenance of a cluster system based on the AC-MCTS algorithm, wherein the cluster system is a large-scale distributed power system cluster, characterized in that: The method comprises the following steps: a) Construct a post-disaster restorative maintenance process for multiple maintenance teams and generate a decision feature tensor for the cluster system at any time point t X t ; b) Using Actor-Critic neural network X t Evaluate and output the prior parameters, which are used as input to the Monte Carlo tree search algorithm; c) Based on the Monte Carlo tree search algorithm X t The search operation of the maintenance action is performed under the conditions, including: i) Select: X t Acts as the root node of the Monte Carlo tree search and selects the maintenance action with the best action value based on the upper confidence interval algorithm; ii) Expansion and evaluation: Add leaf nodes in the search tree to the queue, evaluate their strategic value using the ResNet neural network, and update the statistics of related nodes in the search tree; iii) Backtracking: Based on the evaluation results, backtrack along the search path and update the number of visits and action value of each branch on the search path; iv) Execution: By iterating the above steps, the improved maintenance state transition probability π is obtained and the global optimal maintenance action is selected a * t , transfer the cluster system from time point t to t+1; d) Repeat steps a) to c) until the corrective maintenance task of the cluster system is completed; After the disaster, the corrective repairs will be carried out until t At this moment, for the decision feature tensor X t , using the prior parameters obtained from the Actor-Critic neural network prediction model ( P t , V t ) is used as input, and the maintenance action search operation is performed based on the Monte Carlo tree search method, and finally the improved maintenance state transition probability can be obtained. π and best maintenance actions a t * , whose expression is: ； Monte Carlo Tree Search searches each tree node X All branches of X , a ), corresponding to all the next maintenance actions of this tree node a ∈ Action ( X ), each branch ( X , a ) stores a set of statistics as follows: ； in N ( X , a ) indicates the number of visits; W ( X , a ) represents the sum of total action values; Q ( X , a ) represents the average action value; P ( X , a ) indicates selecting a branch ( X , a )’s prior probability; The Monte Carlo tree search algorithm is based on X t The search operation of the maintenance action is performed under the condition of: i) Select: First, t The decision feature tensor X of the moment cluster system t As the root node X of the Monte Carlo tree 0 , that is, the cluster system status at each moment, and from the root node X 0 Start searching until L Time reaches the leaf node X at the end of the search tree L Ends at l time, l < L , based on the upper confidence interval tree search algorithm and the maximum upper confidence interval algorithm, the best action a under the current iteration conditions is selected according to the statistical data stored in each branch of the search tree l , whose expression is: ， ； where c puct is a constant determined by the degree of Monte Carlo tree search; a∈Action(X), Action(X) represents the action space; ii) Extension and Evaluation The leaf node X L Add to the queue as the input of ResNet, and then the leaf node X L To expand, each edge of the leaf node expansion (X L ,a) Initialize storage data: ； All positions in the queue are evaluated by the neural network, resulting in leaf nodes X L The strategic value of v , the search thread is locked until the strategy evaluation is completed; iii) Backtracking After expansion and evaluation, the leaf node X L The strategic value of v Backtrack along the original path to the parent node; during the backtracking process, each branch (X l ,a l ) The access count parameter stored in the memory is updated once, and its expression is: ； At the same time, search for the branch (X l ,a l ) maintenance action a l The total action value and average action value parameters are also updated once, and their expressions are: ； iv) Execution By iterating the above three steps 1000 times, the maintenance status probability is obtained π , and select the feature tensor of the cluster system at the current moment X t Global optimal maintenance action under the condition a t * ; Among them, the improved cluster system maintenance status probability π It is exponentially related to the number of visits, and its expression is: ； Where τ is an empirical parameter that controls the search process; Execute optimal maintenance actions a t * Afterwards, the feature tensor of the cluster system maintenance status is transformed from t Time X t Transfer to t+ 1 Moment X t+1 , the search tree will be X t+1 The new root node X 0 , and keep the node X in the original search tree t+1 All child nodes below it and their statistics, while discarding the branches of the remaining nodes at the same level as this node.

2. The method according to claim 1, characterized in that The Actor-Critic neural network is a dual-output residual neural network, including a strategy module and a value module; wherein: a) The strategy module outputs the prior repair probability matrix through multiple convolutional layers and fully connected layers P t , the elements of this matrix represent the probability of selecting a repair action; b) The value module outputs a scalar value V through the residual network structure t , used to evaluate the selection reward of the current corrective repair action.

3. The method according to claim 1, characterized in that The Actor-Critic Neural Network f θ ( S, A ) =( p , v ) is a dual-output residual neural network, which is used as a maintenance strategy prediction model. The network parameters are randomly initialized. θ 0; At every moment t , AC-Net f θ The input is the current repair state S before damage t =[ S f t , S c tc ] and the adjacency matrix A; S f t It can be written as follows: ； Among them A f t is the time step of MCRP t The adjacency matrix at , if t<1, then A f t = A f 1 =A damage damage; at the same time, S c tc Written as follows: ； Among them A c tc It's the maintenance team c The time step t of the repair process c The adjacency matrix of c <1, then A c tc = A 1 c =A d ; The output of the Actor module and the Critic module is the prior repair probability matrix P t Sum value V t ; Matrix P t is a N × N The adjacency matrix, whose element P ij Yes Select Repair Action a = ( i , j ) probability; value V t is a standardized parameter used to evaluate the reward of the current corrective repair action selection; The AC-Net in the restorative repair agent uses a residual module consisting of residual blocks of convolutional layers with ReLU functions to process the input feature S t , while the input feature A is a constant matrix, which is processed by a multi-layer module composed of fully connected layers; the residual module and the multi-layer module include shared layers and are merged into two independent modules, namely policy and value; in addition, the policy and value modules get their inputs from the shared layer and output the prior repair probability P using a fully connected layer and a sigmoid function t Sum value V t .

4. The method according to claim 1, characterized in that: The selection basis of the upper confidence interval algorithm includes: a) maximize the upper confidence interval; b) Select maintenance actions with lower visit counts and higher prior probabilities.

5. The method according to claim 1, characterized in that: The ResNet neural network structure includes: a) A convolutional module consisting of 256 3×3 filters; b) a deeply stacked residual module, wherein the residual module includes a structure consisting of a plurality of convolutional layers, batch normalization layers, ReLU function layers, and Add modules; c) Strategy module and value module, which are used to output the prior maintenance probability matrix p and the standardized value v respectively.

6. The method according to claim 5, characterized in that The expression of the ResNet prediction model is f θ ( X ) = ( p , v ), θ is the network parameter of ResNet, the initial network parameter θ 0 is obtained by random initialization and is optimized and updated as the algorithm iterates.

7. A system for implementing the method according to any one of claims 1 to 6, the system comprising: a) Cluster system feature extraction module, used to generate the decision feature tensor of the cluster system at any time point t X t ; b) Actor-Critic neural network module, communicating with the cluster system feature extraction module, for X t Perform evaluation and output prior parameters; c) a Monte Carlo tree search module, communicating with the Actor-Critic neural network module, for performing Monte Carlo tree search based on the X t The Monte Carlo tree search module is used to perform the search operation of the maintenance action under the conditions of: i) Select the module to X t Acts as the root node of the Monte Carlo tree search and selects the maintenance action with the best action value based on the upper confidence interval algorithm; ii) Expansion and evaluation module, which is used to add leaf nodes in the search tree to the queue, evaluate their strategic value using the ResNet neural network, and update the statistics of related nodes in the search tree; iii) A backtracking module, which is used to backtrack along the search path according to the evaluation results and update the visit count and action value of each branch on the search path; iv) Execution module, which is used to obtain the improved maintenance state transition probability π by iterating the above steps and select the global optimal maintenance action a t * , transfer the cluster system from time point t to t+1.

8. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images