A deep actor-critic architecture system for multi-team post-disaster restorative repair decision-making

Abstract

The present invention relates to the field of artificial intelligence technology, and in particular to a deep Actor-Critic architecture system for multi-team post-disaster restorative maintenance decision-making. The architecture realizes iterative optimization of multi-team maintenance strategies by introducing a constrained Markov decision process (CMDP) and combining it with the AC-MCTS algorithm. Specifically, the system generates local damage and system status through a simulation environment, selects the optimal maintenance action through a restorative maintenance agent during the strategy iteration process, uses elasticity analysis to evaluate the effect of system recovery, and continuously adjusts the parameters of the Actor-Critic network to improve the decision-making efficiency of the strategy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a deep Actor-Critic architecture system for multi-team post-disaster restorative maintenance decision-making. Background Art

[0002] With the rapid development of modern society, cluster systems have been widely used in large-scale distributed infrastructures such as power systems, communication networks, and transportation networks. These cluster systems are usually composed of multiple key nodes and links to ensure the stable operation and efficient service of the system. However, natural disasters such as earthquakes and typhoons may cause serious damage to some nodes and links in the cluster system, significantly reducing the overall performance of the system, or even causing partial or complete failure of the system.

[0003] Post-disaster restorative maintenance is a key issue in cluster system operation and maintenance, especially how to quickly restore the normal operation of the system after a disaster. Traditional post-disaster repair strategies often rely on manual experience or simple rule algorithms, which show obvious shortcomings in dealing with the complexity and large-scale distributed characteristics of modern cluster systems. Specifically, existing methods are difficult to handle a large number of damaged nodes and links within a limited time, and lack effective optimization methods to coordinate the collaborative work of multiple maintenance teams, resulting in a time-consuming and inefficient repair process, and the system's recovery speed and effect are difficult to be satisfactory.

[0004] In the process of corrective maintenance, two basic and challenging issues are repair timing and path planning. Specifically, how to arrange multiple maintenance teams within a limited time to repair the damaged system and its components in the optimal order and path is the core issue to ensure the rapid recovery of the system. Traditional maintenance methods usually focus on the maintenance tasks of a single team. However, with the expansion of the scale of network systems, more and more research has begun to focus on maintenance strategies under multi-team collaboration.

[0005] In recent years, methods based on deep reinforcement learning (DRL) have shown significant advantages in solving complex decision-making problems. By simulating the interaction between the agent and the environment, the DRL algorithm can gradually optimize the strategy in multi-step decision-making tasks, and is suitable for solving complex, multi-level decision-making problems. In particular, the deep reinforcement learning method based on the Actor-Critic architecture can effectively cope with dynamic and diverse environments by separating the policy evaluation (Critic) and policy generation (Actor) processes. Summary of the invention

[0006] In order to solve the problem of restorative maintenance of network cluster systems under large-scale local damage, the present invention proposes a deep Actor-Critic architecture system for multi-team post-disaster restorative maintenance, aiming to optimize the timing and path planning of multi-team maintenance. This architecture realizes the iterative optimization of multi-team maintenance strategies by introducing the constrained Markov decision process CMDP and combining it with the AC-MCTS algorithm. Specifically, the system generates local damage and system status through a simulation environment, selects the optimal maintenance action through a restorative maintenance agent during the strategy iteration process, uses elasticity analysis to evaluate the effect of system recovery, and continuously adjusts the parameters of the Actor-Critic network to improve the decision-making efficiency of the strategy.

[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0008] A deep Actor-Critic architecture system for multi-team post-disaster restorative repair, the system comprising:

[0009] A simulation environment module for generating local damage and network cluster system states;

[0010] A corrective maintenance agent module executes the constrained Markov decision process CMDP of multiple maintenance teams and generates a series of maintenance actions based on the AC-MCTS algorithm. The corrective maintenance agent module uses the search algorithm integrated with AC-Net to select maintenance actions for each maintenance team, and extracts the current recovery state of the network cluster system and the time series characteristics of multiple maintenance teams through AC-Net to estimate the value function and policy function of the maintenance action.

[0011] A resilience analysis and reward module, which is used to evaluate the resilience loss of system recovery according to the execution of the maintenance strategy and generate the corresponding reward value;

[0012] An Actor-Critic network training module adjusts AC-Net parameters through training datasets to provide more effective maintenance action search support for the AC-MCTS algorithm.

[0013] Preferably, the multi-maintenance team constrained Markov decision process CMDP executed by the corrective maintenance Agent module includes:

[0014] A multi-repair team maintenance process module, in the event of partial system damage, restores the performance of the cluster system through a series of maintenance actions performed in sequence by each maintenance team;

[0015] A multi-maintenance team time sequence module determines the maintenance timing and path planning of multiple maintenance teams to optimize the time efficiency of system recovery.

[0016] Preferably, a cre The maintenance team's support system cluster performs a series of maintenance actions during the MCRP period. The maintenance action sequence set is as follows:

[0017]

[0018] The action sequence set of maintenance team c can be described as follows:

[0019]

[0020] Corrective maintenance actions are defined as a * t∣tc = a * tc∣t =( i , j ), which means that the maintenance team c repairs the node i With Node j After repairing the link, if the node j If it is a damaged node, the maintenance team continues to repair the node, otherwise it continues to perform the next maintenance action; at each time t, the restorative maintenance agent uses the search algorithm integrated with AC-Net to select a maintenance action for the maintenance team c that has just completed the last maintenance action a * t∣tc ;

[0021] The decision making of the corrective maintenance agent is described as follows:

[0022]

[0023] in S= [ S f , S c ] is the current maintenance status, S f The status of repairs performed on behalf of the maintenance team, S c And the maintenance status performed on behalf of maintenance team c;

[0024] For having n cre The maintenance team's support system cluster has a time sequence and n cre The timing of each maintenance team;

[0025] Maintenance Team c The timing description is as follows:

[0026]

[0027] where τ tc Indicates that the maintenance team c Complete its maintenance tasks a * tc∣t =( i , j ) time, τ tc The calculation method of is as follows:

[0028]

[0029] Where τ( a * tc∣t ) indicates execution of an action a * tc∣t The maintenance time consists of four situations; in the first situation, the maintenance team is at the node i The location of the node j Normal; in the second case, the maintenance team is at the node i The location of the node j in a state of destruction; in the third case, the repair team is at a node i ' position, and the node j Normal; in the fourth case, the maintenance team is at a node i ' position, and the node j in a state of destruction;

[0030] τ( a * tc∣t ) is calculated as follows:

[0031]

[0032] Repair damaged nodes i The average time of MTTR i The time to repair each edge is expressed as follows:

[0033]

[0034] in v edge Indicates the speed at which a broken link is repaired; i Move to j The time can be expressed as follows:

[0035]

[0036] in v move Indicates the movement speed of the maintenance team;

[0037] The time series of the security system cluster can be described as follows:

[0038]

[0039] Where τ t =0=τ d , τ t Complete actions on behalf of the maintenance team a * t∣tc = a * tc∣t =( i , j ) maintenance time.

[0040] Preferably, the elasticity analysis and reward module further comprises:

[0041] An elastic loss calculation unit calculates the elastic loss RL of the repair process by integrating the performance recovery curve of the cluster system during the MCRP period;

[0042] A reward calculation unit, based on elastic loss RL, defines an inverse proportional function as the reward of the maintenance strategy and optimizes the maintenance strategy to minimize the elastic loss.

[0043] As a further optimization, in the MCRP process, for maintenance actions a * t∣tc = a * tc∣t =( i , j ), if the node j is normal, then repair the link ( i , j ) after which the link is assumed to be operational again; if the node j is damaged, then repair the link ( i , j ) and nodes j Afterwards, assuming that the link and the node are back up and running;

[0044] In the MCRP process, the FOM (τ t ) is calculated according to the following formula;

[0045]

[0046] Where E*(G) is the average efficiency before the local attack event occurs, E τ (G) is the average efficiency at time τ;

[0047] When the maintenance teams perform their respective maintenance actions, the FOM (τ t ) is considered unchanged; whenever a maintenance team completes a maintenance action, the repaired links and nodes resume operation, and the FOM (τ t ) is considered a step improvement; after completing MCRP, the FOM(τ t ) Integrate over the duration of MCRP to calculate the elastic loss of the entire repair process RL ; Elasticity loss RL The calculation process is as follows: Assume that the local attack event occurs at time τ d , which causes a sharp drop in cluster functional performance; then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and at time τ d Reaching FOM threshold FOM thr , using the following elastic loss RL To quantify the loss suffered by the cluster due to a local attack:

[0048] ;

[0049] Optimized maintenance strategy requires less elasticity loss RL , then the reward of the maintenance strategy executed by the entire MCRP process is defined as the elastic loss RL The inverse proportional function of the inverse proportional function is the integral of the cluster FOM without elastic loss in the specified time interval, as shown below:

[0050]

[0051] Preferably, the Actor-Critic network training module comprises:

[0052] An AC-Net training unit, which adjusts the parameters θ of AC-Net by using the empirical dataset (S, A, π, z), adopts a loss function including cross entropy loss (CELs) and mean squared error (MSEs) to maximize the similarity between the prior repair probability p and the tree search probability π, and minimize the error between the prior value v and the reward z;

[0053] A batch training unit randomly selects a small batch of training samples from the event data of all time steps and uses the batch training method to adjust the AC-Net parameters to improve the decision-making ability of the corrective maintenance agent.

[0054] As a further optimization, for each moment t , Corrective Maintenance Agent uses AC-Net f θ ( S, A ) = ( p , v ) is a variant of the MCTS algorithm f ts ( S,p,v ) = ( π , a * ) provides search guidance, where p is the prior repair probability matrix, which is used to guide the tree search to select corrective repair actions, and v is a normalization parameter used to evaluate the return of the current corrective repair action selection; the search algorithm uses the parameter p and v To output the search probability π , to select high-quality repair actions a * ;

[0055] Every moment t The AC-Net training data is stored as ( S t , A , π t , z t ),in It's the action a * t∣tc reward; α and β are experience parameters, ;

[0056] When the corrective maintenance agent completes the MCRP, an episode is completed and the data ( S , A , π , z ) is sampled from all time steps of this MCRP; AC-Net then uses the empirical data ( S , A , π , z ) to adjust its parameters θ ; The training goal is to maximize the prior repair probability p With tree search probability π similarity and minimize the prior value v and rewards z The error between ; by using gradient descent to adjust the parameters θ, the loss function includes cross-entropy losses (CELs) and mean squared errors (MSEs), as shown below:

[0057]

[0058] in C L2 is a constant used to control the L2 regularization level to prevent overfitting;

[0059] For each episode, a small batch of training samples is randomly selected from the dataset consisting of all episode data, and then batch training is used to adjust the parameters θ ; Adjust parameters θ Afterwards, AC-Net improves the decision-making ability of the corrective maintenance agent in the next episode.

[0060] Preferably, the AC-Net parameter θ is initialized by random generation and gradually optimized during the training process of multiple episodes, so that the MSE and CEL tend to be stable, and finally a decision support system is formed that can effectively solve the CMD problem under large-scale local damage.

[0061] Preferably, the maintenance strategy generated by the system during the operation phase of the cluster system can directly provide the cluster system with a maintenance strategy for coping with large-scale local damage without retraining the AC-Net.

[0062] Furthermore, the present invention also discloses a computer-readable storage medium having a computer program or instruction stored thereon, which implements the deep Actor-Critic architecture system when executed by a processor.

[0063] The present invention adopts the above-mentioned technical solution and proposes a two-level decision-making system for multi-team maintenance timing and path planning based on a deep Actor-Critic architecture, which has significant technical effects for the problem of corrective maintenance of network cluster systems under large-scale local damage, as follows:

[0064] 1. Improve the efficiency and accuracy of maintenance decisions

[0065] By introducing a deep Actor-Critic architecture, the present invention can effectively generate and optimize maintenance strategies in complex and changing environments. The corrective maintenance agent uses AC-Net to extract the current recovery state of the network cluster system and the multi-team timing characteristics, and estimates the value function and strategy function of the maintenance action, so that the optimal maintenance action can be selected more accurately. This greatly improves the efficiency and accuracy of maintenance decisions under multi-team collaboration and shortens the time for system recovery.

[0066] 2. Optimize maintenance sequence and path planning for multi-team collaboration

[0067] The present invention realizes the global optimization of maintenance sequence and path under multi-team collaboration by combining the multi-team constrained Markov decision process CMDP and the AC-MCTS algorithm. In view of the complexity problems faced in the multi-team maintenance process, the present invention can dynamically adjust the maintenance sequence and path of each team according to the system status and damage situation, ensure more efficient collaboration between teams, avoid waste and conflict of maintenance resources, and maximize the recovery speed of the system.

[0068] 3. Improve system resilience and stability

[0069] During the corrective maintenance process, the present invention evaluates the system's performance recovery through elasticity analysis, and uses this as a reward signal to guide the optimization of the maintenance strategy. By minimizing the elasticity loss of the system, the present invention ensures that the system can quickly recover to normal operation under different damage scenarios, and enhances the recovery elasticity and stability of the network cluster system. This technical effect is particularly significant in the case of natural disasters, helping to reduce the economic and social impact of system failures.

[0070] 4. Reduce manual intervention and improve automation level

[0071] The present invention adopts the method of deep reinforcement learning, so that the corrective maintenance decision-making process can be automated to a large extent. Through continuous strategy iteration and parameter optimization, the system can autonomously learn and adapt to different environments and damage situations, reducing dependence on human intervention. This technical effect not only improves the automation level of maintenance work, but also reduces labor costs and reduces the risk of human decision-making errors.

[0072] 5. Improve adaptability and promotion across scenarios

[0073] Since the architecture of the present invention is based on deep learning and reinforcement learning methods and has strong generalization capabilities, it can be widely used in different types of network cluster systems, including but not limited to power networks, transportation networks, communication networks, etc. Whether in the specific form of local damage or the structural complexity of the system, the solution of the present invention can adapt and provide effective decision support, and has good cross-scenario adaptability and promotion.

[0074] In summary, the present invention has demonstrated significant technical effects in improving the efficiency of multi-team collaborative maintenance, optimizing maintenance timing and path planning, enhancing system recovery resilience, reducing manual intervention, and improving cross-scenario adaptability. It has broad application prospects and important socio-economic value. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Fig. 1 Complex network model of cluster system under large-scale destruction.

[0076] Fig. 2 Schematic diagram of the residual module structure.

[0077] Fig. 3. Construction diagram of complex network model of distributed power system cluster.

[0078] Figure 4 A large-scale local area of ​​a power system cluster in a regional power grid.

[0079] Figure 5 Diagram of the corrective maintenance process for multiple maintenance teams.

[0080] Figure 6 AC-Net learning and training analysis diagram.

[0081] Figure 7 Resilience loss analysis plots for different approaches for various post-disaster damage scenarios.

[0082] Figure 8 Maintenance strategy elasticity analysis diagram for different maintenance team sizes.

[0083] Figure 9: Maintenance strategy elasticity analysis diagram for maintenance teams with different business capabilities. DETAILED DESCRIPTION

[0084] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0085] The present invention aims at the post-disaster repair strategy problem of cluster system, considers the two-level decision of repair sequence and path planning, and studies the restorative maintenance decision method based on Actor-Critic Monte Carlo Tree Search (AC-MCTS). The designed restorative maintenance decision method needs to consider the spatial position of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, and comprehensively analyzes the two-level decisions of node link repair sequence and multi-team path planning, which includes two core ideas: a deep reinforcement learning Actor-Critic architecture considering the two-level decision of multi-team repair sequence and path planning, and an AC-MCTS algorithm for restorative maintenance of cluster systems. For the two-level decision of the repair sequence of the cluster system and the path planning of the maintenance team, solving the global optimal decision under the condition of uneven benefits of multiple maintenance teams is the core feature that needs to be focused on. Therefore, it is necessary to focus on the restorative maintenance strategy considering multiple maintenance teams, and be able to plan the optimal path for multiple maintenance teams to repair damaged systems and components, as well as the optimal timing for each maintenance team to repair damaged systems and links between systems. In order to solve the two-level decision-making characteristics of the maintenance strategy between multiple maintenance teams and a single maintenance team in this problem, the AC-MCTS method designed by the present invention uses the damage information of cluster system nodes and links to determine the optimal maintenance actions and paths of multiple maintenance teams. In addition, the above-mentioned Actor-Critic architecture needs to design a strategy-value neural network to estimate the value function and the strategy function, and then perform the maintenance action search process based on AC-MCTS, and finally obtain the maintenance action search probability and the optimal maintenance action. Based on the proposed AC-MCTS restorative maintenance decision algorithm, the present invention conducts case verification in the case analysis part for post-disaster repair of multiple maintenance teams in the distributed power system cluster disaster scenario. The case analysis results show that the proposed restorative maintenance decision algorithm can provide an efficient restorative maintenance solution in the post-disaster repair work of the distributed power system cluster, and realizes the rapid recovery of the power system cluster performance under large-scale damage.

[0086] 1.1 Description of cluster system post-disaster repair problems

[0087] 1.1.1 Large-scale destruction and repair of cluster systems

[0088] First, we analyze the cluster system and its large-scale destruction, consider the interconnection relationship between nodes in the distributed cluster system, and build its complex network model, such as Figure 1 shown.

[0089] (1) Destruction

[0090] Based on the cluster, the destruction center and destruction radius are randomly generated. The red area in the figure is the local destruction area. Assume that all nodes and links in the red area are invalid. When these links are removed from the graph, the nodes are isolated. Then, the state of the links between nodes can be described as the current adjacency matrix after the local attack. A damage ,The state vector of a node in a cluster system can be described as:

[0091] When node n i When destroyed i node =0, otherwise S i node =1.

[0092] (2) Repair

[0093] This paper will propose a rich and executable strategy to support maintenance activities, considering maintenance teams, maintenance time series and maintenance objects. The proposed strategy aims to make up for the limitations of other current studies, which mainly consider the repair object. The maintenance activities are performed in parallel by multiple maintenance teams. Repairing damaged nodes i The average time of MTTR i The time to repair each edge can be expressed as follows:

[0094]

[0095] in v edge Indicates how quickly broken links are repaired. i Move to j The time can be expressed as follows:

[0096]

[0097] in v move Indicates the movement speed of the repair team.

[0098] (3) Recovery

[0099] Elasticity is currently widely used to characterize the recovery performance of a cluster after a local attack. Therefore, the present invention introduces cluster elasticity to evaluate the effectiveness of the recovery strategy, with the goal of improving cluster performance and shortening recovery time. In order to describe the quantitative measurement of cluster elasticity, it is necessary to define a quantity that represents the functional performance of the cluster. The present invention selects cluster effectiveness (Figure of Merit, FOM) as a quantitative performance indicator. On this basis, the average efficiency of the cluster system is E ( G) is normalized as follows to define the FOM over time τ:

[0100] (1.4)

[0101] Where E*(G) is the average efficiency before the local attack event occurs, E τ (G) is the average efficiency at time τ. In addition, other performance parameters of the power grid can also be used in the definition of FOM.

[0102] Assume that the local attack event occurs at time τ d , and the cluster functional performance drops sharply. Then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and at time τ d Reaching FOM threshold FOM the The following resilience indicators are used to quantify the loss suffered by the cluster due to local attacks:

[0103]

[0104] 1.1.2 Corrective maintenance decision-making problem for post-disaster emergency repair

[0105] Cluster post-disaster repair is considered as a corrective maintenance decision (CMD) problem for multiple maintenance teams. The constraints are the recovery time and the recovery level, and the optimization goal is the elasticity loss of the cluster system. RL , the basic decision variable is the restorative maintenance action. From the nature of the decision variables, this problem is an integer programming problem using binary variables. For each maintenance team, considering the three actions related to post-disaster repair, namely, repairing nodes, repairing links, and moving, the number of strategies is 2 N+K ; For multiple maintenance teams, their strategy spaces are coupled to each other, so this problem is an NP-hard problem. In order to achieve optimal recovery, the CMD problem aims to minimize under limited recovery time and a certain recovery level. RL Therefore, the CMD problem for multiple maintenance teams can be expressed as follows:

[0106]

[0107] In the constraint (1.62), c is the maintenance team number, F={1,2,…, n cre} is composed of n cre A maintenance team consisting of a support system cluster, t c It's the maintenance team c The maintenance time step is T c ={1,2,…,T c} is the time step set in its own maintenance process, a * tc represents the time step t c Maintenance action, Ω c It's the maintenance team c In the constraint (1.63), t is the maintenance time step of the maintenance team and T ={1,2,…, T c} is the maintenance team time step set, a * t Indicates the time step t The maintenance action is, Ω is the maintenance team action sequence set. Constraint (1.64) represents the maintenance system cluster action sequence set Ω and each maintenance team action sequence set Ω c In the constraint condition (1.65), τ( a * tc ) indicates execution of an action a * tc The time interval, τ thr is the recovery time threshold. Constraint (1.65) forces the total repair time of the maintenance team to be lower than the threshold recovery time, while constraint (1.66) forces the cluster performance to be within time τ r Reached FOM threshold.

[0108] 1.2 Corrective maintenance decision-making algorithm considering timing and path planning

[0109] The corrective maintenance decision-making for post-disaster repair of cluster systems needs to consider the spatial position of large-scale distributed nodes in the cluster system and the network characteristics of the links between nodes, comprehensively analyze the node link maintenance timing and multi-team path planning, and design a corrective maintenance decision-making method based on the deep reinforcement learning Actor-Critic architecture. Considering the two-level decision-making of multi-team maintenance timing and path planning, it supports corrective maintenance work such as the generation of corrective maintenance plans for cluster systems with network characteristics. This method contains two core ideas: a deep Actor-Critic architecture that considers the two-level decision-making of multi-team maintenance timing and path planning, and an AC-MCTS algorithm for corrective maintenance of cluster systems.

[0110] 1.2.1 Actor-Critic architecture considering two-level decision-making of multi-team maintenance timing and path planning

[0111] Regarding the research content of corrective maintenance considering timing and path planning, especially for the corrective maintenance work of network cluster systems such as infrastructure networks, two basic challenging problems are the two-level decisions of the repair timing and path planning of the network cluster system. Therefore, the present invention focuses on the corrective maintenance strategy considering multiple maintenance teams, which can plan the optimal path for multiple maintenance teams to repair damaged systems and components, as well as the optimal timing for each maintenance team to repair damaged systems and inter-system links. In order to solve the two-level decision-making characteristics of the maintenance strategy between multiple maintenance teams and a single maintenance team in this problem, the present invention intends to design a deep Actor-Critic architecture that considers the two-level decision-making of multi-team maintenance timing and path planning. The idea of ​​this decision-making architecture uses the damage information of the network cluster system to determine the optimal maintenance actions and paths of multiple maintenance teams.

[0112] The main idea of ​​the deep Actor-Critic architecture considering two-level decisions of multi-team maintenance timing and path planning is to iteratively execute the multi-repair team CMDP process in the policy iteration process. First, the simulation environment generates local damage and network cluster system status. Then, the restorative maintenance agent executes the multi-repair team CMDP process and generates a series of maintenance actions based on the AC-MCTS algorithm. When generating each maintenance action, the restorative maintenance agent uses AC-Net to extract the current recovery state of the network cluster system and the multi-repair team timing characteristics, and estimates the value function and policy function of the maintenance action to support the AC-MCTS algorithm to search for maintenance actions. Subsequently, the simulation environment rewards the maintenance strategy based on evaluation methods such as elasticity analysis. The maintenance strategy consists of a series of maintenance actions generated by the CMDP process. Finally, the AC-Net parameters are adjusted based on the training sample data set; the new AC-Net parameters are used in the next policy iteration process to provide MCTS with more effective maintenance action search support. For the CMD problem, the deep reinforcement learning framework proposed in the present invention. The framework shows four parts: multiple crews repair process (MCRP), multi-repair team repair timing, elasticity analysis and reward, and neural network training, as described below.

[0113] (1) Maintenance process of multiple maintenance teams

[0114] Before the local attack, the cluster system has an adjacency matrix A Then, the local damage occurs at time τ d The damage center and damage radius are randomly initialized. Then the damage adjacency matrix is ​​generated A d and the node vector S d nodes . FOM(τ d) can be calculated according to formula (1.4).

[0115] To restore cluster performance, a cre The maintenance team's support system cluster performs a series of maintenance actions during the MCRP period. This maintenance action sequence set is as follows:

[0116]

[0117] Maintenance Team c The action sequence set can be described as follows:

[0118]

[0119] Corrective maintenance actions are defined as a * t∣tc = a * tc∣t =( i , j ), which means that the maintenance team c Repair Node i With Node j After repairing the link, if the node j If the node is damaged, the maintenance team will continue to repair the node, otherwise continue to perform the next maintenance action. t The corrective maintenance agent uses the integrated AC-Net search algorithm to find the maintenance team that has just completed the last maintenance action. c Select maintenance action a * t∣tc .

[0120] The decision of the corrective maintenance agent can be described as follows:

[0121]

[0122] in S=[S f ,S c ] The current maintenance status. S f represents the status of repairs performed by the maintenance team, and S c Representative maintenance team c The status of the repairs performed.

[0123] (2) Time sequence of multiple maintenance teams

[0124] For having n creThe maintenance team's support system cluster has a time sequence and n cre The timing of each maintenance team.

[0125] Maintenance Team c The timing can be described as follows:

[0126]

[0127] where τ tc Indicates that the maintenance team c Complete its maintenance tasks a * tc∣t =( i , j ) time, τ tc The calculation method of is as follows:

[0128]

[0129] Where τ( a * tc∣t ) indicates execution of an action a * tc∣t The maintenance time consists of four situations. In the first situation, the maintenance team is at the node i The location of the node j Normal. In the second case, the maintenance team at the node i The location of the node j In the third case, the maintenance team is at the node i ' position, and the node j Normal. In the fourth case, the maintenance team is at the node i ' position, and the node j is in a damaged state. According to (1.2) and (1.3), τ( a * tc∣t ) is calculated as follows:

[0130]

[0131] The time series of the security system cluster can be described as follows:

[0132] Where τ t =0=τ d , τ t Complete actions on behalf of the maintenance team a *t∣tc = a * tc∣t =( i , j ) maintenance time.

[0133] (3) Elasticity analysis and rewards

[0134] In the above MCRP process, for maintenance actions a * t∣tc = a * tc∣t =( i , j ), if the node j is normal, then repair the link ( i , j ) after which the link is assumed to be operational again; if the node j is damaged, then repair the link ( i , j ) and nodes j Afterwards, it is assumed that the link and the node are back in operation. In the MCRP process, the FOM (τ t ) can be calculated according to formula (1.4). When the maintenance teams perform their respective maintenance actions, the FOM (τ t ) can be considered unchanged; whenever a maintenance team completes a maintenance action, the repaired links and nodes resume operation, and the FOM (τ t ) can be considered as a step increase. Therefore, FOM(τ t ) is a step function. After completing MCRP, FOM(τ t ) Integrate over the duration of MCRP to calculate the elastic loss of the entire repair process RL .

[0135] The repair sequence of damaged links and damaged nodes in the cluster, as well as the path planning of each maintenance team, will affect the step function FOM (τ d ) performance recovery curve, thus affecting elasticity loss RL Therefore, the optimized maintenance strategy needs to obtain a smaller elastic loss. RL , then the reward of the maintenance strategy executed by the entire MCRP process can be defined as the elastic loss RL The inverse proportional function of the inverse proportional function is the integral of the cluster FOM without elastic loss in the specified time interval, as shown below:

[0136]

[0137] (4) Actor-critic network training

[0138] For every moment t , Corrective Maintenance Agent uses AC-Net f θ ( S, A ) = ( p , v ) is a variant of the MCTS algorithm f ts ( S,p,v ) = ( π , a * ) provides search guidance, where p is the prior repair probability matrix, which is used to guide the tree search to select corrective repair actions, and v is a normalization parameter used to evaluate the reward of the current corrective repair action selection. The search algorithm uses the parameter p and v To output the search probability π , to select high-quality repair actions a * .

[0139] Every moment t The AC-Net training data is stored as (S t ,A, π t ,Z t ),in It's the action a * t∣tc α and β are empirical parameters. When the corrective maintenance agent completes the MCRP, an episode is completed and the data ( S , A , π , z ) is sampled from all time steps of this MCRP. AC-Net then uses the empirical data ( S , A , π , z ) to adjust its parameters θ The training goal is to maximize the prior repair probability p With tree search probability π similarity and minimize the prior value v and rewards z Therefore, by using gradient descent to adjust the parameters θ , the loss function includes cross-entropy losses (CELs) and mean squared errors (MSEs), as shown below:

[0140]

[0141] in C L2 is a constant used to control the level of L2 regularization to prevent overfitting.

[0142] For each episode, a small batch of training samples is randomly selected from the dataset consisting of all episode data, and then batch training is used to adjust the parameters θ . Adjust the parameters θ After that, AC-Net can improve the decision-making ability of the corrective maintenance agent in the next episode. In addition, the initialized AC-Net parameters θ is randomly generated. As the number of episodes increases, the trained AC-Net can provide stronger search guidance for tree search. After enough episodes, AC-Net has been well trained, and MSE and CEL tend to be stable. The restorative maintenance agent can provide effective decision support for cluster systems to solve CMD problems under large-scale local damage. In addition, when generating maintenance strategies for large-scale local damage during the operation phase of the cluster system, there is no need to train AC-Net again.

[0143] 1.2.2 AC-MCTS algorithm for corrective maintenance of cluster systems

[0144] When the multi-maintenance team CMDP process generates each maintenance action, that is, any maintenance action of the CMDP process t At this moment, the cluster system feature tensor X t Under the condition of, the result of estimating the value function and strategy function by AC-Net ( P t , V t ) as input, and the maintenance action search process is performed based on AC-MCTS, which mainly includes four steps: selection, expansion and evaluation, backtracking, and execution. Finally, the maintenance action search probability can be obtained. π and best maintenance actions .

[0145] 1.2.2.1 AC-MCTS Algorithm

[0146] In CMDP implementation tAt this moment, for the decision feature tensor X t , using the prior parameters obtained by the AC-Net prediction model ( P t , V t ) as input, and the maintenance action search operation is performed based on the MCTS method, and finally the improved maintenance state transition probability can be obtained. π and best maintenance actions a t * , whose expression is:

[0147]

[0148] MCTS search tree each tree node X All branches of X , a ), corresponding to all the next maintenance actions of this tree node a ∈ Action ( X ), each branch ( X , a ) stores a set of statistics as follows:

[0149]

[0150] in N ( X , a ) indicates the number of visits; W ( X , a ) represents the sum of total action values; Q ( X , a ) represents the average action value; P ( X , a ) indicates selecting a branch ( X , a )’s prior probability.

[0151] When the multi-maintenance team CMDP process generates each maintenance action, that is, at any time in the CMDP process t At this moment, the AC-MCTS algorithm performs the following four steps.

[0152] 1) Select

[0153] First, t Characteristic tensor of the moment cluster system X t As the root node of the Monte Carlo tree X 0(i.e. the cluster system status at each moment), and from the root node X 0 Start searching until L The leaf node at the end of the search tree is reached at any time X L Ends at l time( l < L ), based on the Upper Confidence Bound Apply to Tree (UCT) and the Maximum Upper Confidence Bound (UCB) algorithm, the best action under the current iteration conditions is selected according to the statistical data stored in each branch of the search tree. , whose expression is:

[0154] where c puct is a constant determined by the degree of MCTS search; a∈Action(X), Action(X) represents the action space;

[0155] In the early stage, this search strategy tends to select maintenance actions that usually have higher prior probabilities and lower visit times, but as the search proceeds, it tends to select "unexpanded" nodes, so that the MCTS search takes into account both breadth and depth.

[0156] 2) Extension and evaluation

[0157] The leaf node X L Add to the queue as the input of ResNet, and then the leaf node X L Expand, each edge (X L ,a) Initialize storage data:

[0158]

[0159] All positions in the queue are evaluated by the neural network and the leaf nodes are obtained. X L The strategic value of v , the search thread is locked until the policy evaluation is completed.

[0160] 3) Backtracking

[0161] After expansion and evaluation, the leaf nodes X L The strategic value of v Backtrack along the original path to the parent node. During the backtracking process, each branch of the search path The stored access count parameter is updated once, and its expression is:

[0162]

[0163] At the same time, search for the branch The total action value and average action value parameters of the maintenance action are also updated once, and their expressions are:

[0164]

[0165] 4) Execution

[0166] By iterating the above three steps 1000 times, the maintenance status probability is obtained π , and select the feature tensor of the cluster system at the current moment X t Global optimal maintenance action under the condition a * t Among them, the improved cluster system maintenance status probability π It is exponentially related to the number of visits, and its expression is:

[0167]

[0168] where τ is an empirical parameter that controls the search process.

[0169] Execute optimal maintenance actions a * t Afterwards, the feature tensor of the cluster system maintenance status is transformed from t time X t Transfer to t+ 1 moment X t+1 , the search tree will be X t+1 The new root node X 0 , and keep the nodes in the original search tree X t+1 All child nodes below it and their statistics, while discarding the branches of the remaining nodes at the same level as this node.

[0170] In summary, the AC-MCTS search algorithm can be expressed as f ts ( S,p,v ) = ( π , a * ).

[0171] 1.2.2.2Actor-Critic Neural Network Prediction Model

[0172] AC-Net f θ ( S, A ) = ( p , v ) is a dual-output residual neural network (ResNet), which is used as a maintenance strategy prediction model. The network parameters are randomly initialized. θ 0 .

[0173] At every moment t , AC-Net f θ The input is the current repair state before damage S t = [ S f t , S c tc ] and the adjacency matrix A . S f t It can be written as follows:

[0174]

[0175] Among them A f t is the time step of MCRP t The adjacency matrix at , if , then A f t = A f 1 =A damage Damage. The historical adjacency matrix is ​​necessary because the restorative maintenance agent cannot fully observe all state information from the current adjacency matrix, because MCRP is a sequential decision process. At the same time, S c tc It can be written as follows:

[0176]

[0177] Among them A c tc It's the maintenance team c Time step of the repair process t c If the adjacency matrix of , then A c tc= A 1 c =A d .

[0178] The output of the Actor module and the Critic module is the prior repair probability matrix P t Sum V t .matrix P t is a N × N The adjacency matrix, whose elements P ij Yes Select Repair Action a = ( i , j ) probability. V t is a standardized parameter used to evaluate the reward of the current corrective repair action selection.

[0179] This AC-Net in the restorative repair agent uses a residual module consisting of residual blocks of convolutional layers with ReLU functions to process input features. S t , and the input features A is a constant matrix and is processed by a multi-layer module consisting of fully connected layers. The residual module and the multi-layer module include shared layers and are merged into two independent modules, namely policy and value. In addition, the policy and value modules take their inputs from the shared layers and output the prior repair probability using fully connected layers and sigmoid functions. P t Sum V t .

[0180] The ResNet network structure includes convolution module, residual module, batch normalization module, ReLU function module, etc. Input feature tensor X It is processed by a deeply stacked module, which consists of a single convolution module and an intermediate layer module that deeply stacks multiple residual modules:

[0181] a) Separate convolutional modules:

[0182] ① A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0183] ② ReLU function layer;

[0184] b) Intermediate layer module: The intermediate layer of the deep neural network is constructed by stacking residual modules (taking 9 layers of stacked residual modules as an example). The structure of the residual module is as follows: Figure 2As shown, it includes the following structures:

[0185] ① A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0186] ② Batch normalization layer;

[0187] ③ ReLU function layer;

[0188] ④ A convolutional layer consisting of 256 3×3 filters with a stride of 1;

[0189] ⑤ Batch normalization layer;

[0190] ⑥ Addition module: Add the input feature tensor of the residual module to the output feature tensor of the previous layer;

[0191] ⑦ ReLU function layer;

[0192] The output of the deeply stacked residual module is divided into two parts: the policy module and the value module:

[0193] c) Strategy Module:

[0194] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0195] ② ReLU function layer;

[0196] ③ Fully connected output layer: outputs a feature vector of size m×n, corresponding to the logit of the prior maintenance state transition probability p of the cluster;

[0197] d) Value Module:

[0198] ① A convolutional layer consisting of 2 1×1 filters with a stride of 1;

[0199] ② A linear fully connected layer with a size of 256;

[0200] ③ ReLU function layer;

[0201] ④ Linear fully connected layer;

[0202] ⑤ Sigmoid output layer: outputs a scalar value in the interval (0,1).

[0203] The expression of the above ResNet prediction model is f θ ( X ) = ( p , v ), θ is the network parameter of ResNet, the initial network parameter θ 0Obtained by random initialization.

[0204] 1.3 Analysis of post-disaster emergency repair cases

[0205] In the case analysis part, the present invention selected a large-scale distributed power system cluster for case study, and took the power system cluster in a certain regional power grid as the case object, and analyzed the performance of the power system cluster in a certain regional power grid under disasters through Python simulation, and generated a corrective maintenance strategy based on the proposed AC-MCTS algorithm, and then analyzed the recovery effect. In addition, the power system cluster in the regional power grid consists of 228 nodes and 612 links. For each maintenance team, considering the three types of actions related to post-disaster repair, namely repairing nodes, repairing links, and moving, based on the basic assumptions of the corrective maintenance decision problem, it can be known that the number of strategies is 2 228+612 =7.33×10 252 ; For multiple maintenance teams, their strategy spaces are coupled with each other, and the total number of strategies will increase further.

[0206] 1.3.1 Power system cluster analysis

[0207] Taking the distributed power system cluster in the power network as an example, the power system is abstracted into nodes, focusing on the interconnection between nodes, and building its complex network model. On this basis, the temporal dynamic characteristics of nodes and links in the power system cluster under large-scale damage scenarios are considered, including various operation and maintenance situations such as normal, damage, and repair. Figure 4 shown.

[0208] Taking a regional power grid as an example, we first comprehensively analyze the basic information of the nodes in the power system cluster in a regional power grid and the basic information of the links between nodes. As shown in Table 1, the node numbers and geographical location information of the power system cluster in a regional power grid are given. Table 2 gives the link information of the power system cluster in a regional power grid, that is, the adjacency matrix information of the complex network model of the power system cluster. Based on the above information, the complex network modeling method can be used to establish a complex network model of the power system cluster in a regional power grid. G .

[0209] Table 1 Basic information of power system cluster nodes in a certain area power grid

[0210]

[0211] Table 2 Basic information of power system cluster links in a certain area power grid

[0212]

[0213] 1.3.2 Post-disaster recovery analysis

[0214] 1.3.2.1 Analysis of sabotage cases

[0215] Based on the complex network model of the power system cluster in a certain area of ​​the power grid in the previous section G Modeling, continue the operation and maintenance situation of the power system cluster under disasters, consider using a circular area to represent earthquake as an example, after randomly generating the damage center and damage radius of the circular area, the power system cluster in a certain area of ​​the power grid G The destructive situation Figure 5 As shown, pink nodes represent failed nodes.

[0216] The maintenance task is performed in parallel by multiple maintenance teams with different maintenance capabilities. In the present invention, five maintenance teams with different V edge and V move , and divide the nodes into three levels: the node weight of the first level node is w ∈(0, 2]; the node weight of the second-level node is w ∈(2, 4]; the node weight of the third-level node is w ∈(4, +∞). Each maintenance team has different MTTR : MTTR 1st , MTTR 2nd and MTTR 3rd ,The maintenance capability information of each maintenance team is shown in Table 3. In addition, the maintenance capability of the ,maintenance team can also be defined according to the actual situation of the ,maintenance team.

[0217] Table 3 Maintenance team information

[0218]

[0219] Figure 5 Four different scale damage cases of the power system cluster are analyzed, and the damage information of each damage case is shown in Table 4. Figure 5 The four MCRPs in are performed by the same maintenance team group, which consists of maintenance team 1, maintenance team 2 and maintenance team 3.

[0220] Table 4 Related information of different damage cases

[0221]

[0222] 1.3.2.2 Maintenance strategy and flexibility analysis

[0223] for Figure 5 For each failure case in damage=200h, a large-scale local damage event occurs, which causes a sharp decline in the functional performance of the power system cluster. Then corrective maintenance work is performed on the power system cluster to recover from the large-scale local damage, after which the functional performance is improved and restored at time τ restoration Reaching FOM threshold FOM thr = 0.96, set τ thr The parameter can be set according to the actual situation. Generally speaking, different MCRPs have different τ restoration , the elasticity analysis of the above four MCRPs is as follows Figure 5 As shown, its elastic loss RL As shown in Table 5.

[0224] Table 5 Elasticity loss of corrective maintenance by multiple maintenance teams

[0225]

[0226] 1.3.2.3 AC-Net Learning and Training Analysis

[0227] The proposed AC-MCTS method is applied to train the DRL agent using a server with an Intel i9 7980XE CPU and four RTX2080 TI-A11G GPUs. The training starts with a completely random AC-Net and lasts for about 30 hours without intervention. During the training process, each MCTS uses 800 simulations to perform 10,000 MCRPs, which is equivalent to nearly 0.2 seconds of thinking time for each repair action selection. AC-Net uses the dataset generated by MCRP to train parameters and applies min-batches for update.

[0228] Figure 6 The mean square error and cross entropy parameter training of AC-Net after each MCRP simulation are analyzed. MSE is between the actual results z and AC-Net predicted value v The goal is to predict the elastic loss RL CEL represents the AC-Net prediction prior probability p and MCTS strategy π The goal is to predict the strategy π Although the mathematical models of MSE and CEL are different, AC-Net is trained using the same training dataset and the same loss function, Figure 6It can be seen that in some training time periods, the two curves have the same trend. In addition, some empirical parameters: α=2, β=4, c L2 = 10 -4 , c puct = 5. After about 8000 MCRPs, MSE and CEL tend to be stable, and then AC-Net can effectively guide MCTS to select maintenance actions for multiple teams.

[0229] 1.3.3 Discussion and Analysis

[0230] The main goal of post-disaster recovery of power system clusters is to generate feasible multi-team corrective maintenance strategies. The present invention conducts an extended analysis of multi-team corrective maintenance strategies to determine the impact of different factors on the optimal strategy.

[0231] 1.3.3.1 Analysis of decision-making methods

[0232] This paper proposes an efficient AC-MCTS method for post-disaster recovery of cluster systems. In addition, the heuristic hybrid game (HHG) method and the DQN method are also used to solve the problem. The three methods are analyzed separately by running Python programs for each method to solve the above four post-disaster damage situations. AC-MCTS gives the optimal solution with better RL than other methods, such as Figure 7 As shown in Figure 2. In the early MCRP stage under each damage situation, the DQN method and the HHG method can achieve better FOM than AC-MCTS because these two methods select each repair action with the goal of achieving the best FOM improvement in the current repair stage. However, they may also lead to local optimality, such as in the middle and late stages of MCRP when multiple repair team members have to spend more time moving. According to Figure 7 As shown in the results, AC-MCTS achieves better performance in a shorter time than other methods. RL , because it uses the value predicted by DNN, aiming to achieve better RL Therefore, the results show that AC-MCTS generates the optimal strategy for post-disaster recovery.

[0233] The efficiency of these methods is analyzed in Table 6. According to these results, the solving speeds of AC-MCTS, DQN, and HHG are stable, and they do not decrease exponentially with the increase in the number of destroyed nodes and links. However, the solving speed of the HHG method decreases significantly with the increase in the degree of destruction.

[0234] Table 6 Running time of different methods (seconds)

[0235]

[0236] 1.3.3.2 Maintenance teams of different sizes

[0237] Keeping the same post-disaster damage situation ( Figure 5 In the case of post-disaster damage scenario 1), four methods are used to consider three, four, and five different repair team sizes. The corrective repair results of the three repair teams (repair teams 1, 2, and 3) are shown in Figure 1. Figure 7 As shown in (a) of the figure, the corrective maintenance of the four maintenance teams (teams 1, 2, 3 and 4) and the five maintenance teams (members 1, 2, 3, 4 and 5) are as follows: Figure 8 The elasticity loss under different numbers of maintenance teams is shown in Table 7. As the number of maintenance teams increases, the elasticity loss RL and total repair time is reduced, the corrective maintenance agent also obtains better RL Therefore, these results show that the Restorative Repair Agent exhibits stable performance for the disaster recovery problem with different numbers of repair teams, because the number of repair teams is randomly generated during the Restorative Repair Agent training process.

[0238] Table 7 Analysis of elasticity loss for different maintenance team sizes

[0239]

[0240] 1.3.3.3 Maintenance teams at different levels of expertise

[0241] Maintaining the same post-disaster damage status ( Figure 5 In the post-disaster damage case 1), based on the maintenance strategy generated by the restorative maintenance agent, the difference in cluster system performance recovery after the maintenance strategy is executed by the maintenance teams with different business levels is analyzed. The elasticity analysis of the cluster system after the maintenance team 1, 2 and 3 performs the restorative maintenance task is shown in the figure. Figure 5 As shown in (MCRP and resilience analysis of post-disaster damage case 1), the relevant analysis of the support system cluster composed of maintenance teams 2, 3 and 4 and maintenance teams 3, 4 and 5 are shown in Fig. 9As shown in (a) and (b). Considering that the maintenance teams in the guarantee system cluster have different business capabilities, the elasticity loss of the cluster system recovery process is shown in Table 8. For the same corrective maintenance agent, the generated maintenance strategy achieves similar elasticity loss in almost the same total repair time. Therefore, these results show that the proposed method shows stable performance for maintenance teams with different business capabilities.

[0242] Table 8 Analysis of elasticity loss for maintenance teams with different business capabilities

[0243]

Claims

1. A deep Actor-Critic architecture system for post-disaster restorative maintenance decision-making of multi-teams in large-scale distributed power system clusters, characterized by: The system comprises: A simulation environment module for generating local damage and network cluster system states; A corrective maintenance agent module executes the constrained Markov decision process (CMDP) of multiple maintenance teams and generates a series of maintenance actions based on the AC-MCTS algorithm. The corrective maintenance agent module uses the search algorithm integrated with AC-Net to select maintenance actions for each maintenance team, and extracts the current recovery status of the network cluster system and the time series characteristics of multiple maintenance teams through AC-Net to estimate the value function and policy function of the maintenance action. A resilience analysis and reward module, which is used to evaluate the resilience loss of system recovery according to the execution of the maintenance strategy and generate the corresponding reward value; An Actor-Critic network training module that adjusts AC-Net parameters through training datasets to provide more effective maintenance action search support for the AC-MCTS algorithm; One has n cre The maintenance team support system cluster performs a series of maintenance actions during the multi-maintenance team maintenance process. The maintenance action sequence set is as follows: ； The action sequence set of maintenance team c is described as follows: ； Corrective maintenance actions are defined as a * t∣tc = a * tc∣t =( i , j ), which means that the maintenance team c repairs the node i With Node j After repairing the link, if the node j If it is a damaged node, the maintenance team continues to repair the node, otherwise it continues to perform the next maintenance action; at each time t, the restorative maintenance agent uses the search algorithm integrated with AC-Net to select a maintenance action for the maintenance team c that has just completed the last maintenance action a * t∣tc ; The decision making of the corrective maintenance agent is described as follows: ； in S= [ S f , S c ] is the current maintenance status, S f The status of repairs performed on behalf of the maintenance team, S c And the maintenance status performed on behalf of maintenance team c; For having n cre The maintenance process of multiple maintenance teams has a support system cluster with a time sequence and n cre The timing of each maintenance team; Maintenance Team c The timing description is as follows: ； where τ tc Indicates that the maintenance team c Complete its maintenance tasks a * tc∣t =( i , j ) time, τ tc The calculation method of is as follows: ； Where τ( a * tc∣t ) indicates execution of an action a * tc∣t The maintenance time consists of four situations; in the first situation, the maintenance team is at the node i The location of the node j Normal; in the second case, the maintenance team is at the node i The location of the node j in a state of destruction; in the third case, the repair team is at a node i ' position, and the node j Normal; in the fourth case, the maintenance team is at a node i ' position, and the node j in a state of destruction; τ( a * tc∣t ) is calculated as follows: ； Repair damaged nodes i The average time of MTTR i The time to repair each edge is expressed as follows: ； in v edge Indicates the speed at which a broken link is repaired; i Move to j The time is expressed as follows: ； in v move Indicates the movement speed of the maintenance team; The time series of the guarantee system cluster is described as follows: ； Where τ t =τ d , τ t Complete actions on behalf of the maintenance team a * t∣tc = a * tc∣t =( i , j ) maintenance time.

2. The deep Actor-Critic architecture system according to claim 1, characterized in that: The multi-maintenance team constrained Markov decision process CMDP executed by the corrective maintenance Agent module includes: A multi-repair team maintenance process module, in the event of partial system damage, restores the performance of the cluster system through a series of maintenance actions performed in sequence by each maintenance team; A multi-maintenance team time sequence module determines the maintenance timing and path planning of multiple maintenance teams to optimize the time efficiency of system recovery.

3. The deep Actor-Critic architecture system according to claim 1, characterized in that: The elasticity analysis and reward module further includes: An elastic loss calculation unit calculates the elastic loss RL of the repair process by integrating the performance recovery curve of the cluster system during the MCRP period; A reward calculation unit, based on elastic loss RL, defines an inverse proportional function as the reward of the maintenance strategy and optimizes the maintenance strategy to minimize the elastic loss.

4. The deep Actor-Critic architecture system according to claim 3, characterized in that: In the MCRP process, for maintenance actions a * t∣tc = a * tc∣t =( i , j ), if the node j is normal, then repair the link ( i , j ) after which the link is assumed to be operational again; If the node j is damaged, then repair the link ( i , j ) and nodes j Afterwards, assuming that the link and the node are back up and running; In the MCRP process, the FOM (τ t ) is calculated according to the following formula; ； Where E*(G) is the average efficiency before the local attack event occurs, E τ (G) is the average efficiency at time τ; When the maintenance teams perform their respective maintenance actions, the FOM (τ t ) is considered unchanged; whenever a maintenance team completes a maintenance action, the repaired links and nodes resume operation, and the FOM (τ t ) is considered as a step improvement; after completing MCRP, the FOM(τ t ) Integrate over the duration of MCRP to calculate the elastic loss of the entire repair process RL ; Elasticity loss RL The calculation process is as follows: Assume that the local attack event occurs at time τ d , which causes a sharp drop in cluster functional performance; then, the cluster performs corrective maintenance actions to recover from the local attack, after which its functional performance improves and at time τ d Reaching FOM threshold FOM thr , using the following elastic loss RL To quantify the loss suffered by the cluster due to a local attack: ； Optimized maintenance strategy requires less elasticity loss RL , then the reward of the maintenance strategy executed by the entire MCRP process is defined as the elastic loss RL The inverse proportional function of the inverse proportional function is the integral of the cluster FOM without elastic loss in the specified time interval, as shown below: 。 5. The deep Actor-Critic architecture system according to claim 1 or 2, characterized in that: The Actor-Critic network training module includes: An AC-Net training unit, which adjusts the parameters θ of AC-Net by using the empirical dataset (S, A, π, z), adopts a loss function including cross entropy loss and mean squared error to maximize the similarity between the prior repair probability p and the tree search probability π, and minimize the error between the prior value v and the reward z; A batch training unit randomly selects a small batch of training samples from the event data of all time steps and uses the batch training method to adjust the AC-Net parameters to improve the decision-making ability of the corrective maintenance agent.

6. The deep Actor-Critic architecture system according to claim 5, characterized in that: For every moment t , Corrective Maintenance Agent uses AC-Net f θ ( S, A ) = ( p , v ) is a variant of the MCTS algorithm f ts ( S,p,v ) = ( π , a * ) provides search guidance, where p is the prior repair probability matrix, which is used to guide the tree search to select corrective repair actions, and v is a normalization parameter used to evaluate the return of the current corrective repair action selection; the search algorithm uses the parameter p and v To output the search probability π , to select high-quality repair actions a * ; Every moment t The AC-Net training data is stored as ( S t , A , π t , z t ),in It's the action a * t∣tc reward; α and β are experience parameters, ; When the corrective maintenance agent completes the MCRP, an episode is completed and the data ( S , A , π , z ) is sampled from all time steps of this MCRP; AC-Net then uses the empirical data ( S , A , π , z ) to adjust its parameters θ ; The training goal is to maximize the prior repair probability p With tree search probability π similarity and minimize the prior value v and rewards z The error between ; by using gradient descent to adjust the parameters θ , the loss function includes cross-entropy losses (CELs) and mean squared errors (MSEs), as shown below: ； Among them C L2 is a constant used to control the L2 regularization level to prevent overfitting; For each episode, a small batch of training samples is randomly selected from the dataset consisting of all episode data, and then batch training is used to adjust the parameters θ ; Adjust parameters θ Afterwards, AC-Net improves the decision-making ability of the corrective maintenance agent in the next episode.

7. The deep Actor-Critic architecture system according to claim 6, characterized in that: The AC-Net parameter θ is initialized by random generation and gradually optimized during the training process of multiple episodes so that the MSE and CEL tend to be stable, and finally a decision support system is formed that can effectively solve the CMD problem under large-scale local damage.

8. The deep Actor-Critic architecture system according to claim 7, characterized in that: The maintenance strategy generated by the system during the operation phase of the cluster system can directly provide the cluster system with a maintenance strategy for coping with large-scale local damage without the need to train AC-Net again.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the deep Actor-Critic architecture system described in any one of claims 1 to 8 is implemented.

Images