Server cluster-oriented operation and maintenance exception self-healing method and system

Abstract

The invention relates to the technical field of server cluster operation and maintenance management, and discloses a server cluster-oriented operation and maintenance exception self-healing method and system, and the method comprises the steps: collecting multi-source operation and maintenance data of a server cluster, and carrying out the data preprocessing and multi-modal feature fusion; carrying out topological relation extraction and structure learning, identifying the change of a topological dependency relation and carrying out incremental updating; performing server node state prediction, and identifying abnormal nodes; performing causal inference and abnormal root cause positioning; rule reasoning is carried out, and the abnormal type of the root cause node is identified; constructing a strategy network, verifying and deploying the strategy network, and making a decision for the diagnosis identifier; carrying out multi-dimensional risk assessment and grading processing, monitoring an execution effect in real time and carrying out failure rollback; according to the method, a complete closed loop from anomaly detection, root cause positioning to strategy execution is formed, and an effective solution is provided for intelligent operation and maintenance of a large-scale server cluster.

Description

A self-healing method and system for operation and maintenance anomalies in server clusters Technical Field

[0001] This invention relates to the field of server cluster operation and maintenance management technology, and more specifically, to a self-healing method and system for server cluster operation and maintenance anomalies. Background Technology

[0002] With the rapid development of cloud computing and distributed systems, server clusters are expanding in scale and system complexity is increasing dramatically, posing unprecedented challenges to operation and maintenance management. Traditional operation and maintenance methods mainly rely on manual monitoring and handling, which are insufficient to cope with complex anomalies in large-scale cluster environments. Existing anomaly detection methods are typically based on a single data source, such as performance metrics or log information, lacking comprehensive utilization of multi-source heterogeneous data, resulting in insufficient accuracy and comprehensiveness in anomaly detection. Furthermore, traditional methods often ignore the topological dependencies between nodes when processing time-series data, failing to effectively capture the propagation patterns of anomalies within the cluster, and easily leading to false positives and false negatives.

[0003] In the area of ​​root cause localization, existing technologies mainly employ rule-based expert systems or simple statistical analysis methods, which perform poorly when dealing with anomalies in complex distributed systems. Due to a lack of in-depth understanding of the system's dynamic topology, existing methods struggle to accurately identify the true root cause of anomalies, often only pinpointing superficial phenomena, resulting in remedial measures that only address the symptoms, not the underlying cause. Furthermore, traditional root cause analysis methods lack the ability to model temporal causal relationships, failing to effectively distinguish between causal and correlational relationships between anomalous nodes, thus affecting the accuracy of root cause localization.

[0004] Regarding the generation and execution of remediation strategies, existing automated operation and maintenance tools are mainly based on predefined scripts and rules, lacking intelligent decision-making capabilities. These tools cannot dynamically generate optimal remediation strategies based on specific anomalies and system states, often adopting a one-size-fits-all approach, which not only has limited remediation effects but may also negatively impact system stability. Furthermore, existing methods lack effective risk assessment mechanisms, failing to evaluate the potential impact on the system before remediation operations are executed, posing a risk of exacerbating anomalies or triggering new failures due to improper remediation operations. Summary of the Invention

[0005] This invention provides a self-healing method and system for operational anomalies in server clusters, solving the technical problems of low anomaly detection accuracy, difficulty in root cause localization, lack of intelligent decision-making in repair strategies, and high cost of manual intervention in related technologies.

[0006] This invention provides a self-healing method for operational anomalies in server clusters, comprising the following steps: collecting multi-source operational data from the server cluster, performing data preprocessing and multimodal feature fusion to obtain a comprehensive representation vector sequence; acquiring service call logs and network traffic data from the server cluster, combining the comprehensive representation vector sequence to extract topological relationships and learn structures, identifying changes in topological dependencies and incrementally updating to obtain a dynamic topology graph sequence; based on the comprehensive representation vector sequence and the dynamic topology graph sequence, predicting server node states, identifying abnormal nodes and initially classifying them to obtain an abnormal node set and preliminary classification results of abnormal types; based on the dynamic topology graph sequence and the abnormal node set, performing causal inference and root cause localization to obtain root cause node identifiers; based on the root cause node identifiers and preliminary classification results of abnormal types, performing rule reasoning to identify the abnormal types of root cause nodes to obtain diagnostic identifiers; acquiring historical operational data from the server cluster, constructing a policy network, validating and deploying the policy network, making decisions based on the diagnostic identifiers to obtain a set of repair policy candidates; based on the set of repair policy candidates, performing multi-dimensional risk assessment and graded processing, monitoring execution effects in real time and performing failure rollback to obtain a repair execution result report and state recovery record.

[0007] In a preferred embodiment, the data preprocessing includes: running a data acquisition agent on each server node and automatically adjusting the acquisition frequency according to CPU utilization; obtaining a performance indicator time series based on the original performance indicator data stream using a sliding window algorithm; obtaining a structured log event sequence based on the original log event stream using template matching technology; and obtaining a deduplicated alarm list based on the original alarm message stream using a deduplication algorithm.

[0008] In a preferred embodiment, the multimodal feature fusion includes: using a temporal convolutional network to encode features based on a performance index matrix to obtain a performance modality feature vector; performing semantic encoding based on a structured log event sequence to obtain a log modality feature vector, and calculating the attention weight between each event in the structured log event sequence and other events using a self-attention module; using a graph convolutional network to encode a deduplicated alarm list to obtain an alarm modality feature vector, and constructing an alarm association graph based on the spatiotemporal proximity and type correlation of alarms; using a recurrent neural network to encode a configuration modality feature vector based on a configuration change history; calculating cross-modal attention weights using a multi-head attention mechanism, and performing weighted fusion based on the attention weights and the performance modality feature vector, log modality feature vector, alarm modality feature vector, and configuration modality feature vector to obtain a comprehensive representation vector sequence.

[0009] In a preferred embodiment, the step of obtaining the dynamic topology graph sequence includes: based on service call log data in the server cluster, statistically analyzing the call frequency and average response time between each pair of services within a fixed time window to obtain a service dependency graph; based on network traffic data in the server cluster, using traffic statistics and pattern recognition algorithms to obtain the communication relationships and traffic characteristics between server nodes, identifying the application layer protocol of communication based on port number and protocol type to obtain a node communication graph; based on the service dependency graph, node communication graph, and comprehensive representation vector sequence, using a graph structure learning algorithm to obtain a complete topology graph and infer potential implicit dependencies; and using a sliding window mechanism and incremental update algorithm to obtain a dynamically updated topology graph sequence, comparing the topology graph of the new window with the topology graph of the previous window to identify changes in the topology graph.

[0010] In a preferred embodiment, the steps of obtaining the abnormal node set and preliminary classification results of abnormal types include: constructing a graph spatiotemporal convolutional network based on the comprehensive representation vector sequence and the dynamic topology graph sequence; using a sequence prediction method to obtain the state prediction value of each node based on the graph spatiotemporal convolutional network and the node representation vectors and topology graph at the current and historical moments; using a deviation metric and scoring mapping method to obtain the abnormal score of each node based on the state prediction value and the actually observed node state value; using statistical analysis and adaptive adjustment algorithms to obtain an adaptive threshold based on the abnormal score distribution at historical moments and the current cluster operating status; and using a threshold comparison and classification method based on the node abnormal score and the adaptive threshold to obtain the abnormal node set and preliminary classification results of abnormal types.

[0011] In a preferred embodiment, the step of obtaining the root cause node identifier includes: using the Granger causality test method to obtain the temporal causal relationship between pairs of anomalous nodes based on the set of anomalous nodes and the historical state time series of the nodes; using a comprehensive scoring method to obtain the causal strength score based on the causal relationship and significance level of the node pairs; using a graph construction method to obtain a causal dependency graph describing the causal dependence relationship between anomalous nodes based on the causal dependency graph; using a topology analysis method to obtain the starting node of the anomalous propagation chain as a candidate root cause node based on the priority list of candidate root cause nodes and the state anomality score; and using a comprehensive judgment method to obtain the root cause node identifier and the anomalous propagation path based on the comprehensive judgment method.

[0012] In a preferred embodiment, the step of obtaining the diagnostic identifier includes: collecting the experience of operation and maintenance experts and converting it into IF-THEN rule form; matching abnormal features with rules to trigger diagnostic candidates; designing an abnormal classification model, using a fully connected neural network to obtain pattern recognition conclusions; fusing rule reasoning and pattern recognition conclusions; and using a confidence threshold to determine the diagnostic identifier based on the fusion result.

[0013] In a preferred embodiment, the step of obtaining the repair strategy candidate set includes: obtaining a digital twin simulation system based on historical operation and maintenance data and system configuration parameters of the server cluster environment using a data-driven modeling method; obtaining a formal description of self-healing strategy learning based on the digital twin simulation system using a Markov decision process framework; obtaining an optimal policy network based on the Markov decision process model and the digital twin simulation system using a deep reinforcement learning algorithm; obtaining a policy network that satisfies safety constraints using a constrained reinforcement learning method based on the risk characteristics of the repair operation; and obtaining the repair strategy candidate set by combining diagnostic identifiers and current cluster state information using a policy inference method.

[0014] In a preferred embodiment, the steps of obtaining the repair execution result report and the state recovery record include: based on each candidate strategy in the repair strategy candidate set, using a multi-dimensional risk assessment model to obtain a risk level classification for each strategy; based on the repair strategies assessed as low-risk, using an automatic execution queue scheduling method to obtain an automatic execution task queue sorted by priority; based on the repair strategies assessed as medium-risk and high-risk, using a repair suggestion generation and approval process mechanism to obtain a repair suggestion report awaiting manual confirmation; based on the repair strategies to be executed, using state machine control and operation decomposition technology to obtain the repair execution process, decomposing the action sequence of the repair strategy into atomic operation steps; based on the execution process of the repair operation, using real-time monitoring and effect evaluation technology to obtain a repair execution result report; and when a repair operation failure or negative impact is detected, using a rollback recovery mechanism to obtain a state recovery record.

[0015] In a preferred embodiment, a self-healing system for server cluster operation and maintenance anomalies, used to execute the aforementioned self-healing method for server cluster operation and maintenance anomalies, includes: a data acquisition module for acquiring multi-source operation and maintenance data of the server cluster, performing data preprocessing and multi-modal feature fusion to obtain a comprehensive representation vector sequence; a topology learning module for acquiring service call logs and network traffic data of the server cluster, combining the comprehensive representation vector sequence to extract topological relationships and learn structures, identifying changes in topological dependencies and incrementally updating to obtain a dynamic topology graph sequence; and an anomaly detection module for predicting server node states based on the comprehensive representation vector sequence and the dynamic topology graph sequence, identifying and initially classifying abnormal nodes to obtain anomaly detection results. The system comprises the following modules: a set of constant nodes and preliminary classification results of anomaly types; a root cause localization module, which performs causal inference and anomaly root cause localization based on the dynamic topology graph sequence and the set of anomaly nodes, and obtains root cause node identifiers; a diagnostic reasoning module, which performs rule reasoning based on root cause node identifiers and preliminary anomaly type classification results, identifies the anomaly types of root cause nodes, and obtains diagnostic identifiers; a policy generation module, which acquires historical operation and maintenance data of the server cluster, constructs a policy network, verifies and deploys the policy network, makes decisions based on diagnostic identifiers, and obtains a set of repair policy candidates; and a self-healing execution module, which performs multi-dimensional risk assessment and graded processing based on the set of repair policy candidates, monitors the execution effect in real time and performs failure rollback, and obtains a repair execution result report and a status recovery record.

[0016] The beneficial effects of this invention are as follows: By constructing a multimodal data fusion anomaly detection mechanism, this invention can comprehensively utilize multi-source heterogeneous data such as performance indicators, log events, alarm information, and configuration changes, thereby improving the accuracy and comprehensiveness of anomaly detection. By employing a graph spatiotemporal convolutional network to simultaneously model time-series features and topological spatial relationships, it effectively captures the propagation patterns of anomalies in the cluster, reducing false positives and false negatives. Through Granger causality testing and causal dependency graph construction, it achieves accurate anomaly root cause localization, accurately identifying the true source of the anomaly and avoiding the limitations of traditional methods that can only locate surface phenomena.

[0017] This invention establishes a deep reinforcement learning policy network capable of dynamically generating optimal repair strategies based on specific anomalies and system states, breaking free from the constraints of traditional predefined rules and achieving truly intelligent operation and maintenance. Policy verification and multi-dimensional risk assessment are conducted through a digital twin simulation system, ensuring the safety and effectiveness of repair operations. A tiered processing mechanism and a failure rollback mechanism further reduce operational risks, decrease the need for manual intervention, and improve operational efficiency and system reliability. This forms a complete closed loop from anomaly detection and root cause localization to policy execution, providing an effective solution for the intelligent operation and maintenance of large-scale server clusters. Attached Figure Description

[0018] Figure 1 is a main flowchart of a self-healing method for operation and maintenance anomalies of server clusters according to the present invention; Figure 2 is a detailed flowchart of a self-healing method for operation and maintenance anomalies of server clusters according to the present invention; Figure 3 is a block diagram of a self-healing system for operation and maintenance anomalies of server clusters according to the present invention. Detailed Implementation

[0019] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0020] At least one embodiment of the present invention discloses a self-healing method for operational anomalies in server clusters, as shown in Figures 1 and 2, comprising the following steps: Step 1, collecting multi-source operational data from the server cluster, performing data preprocessing and multimodal feature fusion to obtain a comprehensive representation vector sequence; Step 1.1, deployment and configuration of a lightweight data acquisition agent; based on the topology and node distribution of the server cluster, a containerized deployment method is adopted to obtain a data acquisition agent instance running on each server node. The data acquisition agent adopts an event-driven architecture design, obtaining node operational data through system call interfaces and application programming interfaces. The data acquisition agent supports running on server nodes based on ARM architecture CPUs and is compatible with domestic operating system environments such as Galaxy Kylin server operating system and UnionTech UOS. The acquisition agent dynamically adjusts its own resource usage according to the node's resource status, automatically reducing the acquisition frequency when the node's CPU utilization exceeds a set threshold, and increasing the acquisition frequency to obtain more granular data when the node's resources are sufficient. The acquisition agent determines the priority of data acquisition based on the importance of the business carried by the node, using a higher acquisition frequency and a more comprehensive acquisition scope for nodes running core businesses, and a lower acquisition frequency for nodes running non-core businesses.

[0021] Step 1.2: Preprocessing and structuring of multi-source operation and maintenance data; Based on the raw performance indicator data stream obtained by the data acquisition agent output in Step 1.1, a sliding time window aggregation algorithm is used to obtain the performance indicator time series. The raw performance indicators include monitoring items such as CPU utilization, memory usage, disk read / write speed, network reception rate, network transmission rate, number of processes, and number of threads, with a collection granularity at the second level. Time window aggregation segments continuous raw data points according to fixed time windows, calculating statistical characteristic values ​​of data points within each time window, including mean, maximum, minimum, standard deviation, and trend. These statistical characteristic values ​​are used as the aggregated representation of that time window. During the aggregation process, abrupt change points in performance indicators are identified. Abrupt change point is defined as a time point where the data value shows a significant jump relative to the preceding and following windows, identified by calculating the degree of difference in statistical values ​​between adjacent windows. For time windows containing abrupt change points, the window length is reduced to retain the time precision information of the abrupt change. The aggregated performance indicator time series is organized according to node identifiers and indicator types, forming a two-dimensional performance indicator matrix. The rows of the matrix represent different time steps, and the columns represent different indicator types for different nodes.

[0022] Based on the raw log event stream obtained by the data acquisition agent output in step 1.1, log template matching and information extraction techniques are used to obtain a structured log event sequence, including timestamps, log levels, event types, and key parameters. The raw log event stream includes system logs and application logs, with various heterogeneous formats such as text, JSON, and XML. Log parsing first identifies the format, determining the log format type based on its structural characteristics, and then calling the appropriate parser for each format. For text format logs, a template pattern is automatically learned by analyzing a large number of historical logs. The template pattern consists of fixed constants and changing variables. Newly arrived logs are matched with the learned templates to find the most similar template, and key parameter values ​​are extracted from the logs based on the position of variables in the template. For JSON and XML format logs, the required information items are extracted based on predefined field mapping relationships. Timestamp information is extracted from the parsed logs and uniformly converted to a standard time format. Log level information is extracted, including five levels: debug, info, warning, error, and critical. The event type is determined based on the keywords and template type of the log content. Event types include service startup, service shutdown, configuration loading, request processing, exception handling, and resource alerts. The extracted structured information is organized into log event objects and arranged in chronological order to form a structured log event sequence.

[0023] Based on the raw alarm message stream obtained by the data acquisition agent output in step 1.1, a deduplication algorithm based on content similarity and a clustering algorithm based on spatiotemporal correlation are used to obtain a deduplicated alarm list. The raw alarm message stream originates from various monitoring tools deployed in the server cluster, including host monitoring alarms, application monitoring alarms, network monitoring alarms, and business monitoring alarms. Identical or similar anomalies can trigger multiple monitoring tools to generate multiple alarm messages, leading to alarm storms and information redundancy. Alarm deduplication first extracts content features from the alarm messages, extracting attributes such as the source node, alarm time, alarm level, alarm type, and alarm description. The content similarity between alarm pairs is calculated, and the similarity metric comprehensively considers the distance between source nodes, the proximity of time, the similarity of types, and the text similarity of descriptions. For alarm pairs with a similarity exceeding a set threshold, they are judged as duplicate alarms, and the earliest alarm is retained as the representative alarm; other alarms are marked as duplicates and the number of duplicates is recorded. Based on deduplication, alarm clustering is performed, with the goal of grouping related alarms caused by the same root cause into one alarm cluster. Dense regions of alarms are identified in a multi-dimensional feature space of time, space, and type. Alarms within dense regions exhibit characteristics of temporal proximity, spatial adjacency, and type correlation, suggesting they are likely triggered by the same anomalous event. Each alarm cluster is treated as an independent anomalous event, with the representative alarm of the cluster serving as a description of the anomalous event. The size of the alarm cluster reflects the scope of the anomalous event's impact.

[0024] Based on the configuration file change events and configuration management interface call records monitored by the data acquisition agent output in step 1.1, version control and change tracking technologies are used to obtain a configuration change history, including change time, operator, and change content. The server cluster configuration includes multiple layers such as operating system configuration, application configuration, network configuration, and security configuration. Modifications to these configurations may lead to changes in system behavior or even anomalies. Configuration change tracking monitors file system events of configuration files, triggering change events when configuration files are modified, created, or deleted. After capturing a change event, the precise timestamp of the change and the user or process identifier that performed the change operation are recorded. The contents of the configuration files before and after the change are read, the specific changes are calculated, and newly added, deleted, and modified configuration items are identified. For configuration changes made through the configuration management interface, metadata information is obtained from the API call logs. The configuration change information is stored as a structured record, including fields such as node identifier, configuration type, change time, operator, content before change, content after change, and reason for change, forming a configuration change history in chronological order.

[0025] Step 1.3: Time Alignment and Index Construction of Multi-Source Data; Based on the performance indicator matrix, structured log event sequence, deduplicated alarm list, and configuration change history output in Step 1.2, a multi-source operation and maintenance data set is obtained using time benchmark alignment and multi-dimensional indexing techniques. Data from different data sources may have inconsistent time benchmarks, resulting in clock skew and network transmission delays. Time alignment selects a time synchronization server in the server cluster as the unified time benchmark source, which maintains synchronization with the standard time source via a network time protocol. For each piece of collected data, the data generation time and collection time are recorded, and the clock skew between the collection node and the time benchmark source is calculated. The timestamps of the data are corrected based on the clock skew, converting all data timestamps to a unified time benchmark. A multi-dimensional index structure is constructed based on time alignment, with dimensions including time, node, and data type. The time dimension index supports fast data querying by time range, the node dimension index supports fast location of all data related to a node by node identifier, and the data type dimension index supports fast filtering of specific categories of data by data type. The index structure controls index storage overhead while ensuring query efficiency.

[0026] Step 1.4, Feature Encoding of Multimodal Data: Based on the performance index matrix output in Step 1.2, a temporal convolutional network is used for feature encoding to obtain the performance modality feature vector. The temporal convolutional network adopts a causal convolutional structure, containing three one-dimensional convolutional layers with kernel sizes of 3, 5, and 7, and channel numbers of 64, 128, and 256, respectively. ReLU activation and residual connections are used. The network is trained using the mean squared error loss function, the Adam optimizer, a learning rate of 0.001, and a batch size of 32. Global average pooling is performed on the output layer to obtain a fixed-length feature vector. Multiple performance index encoded feature vectors from the same node are concatenated to form the performance modality feature vector for that node.

[0027] Based on the structured log event sequence output in step 1.2, a Transformer encoder is used for semantic encoding to obtain the log modality feature vector. The Transformer encoder consists of 6 layers, each containing a multi-head self-attention mechanism (8 attention heads) and a feedforward neural network (512 hidden layer dimensions), trained using the cross-entropy loss function and the Adam optimizer. Event types are mapped to 128-dimensional vectors through the embedding layer, and key parameters are vectorized and concatenated with the event type vector. After adding positional encoding, the input is fed into the encoder, and the correlation between events is calculated through the self-attention mechanism. The encoded output of the entire sequence is then subjected to attention pooling to obtain the log modality feature vector.

[0028] Based on the deduplicated alarm list output in step 1.2, a graph convolutional network (GCNN) is used for encoding to obtain the alarm modality feature vector. The GCNN consists of three layers with hidden dimensions of 64, 128, and 256, respectively, and uses the ReLU activation function. The network is trained using a node classification loss function, the Adam optimizer, and a learning rate of 0.01. The alarm association graph is constructed based on spatiotemporal proximity and type correlation. The initial feature vector of each alarm node includes attributes such as source node, time, level, and type. Through message passing and feature aggregation, nodes collect information from their neighbors and update their representations. The alarm modality feature vector of a node is obtained by averaging and aggregating all alarm embedding vectors related to the same server node.

[0029] Based on the configuration change history output in step 1.2, a recurrent neural network (RNN) is used for encoding to obtain the configuration modality feature vector. The RNN uses an LSTM structure, containing two LSTM layers with a hidden state dimension of 128, and is trained using the mean squared error loss function and the Adam optimizer. The configuration type is mapped to a 64-dimensional vector through the embedding layer, and the change content is encoded as a 128-dimensional vector and then concatenated to form the input. The LSTM network processes the configuration change sequence chronologically, capturing the long-term dependencies in configuration evolution. The hidden state at the last moment is taken as the configuration modality feature vector for that node.

[0030] Step 1.5, Cross-modal fusion: Based on the performance modality feature vector, log modality feature vector, alarm modality feature vector, and configuration modality feature vector output in Step 1.4, a multi-head attention mechanism is used to calculate the cross-modal attention weights, obtaining the contribution distribution of each modality to the fusion result. The multi-head attention mechanism includes multiple parallel attention heads, each independently calculating attention weights. The calculation process of a single attention head first involves performing three linear transformations on each modality feature vector to obtain the query vector, key vector, and value vector. The dot product of the query vector and all key vectors is calculated; the dot product reflects the correlation between the query and the keys. The dot product is normalized by dividing by a scaling factor, which is the square root of the key vector dimension, to prevent the dot product value from being too large and causing gradient vanishing. The normalized dot product is input into the softmax function to obtain the attention weight distribution, with the weights summing to 1. The attention weights are then used to perform a weighted summation on the value vector to obtain the output of the attention head. The outputs of multiple attention heads are concatenated and subjected to a linear transformation to obtain the final output of the multi-head attention mechanism. The weight distribution of multi-head attention reflects the importance of each modality feature to the fusion result. Modalities with high weights contribute more to the characterization of the current node state, while modalities with low weights contribute less.

[0031] Based on attention weights and feature vectors of each modality, a weighted fusion and residual connection technique is employed to obtain a comprehensive representation vector sequence. The value vectors of each modality are weighted and summed according to the attention weights to obtain the fused feature vector. To avoid losing modality-specific information during fusion, a residual connection mechanism is introduced to concatenate the fused feature vector with the original feature vectors of each modality, forming an extended feature vector containing fusion information and modality-specific information. The extended feature vector undergoes dimensionality reduction transformation, mapping the high-dimensional extended feature vector to a fixed-dimensional comprehensive representation vector through a fully connected layer. The comprehensive representation vector simultaneously contains complementary and interactive information from multiple data sources, comprehensively characterizing the operational state of server nodes. The above fusion process is performed on each server node in the server cluster to obtain the comprehensive representation vectors of all nodes. The comprehensive representation vectors are organized according to the time dimension to form a comprehensive representation vector sequence, where each time step in the sequence corresponds to the set of comprehensive representation vectors of all nodes at a given time.

[0032] Step 2: Obtain service call logs and network traffic data from the server cluster. Combined with the comprehensive representation vector sequence, perform topology relationship extraction and structure learning to identify changes in topology dependencies and incrementally update them to obtain a dynamic topology graph sequence. Step 2.1: Extraction and statistics of service call relationships. Based on the service call log data in the server cluster, log parsing and relationship extraction algorithms are used to obtain the call relationships and call statistics between services. Service call logs record remote procedure calls, message passing, interface access, and other interactive behaviors between services. The logs contain information such as caller identifier, callee identifier, call time, response time, and call result. Extract the identifier information of the caller and callee from the call logs. The identifier can be the service name, service instance address, container identifier, etc. Establish call relationship edges between services, with the direction of the edge from the caller to the callee, representing the dependency relationship. Count the call frequency between each pair of services within a fixed time window. The call frequency reflects the tightness of the dependency between services; high frequency indicates tight dependency, and low frequency indicates loose dependency. Calculate the average response time and variance of the response time between each pair of services. The response time reflects the performance characteristics and stability of the service call. By using call relationships, call frequency, and response time as edge attributes of the service dependency graph, a preliminary service dependency graph is constructed.

[0033] Step 2.2, Network Traffic Pattern Analysis and Communication Relationship Identification: Based on network traffic data in the server cluster, traffic statistics and pattern recognition algorithms are used to obtain the communication relationships and traffic characteristics between nodes. Network traffic data is collected through network monitoring equipment or host agents and includes information such as source address, destination address, port number, protocol type, packet size, and transmission rate. Communication relationships between nodes are identified based on source and destination addresses, indicating data transmission between nodes. The data transmission volume between each pair of nodes is statistically analyzed within a fixed time window, reflecting the communication strength between nodes. The application layer protocol of the communication is identified based on port number and protocol type; different protocols correspond to different service types, such as HTTP for Web services and MySQL for database services. The temporal patterns of the traffic are analyzed to identify characteristics such as periodicity, burstiness, and abnormal fluctuations. Communication relationships, transmission volume, protocol type, and traffic patterns are used as edge attributes of the node communication graph to construct the node communication graph. The service dependency graph and the node communication graph are merged; the service dependency graph reflects the logical dependencies of the application layer, and the node communication graph reflects the physical communication of the network layer. The merged graph contains topology information from both logical and physical layers.

[0034] Step 2.3, Topology Learning Based on Graph Neural Networks: Based on the service dependency graph output in Step 2.1, the node communication graph output in Step 2.2, and the comprehensive representation vector sequence output in Step 1, a graph structure learning algorithm is used to obtain a complete topology graph. The initial topology graph only contains explicit dependencies observed through log and traffic data, but there are unobserved implicit dependencies in the actual system, such as implicit competition due to shared resources and implicit associations due to configuration consistency. The goal of graph structure learning is to infer potential implicit dependencies from the comprehensive representation vectors of nodes. A graph generation network is designed, with the comprehensive representation vectors of all nodes as input and the probability of dependencies between node pairs as output. The graph generation network adopts a variational autoencoder architecture, where the encoder encodes the node representation vectors into latent variables, and the decoder calculates the probability of edge existence based on the latent variables of node pairs. The edge probability is calculated using a bilinear function or a neural network function to measure the similarity or association between the latent variables of two nodes. The goal of training the graph generation network is to maximize the likelihood probability of the observed topology graph while minimizing the difference between the latent variable distribution and the prior distribution. After training, the graph generation network can output the probability of an edge existing for any pair of nodes. For node pairs with probabilities exceeding a threshold, inferred implicit dependency edges are added to the topology graph. The weight of the implicit dependency edge is set to the probability value of the edge's existence, reflecting the uncertainty of the dependency relationship. Explicit and implicit dependency edges are merged to obtain a topology graph containing the complete dependencies.

[0035] Step 2.4, Time-based sliding window update of the topology graph: Based on real-time collected service call logs and network traffic data, a sliding window mechanism and incremental update algorithm are used to obtain a dynamically updated topology graph sequence. The topology of the server cluster changes dynamically with the deployment, shutdown, and migration of services, and a static topology graph cannot reflect this dynamism. The sliding window update mechanism sets a fixed window length and sliding step size. The window length determines the statistical time range of the topology graph, and the step size determines the update frequency of the topology graph. Within each sliding window, the topology relationship extraction and structure learning process of steps 2.1 to 2.3 is re-executed to obtain the topology graph corresponding to that window. The topology graph of the new window is compared with the topology graph of the previous window to identify changes in the topology graph, including added nodes, deleted nodes, added edges, deleted edges, and changes in edge weights. For cases with small topology changes, an incremental update method is used, recalculating only the changed parts and retaining the calculation results of the unchanged parts to reduce computational overhead. For cases with large topology changes, a full update is performed, recalculating the entire topology graph. The topology graph corresponding to each time window is organized in chronological order to form a dynamic topology graph sequence. The topology graph at each moment in the sequence reflects the dependency status of the server cluster at that moment.

[0036] Step 2.5, Calculation and Labeling of Topological Graph Attributes: Based on the dynamic topological graph sequence output in Step 2.4, graph theory algorithms are used to calculate the structural and propagation attributes of the topological graph, resulting in a topological graph labeled with node importance and edge propagation characteristics. The structural attributes of the topological graph include node degree, clustering coefficient, betweenness centrality, and proximity centrality. Node degree represents the number of connections between nodes; nodes with higher degrees have more connections and are more important in the topology. The clustering coefficient represents the connection density between a node's neighbors, reflecting the compactness of the subgraph where the node resides. Betweenness centrality represents the frequency with which a node appears on the shortest path; nodes with high betweenness centrality are key nodes for information propagation. Proximity centrality represents the average distance from a node to other nodes; nodes with shorter distances can quickly influence the global graph. The above structural attribute indicators are calculated for each node, and the indicator values ​​are labeled on the node as its importance score. The propagation attributes of the topological graph describe the dynamic characteristics of anomalies propagating along dependencies. Dependency strength is determined by the combined frequency of edge calls and data transmission volume; edges with high call frequency and large transmission volume have high dependency strength, and anomalies are more likely to propagate along those edges. Propagation delay is determined by the response time of an edge; edges with longer response times have greater propagation delay, meaning it takes longer for anomalies to propagate from upstream nodes to downstream nodes. Dependency strength and propagation delay are labeled on edges as their propagation attributes. The labeled topology graph contains rich structural and propagation information, providing crucial topological knowledge for subsequent anomaly detection and root cause localization.

[0037] Step 3: Based on the comprehensive representation vector sequence and the dynamic topology graph sequence, predict the server node status, identify abnormal nodes, and perform preliminary classification to obtain the abnormal node set and preliminary classification results of abnormal types. Step 3.1: Construction of the graph spatiotemporal convolutional network; Based on the comprehensive representation vector sequence output in Step 1 and the dynamic topology graph sequence output in Step 2, a network architecture combining graph convolution and temporal convolution is adopted to obtain the graph spatiotemporal convolutional network. The network contains 4 graph spatiotemporal convolutional blocks, each containing a temporal convolution module (1D convolution, kernel size 3, number of channels 64) and a spatial graph convolution module (2-layer graph convolution, hidden dimension 128). The network is trained using the mean squared error loss function, the Adam optimizer, and a learning rate of 0.0001. Temporal convolution uses causal convolution and dilated convolution to extract temporal patterns, and spatial graph convolution aggregates neighbor node features. A spatiotemporal attention mechanism is introduced to learn the contribution weights of different time steps and neighbor nodes.

[0038] Step 3.2, Node Future State Prediction: Based on the graph spatiotemporal convolutional network constructed in Step 3.1 and the node representation vectors and topology graphs at current and historical moments, a sequence prediction method is used to obtain the predicted state value of each node. The sequence of node representation vectors and the corresponding topology graph sequence from multiple past time steps are input into the graph spatiotemporal convolutional network. The network learns the evolutionary rules of node states in time and space dimensions through multi-layer spatiotemporal convolution and attention mechanisms. The network's output layer predicts the state representation vector of each node at one or more future time steps. The prediction adopts an encoder-decoder architecture. The encoder encodes the historical state sequence into a hidden state representation, and the decoder gradually generates future state predictions based on the hidden state representations. For multi-step predictions, the decoder uses an autoregressive approach, using the prediction result of the previous step as the input for the next prediction, progressively advancing the prediction time. The graph spatiotemporal convolutional network is trained using historical normal operation data. The training objective is to minimize the error between the predicted state and the actual state, with the error measured by mean squared error or cosine similarity. Through training with a large amount of historical data, the network learns the normal evolutionary pattern of node states.

[0039] Step 3.3, Calculation of State Deviation and Anomaly Scoring: Based on the predicted future state values ​​of nodes output in Step 3.2 and the actually observed node state values, anomaly scores for each node are obtained using deviation measurement and scoring mapping methods. For the current moment, the node state representation vector predicted by the graph spatiotemporal convolutional network is compared with the actually collected state representation vector to calculate the deviation between the predicted and actual values. The deviation measurement uses Euclidean distance or Mahalanobis distance. Euclidean distance measures the straight-line distance between vectors in space, while Mahalanobis distance considers the correlation and variance between the dimensions of the vector. The calculated deviation value is used as the node's original anomaly score. A large deviation indicates that the node's actual state deviates from the normal pattern, with a high probability of anomaly; a small deviation indicates that the node's state conforms to the normal pattern, with a low probability of anomaly. To ensure comparability of anomaly scores between different nodes, the original anomaly scores are normalized, mapping the scores to a fixed numerical range. The normalized anomaly score reflects the degree of anomaly of the node; the higher the score, the greater the degree of anomaly.

[0040] Step 3.4, Dynamic Adjustment of Adaptive Threshold: Based on the historical distribution of abnormal scores and the current operating status of the cluster, an adaptive threshold is obtained using statistical analysis and an adaptive adjustment algorithm. A fixed threshold is difficult to adapt to changes in the cluster environment and the normal fluctuation differences among different nodes, easily leading to false positives and false negatives. The adaptive threshold is dynamically determined based on the statistical characteristics of historical abnormal scores. Abnormal scores of all nodes within past time windows are collected, and the mean and standard deviation of the abnormal scores are calculated. The threshold is determined based on the mean and standard deviation, set as the mean plus a certain multiple of the standard deviation, with the multiplier adjusted according to the desired detection sensitivity. Considering the normal fluctuation differences among different nodes, an adaptive threshold is calculated individually for each node, with the node's threshold determined based on the statistical characteristics of its historical abnormal scores. Considering the periodicity of the cluster's operating status, the normal status distribution of nodes differs during peak and off-peak periods, requiring corresponding threshold adjustments. The threshold is calculated using historical data corresponding to the current business cycle stage. The adaptive threshold is updated in real-time as the time window slides, ensuring that the threshold can dynamically adjust with changes in the cluster environment.

[0041] Step 3.5, Identification and Classification of Abnormal Nodes: Based on the node anomaly scores output in Step 3.3 and the adaptive thresholds output in Step 3.4, a threshold comparison and classification method is used to obtain the set of abnormal nodes and preliminary classification results of anomaly types. The anomaly score of each node is compared with its corresponding adaptive threshold. Nodes with anomaly scores exceeding the threshold are determined to be abnormal nodes, and nodes with anomaly scores below the threshold are determined to be normal nodes. A preliminary anomaly type classification is performed on the abnormal nodes, based on the anomaly's performance pattern in the comprehensive representation vector. The comprehensive representation vector of the abnormal nodes is analyzed to identify which modalities exhibit significant anomalies. If the performance modality features are abnormally prominent, it is preliminarily determined to be a performance-related anomaly, including subtypes such as CPU anomalies, memory anomalies, disk anomalies, and network anomalies. If the log modality features are abnormally prominent, it is preliminarily determined to be a log-related anomaly, including subtypes such as a surge in error logs and the occurrence of abnormal events. If the alarm modality features are abnormally prominent, it is preliminarily determined to be an alarm-related anomaly. If the configuration modality features are abnormally prominent, it is preliminarily determined to be a configuration-related anomaly. If multiple modalities are abnormal simultaneously, it is determined to be a composite anomaly. The abnormal nodes, their abnormal scores, and the preliminary classification results of abnormal types are output to form an abnormality detection report.

[0042] Step 3.6, Anomaly Risk Prediction for Future Time Steps: Based on the predicted node states for future time steps output in Step 3.2 and the adaptive threshold output in Step 3.4, an anomaly risk prediction result is obtained using a prediction bias assessment method. For each future time step, although the actual state value cannot be obtained, the risk of anomaly occurrence can be assessed based on the predicted state value. It is assumed that the actual state value for future time steps follows a probability distribution with the predicted value as the mean, and the variance of the distribution is determined by the uncertainty of the model prediction. The probability that the actual state value exceeds the anomaly detection threshold is calculated, and this probability is used as the anomaly risk score for future time steps. A high anomaly risk score indicates a high probability of anomaly occurrence in the future, while a low anomaly risk score indicates a low probability of anomaly occurrence. Nodes and time steps where the risk score exceeds the warning threshold are identified. The warning threshold is determined based on the statistical distribution of historical risk prediction data and is set as the mean of the anomaly risk score plus one standard deviation, generating an anomaly risk warning report. Risk warnings enable operations and maintenance personnel to understand potential anomaly trends in advance and take preventative measures to reduce the probability of anomalies or mitigate their impact.

[0043] Step 4: Based on the dynamic topology graph sequence and the set of anomalous nodes, perform causal inference and root cause localization to obtain root cause node identifiers; Step 4.1: Temporal causal relationship analysis of anomalous node pairs; Based on the set of anomalous nodes and the historical state time series of the nodes output in Step 3, the Granger causality test method is used to obtain the temporal causal relationship between anomalous node pairs. For each pair of nodes in the set of anomalous nodes, denoted as node A and node B, the state time series of node A and node B within a time window before the occurrence of the anomaly are extracted. An autoregressive model is established for the state time series of node B, which uses only the historical state of node B itself to predict the current state, and the fitting error of the model is calculated. An extended autoregressive model is established for the state time series of node B, which uses both the historical state of node B itself and the historical state of node A to predict the current state of node B, and the fitting error of the extended model is calculated. The fitting errors of the two models are compared. If the error of the extended model is significantly smaller than the error of the autoregressive model, it indicates that the historical information of node A is helpful in predicting the state of node B, and node A is determined to be a Granger cause of node B. The statistical significance of the error differences is assessed using an F-test or likelihood ratio test to obtain the significance level p-value. If the p-value is less than the significance threshold, the null hypothesis is rejected, and the causal relationship is considered valid. Granger causality tests are performed on all outlier node pairs to obtain the causal relationship and significance level between the node pairs.

[0044] Step 4.2, Calculation of Causal Strength: Based on the causal relationships and significance levels of the node pairs output in Step 4.1, a comprehensive scoring method is used to obtain a causal strength score reflecting the strength of the causal relationship. The causal strength score comprehensively considers the statistical significance of the Granger causality test and the consistency of time delay. Statistical significance is reflected by the p-value; the smaller the p-value, the more significant the causal relationship and the greater the causal strength. Time delay consistency examines whether the time interval between the state changes of node A and node B conforms to the propagation delay of the edges in the topological graph. The moments of anomalous state changes are identified from the state time series of nodes A and B, and the time difference between the moments of anomalous changes in node A and node B is calculated. The time difference is compared with the propagation delay of the edges from node A to node B in the topological graph; the closer the time difference is to the propagation delay, the higher the time delay consistency and the more credible the causal relationship. A causal strength scoring function is designed, which integrates statistical significance and time delay consistency, and uses a weighted product or weighted sum to obtain the causal strength score between each pair of nodes. A high causal strength score indicates a strong and credible causal relationship, while a low score indicates a weak or uncertain causal relationship.

[0045] Step 4.3, Construction of the Anomalous Causal Dependency Graph: Based on the causal strength scores of the anomalous node pairs output in Step 4.2, a graph construction method is used to obtain a causal dependency graph describing the causal dependencies between anomalous nodes. The causal dependency graph is a directed graph where nodes represent anomalous nodes, directed edges represent causal relationships, the direction of the edges is from cause to effect, and the weight of the edges is the causal strength score. For node pairs with causal strength scores exceeding a threshold, a directed edge from the cause node to the result node is added to the causal dependency graph, and the weight of the edge is set to the causal strength score. For node pairs with causal strength scores below the threshold, no edge is added to the causal dependency graph, indicating that the causal relationship is not valid or not significant. The causal dependency graph may contain multiple connected components, each representing an independent anomalous propagation chain. The causal dependency graph is simplified by removing redundant edges. If node A indirectly affects node B through intermediate node C, and there is a direct edge from node A to node B, but the causal strength of the direct edge is lower than that of the indirect path, then the direct edge is removed, and the indirect path is retained. The simplified causal dependency graph clearly shows the core causal relationships and propagation chains between anomalous nodes.

[0046] Step 4.4, Identification of Candidate Root Cause Nodes: Based on the causal dependency graph output in Step 4.3, a topological analysis method is used to identify the starting node of the anomaly propagation chain as a candidate root cause node. The root cause node is the source of the anomaly, and other anomaly nodes are the direct or indirect results caused by the root cause node. In the causal dependency graph, the characteristic of a root cause node is that it has no incoming edges or weak incoming edge causal strength. A topological sort is performed on the causal dependency graph, arranging nodes according to the order of causal relationships, with nodes at the beginning being causes and nodes at the end being results. The starting node of the topological sort is the node without incoming edges, and these nodes are considered candidate root cause nodes. If a cycle exists in the causal dependency graph, it indicates that there is a feedback relationship between the anomaly nodes, and topological sorting cannot be completed. A strongly connected component decomposition algorithm is used to decompose the cycle into strongly connected components, and the nodes within the strongly connected components are considered as a whole candidate root cause. The sum of the incoming edge causal strength of each candidate root cause node is calculated; nodes with smaller sums of incoming edge causal strength are more likely to be the true root cause. The candidate root cause nodes are sorted according to the sum of incoming edge causal strength to generate a priority list of root cause nodes.

[0047] Step 4.5, Root Cause Node State Anomaly Assessment: Based on the candidate root cause nodes and their historical state data output in Step 4.4, anomaly measurement methods are used to obtain the state anomaly scores of the candidate root cause nodes. Root cause nodes should exhibit significant state changes before an anomaly occurs; these state changes are the source triggering anomaly propagation. For each candidate root cause node, a state representation vector sequence is extracted within a time window before and after the anomaly occurs. Statistical characteristics of the node's state before and after the anomaly are calculated, including mean, variance, and extreme values, and the degree of difference in statistical characteristics before and after is compared. The degree of difference is measured using relative change rate or statistical tests. The relative change rate calculates the ratio of the change in the characteristic value to the original value, and statistical tests assess whether there is a significant difference in the state distribution between the two time periods. The degree of difference is used as the node's state anomaly score. A high anomaly score indicates that the node's state changed significantly before and after the anomaly, making it more likely to be a root cause; a low anomaly score indicates that the node's state changed little, making it less likely to be a root cause.

[0048] Step 4.6: Determination of the final root cause node and tracing the propagation path. Based on the priority list of candidate root cause nodes output in Step 4.4 and the state anomaly score output in Step 4.5, a comprehensive judgment method is used to obtain the root cause node identifier and the anomaly propagation path. The comprehensive judgment considers two factors: the topological position of the causal dependency graph and the state anomaly of the nodes. A root cause confidence scoring function is designed, which combines the incoming edge causal strength and state anomaly of the candidate root cause nodes, using a weighted sum or weighted product form. Nodes with low incoming edge causal strength and high state anomaly have high root cause confidence, while nodes with high incoming edge causal strength or low state anomaly have low root cause confidence. The root cause confidence of each candidate root cause node is calculated, and the node with the highest confidence is selected as the final root cause node. Starting from the root cause node, a breadth-first search or depth-first search is performed on the causal dependency graph to traverse all directed paths starting from the root cause node. The sequence of nodes on the path is the anomaly propagation path. For each propagation path, the cumulative causality strength of the path is calculated. The cumulative causality strength is the product or weighted sum of the causality strength scores of all edges along the path, reflecting the credibility of the propagation path. Several main propagation paths with the highest cumulative causality strength are identified; these paths describe the main pathways by which the anomaly spreads from the root node to other anomalous nodes. All anomalous nodes covered by the propagation paths are statistically analyzed to assess the scope of the anomaly's impact, which is the set of nodes directly or indirectly affected by the root node. The propagation time of the anomaly is estimated; the propagation time is the sum of the propagation delays of all edges along the path, reflecting the time required for the anomaly to propagate from the root node to the terminal nodes.

[0049] Step 5: Based on the root cause node identifier and preliminary anomaly type classification results, perform rule reasoning to identify the anomaly type of the root cause node and obtain diagnostic identifiers; Step 5.1: Construction of the Operation and Maintenance Knowledge Rule Base; Based on the experience and knowledge of operation and maintenance experts and historical anomaly handling cases, a structured operation and maintenance knowledge rule base is obtained using knowledge engineering methods. The operation and maintenance knowledge rule base stores diagnostic rules, each rule is represented by a production rule, with the rule form being IF antecedent condition THEN consequent conclusion. The antecedent condition describes the characteristic pattern of the anomaly, including anomaly type, abnormal performance indicators, key events in logs, alarm combination patterns, abnormal configuration states, and other aspects. The antecedent condition can contain multiple sub-conditions, which are combined using logical AND, logical OR, and logical NOT. The consequent conclusion provides the diagnostic result of the anomaly, including the specific type of the anomaly, the root cause of the anomaly, the severity of the anomaly, and other information. Each rule is accompanied by a confidence parameter, which reflects the reliability of the rule and is assigned by operation and maintenance experts based on experience or obtained from statistical analysis of historical cases. The rule base construction process begins with collecting diagnostic experience from operations and maintenance experts through interviews and surveys, transforming their natural language descriptions into structured rule representations. Diagnostic rules are then mined from historical anomaly handling cases using association rule mining algorithms to identify frequent patterns between anomaly features and diagnostic conclusions, converting these frequent patterns into diagnostic rules. The mined rules undergo expert review to verify their rationality and accuracy, correcting erroneous rules and supplementing missing ones. Finally, an index structure is established for the rules, based on the characteristics of the rule's antecedent conditions, supporting rapid rule matching and retrieval.

[0050] Step 5.2, Fuzzy Matching of Anomaly Features and Rule Predecessors: Based on the knowledge rule base constructed in Step 5.1, the root cause node anomaly feature data obtained from the root cause node identifiers output in Step 4, and the preliminary anomaly type classification results output in Step 3, a fuzzy matching algorithm is used to obtain the activation degree of each rule. The anomaly feature data of the root cause nodes includes anomaly patterns of performance indicators, key information of log events, alarm types and contents, configuration changes, etc. The anomaly feature data is matched with the predecessor conditions of each rule in the rule base. The matching process considers the fuzziness and uncertainty of feature values ​​and uses fuzzy set theory to calculate the matching degree. For numerical features, a fuzzy membership function is defined, which describes the degree to which a feature value belongs to a certain fuzzy category. The membership degree of the actual feature value in the fuzzy interval defined by the rule predecessor conditions is calculated. For categorical features, exact matching or semantic similarity matching is used. Semantic similarity is calculated through domain ontology or word vectors. For combined conditions, the overall matching degree of the combined conditions is calculated according to logical operators. Logical AND takes the minimum matching degree of each sub-condition, logical OR takes the maximum matching degree of each sub-condition, and logical NOT takes the complement. The overall matching degree of the rule's antecedent conditions is used as the rule's activation degree, which reflects the degree of conformity between the current abnormal feature and the rule's antecedents. Activation degrees are calculated for all rules, and rules with activation degrees exceeding a threshold are selected as candidate diagnostic rules.

[0051] Step 5.3, Reasoning and Diagnostic Conclusion Generation of Knowledge Rules: Based on the candidate diagnostic rules and their activation values ​​output in Step 5.2, a reasoning mechanism is used to obtain the diagnostic conclusion and confidence level of the rule reasoning. For rules with activation values ​​exceeding a threshold, the consequent conclusion of the rule is triggered, and the consequent conclusion is used as a diagnostic candidate. The diagnostic confidence level of a rule is jointly determined by the rule's own confidence level and the rule's activation value, calculated using a product or weighted sum method. The diagnostic confidence level reflects the credibility of the diagnostic conclusion. If multiple rules have the same consequent conclusion, the diagnostic confidence levels of these rules are aggregated. The aggregation method uses the maximum value, average value, or evidence fusion method to obtain the comprehensive confidence level of the diagnostic conclusion. If multiple rules have different and conflicting consequent conclusions, all diagnostic candidates are retained, and their confidence levels are recorded. The final diagnostic conclusion is determined by fusing with the pattern recognition results. The output of rule reasoning is a set of diagnostic candidates and their confidence levels. The diagnostic candidates include information such as anomaly type, anomaly cause, and severity.

[0052] Step 5.4, Training and Inference of the Deep Learning Diagnostic Model: Based on historical abnormal case data, an abnormality type classification model is trained using deep learning technology to obtain pattern recognition diagnostic conclusions. The classification model uses a 3-layer fully connected neural network with hidden layer dimensions of 256, 128, and 64. ReLU activation function and Dropout (0.3) are used to prevent overfitting.

[0053] Historical anomaly case data includes multimodal features at the time of anomaly occurrence and corresponding diagnostic labels, with the diagnostic labels representing the true anomaly type. Anomaly cases are divided into training, validation, and test sets according to time or randomly. An anomaly type classification model architecture is designed. The input to the anomaly type classification model is the comprehensive representation vector of the root cause node, obtained from the cross-modal fusion in step 1, containing multi-source information such as performance, logs, alarms, and configuration. The classification model employs a multi-layer fully connected neural network or convolutional neural network, extracting high-level abstract representations of features through multi-layer nonlinear transformations. The output layer is a softmax classification layer, trained using the cross-entropy loss function and the Adam optimizer, outputting the probability distribution of each anomaly type. During training, the validation set is used to evaluate the classification model's performance, and early stopping is performed based on the accuracy or loss value on the validation set to prevent overfitting. After training, the generalization ability of the classification model is evaluated on the test set. In actual diagnosis, the root cause node comprehensive representation vector determined based on the root cause node identifier output in step 4 is input into the trained anomaly type classification model. The classification model outputs the probability distribution of each anomaly type, and the category with the highest probability is taken as the anomaly type identified by the classification model, and the probability value is taken as the prediction confidence of the classification model.

[0054] Step 5.5, Fusing Diagnostic Results Based on Evidence Theory: Based on the rule-based reasoning diagnostic conclusions and confidence levels output in Step 5.3 and the pattern recognition diagnostic conclusions and confidence levels output in Step 5.4, Dempster-Shafer evidence theory is used for fusion to obtain a comprehensive diagnostic result and comprehensive confidence level. Evidence theory can handle uncertainty and incomplete information, making it suitable for fusing diagnostic evidence from different sources. The diagnostic conclusions from rule-based reasoning are used as evidence source 1, and the diagnostic conclusions from pattern recognition are used as evidence source 2. For each diagnostic conclusion, a basic probability assignment function is constructed. The basic probability assignment function assigns the confidence level to the proposition corresponding to the diagnostic conclusion, and the remaining uncertainty is assigned to the unknown proposition. Dempster's combination rule is applied to fuse the basic probability assignment functions of the two evidence sources. The combination rule calculates the sum of the products of the support levels of different evidence sources for the same proposition, and after normalization, the fused basic probability assignment function is obtained. The fused basic probability assignment function reflects the confidence level of each diagnostic conclusion after integrating the two evidence sources. If the conclusions from rule-based reasoning and pattern recognition are consistent, the confidence level increases after fusion, indicating that the diagnostic result is reliable. If the two conclusions are inconsistent, the fusion process determines the final diagnosis based on the relative confidence levels of each conclusion. Conclusions with higher confidence levels have greater weight, while the overall confidence level decreases, reflecting the uncertainty of the diagnosis. The diagnosis with the highest confidence level after fusion is selected as the final diagnosis.

[0055] Step 5.6: Identification and Manual Verification Mechanism for Uncertain Diagnoses; Based on the final diagnostic results and overall confidence level output in Step 5.5, a confidence threshold determination method is used to obtain diagnostic identifiers. A confidence threshold is set; diagnostic identifiers with an overall confidence level exceeding the threshold are considered deterministic diagnoses, indicating high reliability and direct applicability to subsequent self-healing strategy generation. Diagnostic identifiers with an overall confidence level below the threshold are considered uncertain diagnoses, indicating significant uncertainty and requiring manual verification. For uncertain diagnoses, a diagnostic report is generated and submitted to operations and maintenance experts. The report includes anomaly feature data, candidate conclusions for rule-based reasoning, candidate conclusions for pattern recognition, the confidence level of each candidate, and an analysis of the causes of diagnostic uncertainty. Operations and maintenance experts perform manual diagnosis based on the report content and their own experience to determine the correct anomaly type and cause. The manually verified diagnostic results are fed back to the system as new training samples to update the knowledge rule base and diagnostic model. For manually confirmed diagnostic conclusions, if they are consistent with the conclusions of rule-based reasoning or pattern recognition, the weight and confidence level of the corresponding method are increased. If a new abnormal pattern is discovered through manual diagnosis, the feature patterns of the new pattern are extracted, transformed into new diagnostic rules, and added to the rule base. At the same time, the new cases are added to the training data to retrain the diagnostic model.

[0056] Step 6: Obtain historical operation and maintenance data of the server cluster, construct a policy network, verify and deploy the policy network, make decisions based on diagnostic indicators, and obtain a candidate set of repair policies; Step 6.1: Construction of a digital twin model of the cluster environment; Based on the historical operation and maintenance data and system configuration parameters of the server cluster environment, a data-driven modeling method is adopted to obtain a digital twin simulation system capable of simulating the behavior of a real cluster. The goal of constructing the digital twin model is to reproduce the response behavior of the real cluster to various repair operations, providing a safe training environment for policy learning. The digital twin system supports modeling of a hybrid cluster environment based on ARM architecture CPUs and x86 architecture CPUs, and can accurately simulate the behavioral characteristics of domestic operating systems such as Galaxy Kylin server operating system, UnionTech UOS, and Euler openEuler. Collect cluster topology data, hardware configuration data, software configuration data, business load data, historical operation and maintenance data, and system response data. Establish a cluster structural model, which describes the static structural information such as the composition of nodes, the connection relationship between nodes, and the deployment distribution of services. Establish a cluster behavioral model, which describes the dynamic response law of the system under different loads and operations. Machine learning methods are employed to learn the system's behavioral patterns from historical data. Time series modeling is used for the evolution of performance indicators, probabilistic graphical models are used for fault propagation, and queuing theory models are used for resource scheduling. By combining structural and behavioral models, a complete digital twin simulation system is constructed. The simulation system receives external operation commands, calculates the impact of the operations on the system state based on the behavioral model, updates the system's state variables, and simulates the operation of the real system. The digital twin model is validated by comparing the output of the simulation system under the same initial state and operation sequence with the historical records of the real system to evaluate the accuracy of the simulation. Model parameters are adjusted based on the errors to improve the realism of the simulation.

[0057] Step 6.2, Modeling the Markov Decision Process: Based on the digital twin simulation system constructed in Step 6.1, the Markov Decision Process framework is adopted to obtain a formal description of the self-healing strategy learning. The Markov Decision Process includes four elements: state space, action space, state transition probabilities, and reward function. The state space is defined as the operating state of the cluster, including information such as the performance indicators of all nodes, the running status of services, the occurrence of anomalies, and currently executed repair operations. This state information is organized into a state vector. The action space is defined as the set of executable repair operations, including types such as restarting services, restarting nodes, switching to standby nodes, adjusting resource quotas, modifying configuration parameters, isolating faulty nodes, and rolling back versions. Each action is determined by the operation type and operation parameters. The state transition probability describes the change in system state after executing an action. State transitions are determined by the behavioral model of the digital twin model. Given the current state and the executed action, the simulation system calculates the state at the next moment. State transitions may have randomness, reflecting the uncertainty of the system. The reward function evaluates the effect after executing the action, comprehensively considering factors such as whether the anomaly is repaired, the time required for repair, the impact of the operation on business, and the resource cost of the operation. A positive reward is given for successfully fixing an anomaly, while a negative reward is given for failed fixing or the occurrence of a new anomaly. Rewards are reduced for long fixing times, significant business impact, and high operational costs. The specific form of the reward function is designed, using a weighted sum or weighted product to synthesize various factors into a scalar reward value.

[0058] Step 6.3, Training the Deep Reinforcement Learning Policy Network: Based on the Markov Decision Process model and digital twin simulation system established in Step 6.2, a deep reinforcement learning algorithm is used to obtain the optimal policy network. The policy network is a deep neural network. The input is the current state vector, and the output is a probability distribution in the action space, representing the tendency to choose each action in the current state. The architecture of the policy network includes an input layer, multiple hidden layers, and an output layer. The input layer receives the state vector, and its dimension is the product of the number of cluster nodes and the dimension of node features. The first hidden layer is a fully connected layer with 512 neurons, using a modified linear unit activation function, and a batch normalization layer is added to improve training stability. The second hidden layer is a fully connected layer with 256 neurons, also using a modified linear unit activation function. The third hidden layer is a fully connected layer with 128 neurons, used to further extract high-level features. The number of neurons in the output layer is equal to the size of the action space, and a softmax activation function is used to generate the action probability distribution. Dropout layers are added between the hidden layers, with a dropout rate set to 0.2 to prevent overfitting. The network parameters are initialized using the Xavier initialization method.

[0059] The reinforcement learning training process employs a proximal policy optimization algorithm to initialize the policy network and value network parameters. The value network has a similar structure to the policy network, but its output layer consists of a single neuron used to estimate state values. Interactive learning is conducted within a digital twin simulation system. Each training round starts from the initial state. The policy network outputs an action probability distribution based on the current state, samples actions according to the probability distribution, executes the actions in the simulation system, and the simulation system returns the next state and reward. The policy network continues to select actions based on the new state, repeating this process until the anomaly is corrected or the maximum number of steps is reached. The state, action, and reward sequence for the entire round is recorded, and the cumulative reward and advantage function for each time step are calculated. The policy gradient method is used to update the policy network parameters, while simultaneously updating the value network parameters to improve the accuracy of state value estimation. The policy gradient algorithm optimizes the policy by increasing the probability of obtaining high-reward actions and decreasing the probability of obtaining low-reward actions. The gradient calculation is based on the logarithmic derivative of the cumulative reward and action probabilities. An experience replay mechanism is used to store historical interaction data, and batches of data are randomly sampled from the experience pool for parameter updates during each training session. The learning rate is set to 0.0003, and the Adam optimizer is used for parameter updates. By repeating the training rounds multiple times, the policy network parameters are gradually optimized, the average reward of the policy is gradually increased, and finally converges to the optimal or near-optimal policy.

[0060] Step 6.4, Introduction of Security Constraint Mechanism: Based on the risk characteristics of repair operations, a constraint reinforcement learning method is adopted to obtain a policy network that satisfies security constraints. Repair operations may negatively impact system stability and business continuity, necessitating the introduction of security constraints during policy learning to avoid learning dangerous operational policies. Security constraints are defined, describing the security requirements the system must meet, such as critical services not being interrupted, node restart frequency not exceeding a threshold, and resource utilization not exceeding an upper limit. These constraints are formalized as constraint functions, taking states and actions as input and outputting the degree of constraint violation. The output is 0 when the constraint is satisfied and positive when it is violated. A penalty term for constraint violation is added to the reward function of policy learning, imposing a large negative reward for constraint violation to guide the policy to avoid actions that violate constraints. Constraint checks are added during policy execution. After the policy network outputs the action probability distribution, actions that violate constraints are filtered out, and sampling is performed only from the set of safe actions. Security reinforcement learning algorithms, such as constraint policy optimization or the security Actor-Critic algorithm, are employed. These algorithms explicitly constrain the policy to meet security conditions while optimizing policy performance. The policy network, trained with safety constraints, avoids performing high-risk operations while ensuring repair effectiveness, thus improving the safety of the self-healing system.

[0061] Step 6.5, Policy Network Verification and Deployment: Based on the policy network output in Step 6.3 and the security constraint policy network output in Step 6.4, simulation verification and performance evaluation methods are used to obtain a verified deployable policy model. The policy network is fully verified in a digital twin simulation system, constructing test cases for various abnormal scenarios and cluster states, covering common anomaly types, rare anomaly types, and extreme anomaly situations. For each test case, the policy network is run to generate a repair policy and executed in the simulation system, recording indicators such as whether the repair is successful, the number of steps required for repair, and whether security constraints are violated. The verification results of all test cases are statistically analyzed, and performance indicators such as the policy's success rate, average repair steps, and constraint violation rate are calculated. Verification standards are set, such as a success rate exceeding a threshold and a constraint violation rate of 0. Only policy networks that pass the verification standards can be deployed to the real environment. For policy networks that fail verification, the reasons for failure are analyzed, training parameters, reward functions, or constraint conditions are adjusted, and retraining is performed until verification is passed. Verified policy networks are packaged into policy model files and deployed to the policy decision module of the self-healing system for use in actual anomaly handling.

[0062] Step 6.6: Generating a strategy for the current anomaly. Based on the deployment strategy network output in Step 6.5, the diagnostic identifier output in Step 5, and the current cluster status information, a strategy inference method is used to obtain a set of candidate repair strategies. The diagnostic results of the current anomaly are converted into part of a state vector, including information such as anomaly type, root cause node, and scope of impact. Real-time status information of the current cluster is obtained, including performance metrics of all nodes, service running status, and resource usage, to construct a complete current state vector. The current state vector is input into the deployed strategy network, which outputs a probability distribution in the action space. A set of candidate repair strategies is generated based on the probability distribution, using multiple sampling strategies to generate multiple candidate strategies. The first sampling strategy is a greedy strategy, selecting the action with the highest probability as the candidate strategy, which is considered the optimal repair solution by the strategy network. The second sampling strategy is random sampling, which generates several candidate strategies by randomly sampling according to the probability distribution. Random sampling can explore the second-highest probability actions, providing alternative solutions. The third sampling strategy is a bundle search, maintaining multiple candidate action sequences, expanding the highest probability actions at each step, ultimately obtaining multiple complete repair strategy sequences. For each candidate strategy, a pre-run is performed in a digital twin simulation system to simulate the strategy's execution and evaluate its expected effects, including metrics such as expected repair success rate, expected repair time, and expected business impact. The candidate strategies and their expected effects are then output as a candidate set of repair strategies.

[0063] Step 7: Based on the candidate set of repair strategies, conduct multi-dimensional risk assessment and graded processing, monitor the execution effect in real time and perform failure rollback, and obtain a repair execution result report and status recovery record; Step 7.1: Multi-dimensional risk assessment of repair operations; Based on each candidate strategy in the candidate set of repair strategies output in Step 6, a multi-dimensional risk assessment model is used to obtain the risk level classification of each strategy. The risk assessment is conducted from four dimensions: impact on system stability, impact on business continuity, operational reversibility, and historical execution success rate. The impact on system stability is assessed by evaluating the potential threat of the repair operation to the overall stability of the cluster. Operations such as restarting core nodes and modifying critical configurations have a significant impact on stability and are high-risk, while operations such as adjusting non-critical parameters and isolating edge nodes have a small impact on stability and are low-risk. The impact on system stability is comprehensively assessed by analyzing the importance of the nodes involved in the operation in the topology graph, the number of services affected by the operation, and the cascading effects that the operation may cause. Combined with the anomaly propagation path output in Step 4, the potential impact of the repair operation on the nodes in the propagation chain is analyzed. The propagation path analysis includes assessing whether the repair operation will interrupt the anomaly propagation chain, whether it will affect the key transit nodes in the propagation path, and the recovery time of downstream nodes in the propagation chain after repairing the root cause node. For repair operations involving critical nodes in the propagation path, the system stability impact score will be increased accordingly.

[0064] Business continuity impact assessment evaluates the degree of disruption that remediation operations cause to ongoing services. Operations that lead to service interruption or performance degradation have a significant impact and high risk, while operations that are transparent to the business have a small impact and low risk. Business continuity impact is assessed by analyzing factors such as service availability during the operation, expected business request failure rate, and the degree of user experience degradation. Operation reversibility assessment evaluates whether a remediation operation can be quickly rolled back and recovered after failure. Reversible operations, such as switching traffic and adjusting parameters, can be quickly undone and have low risk, while irreversible operations, such as deleting data and formatting disks, cannot be recovered and have high risk. Historical execution success rate is calculated by statistically analyzing the number of successful and failed operations of this type of remediation operation in historical cases. A high success rate indicates reliable operation with low risk, while a low success rate indicates high uncertainty and high risk. The assessment results for each dimension are quantitatively scored, and the multi-dimensional scores are combined into an overall risk score using a weighted sum method. The weights of each dimension are configured according to business needs and operational strategies. Based on the anomaly risk prediction results output in step 3.6, when the anomaly risk score exceeds the warning threshold, the risk level classification threshold is lowered, allowing more remediation strategies to be classified as low-risk, increasing the proportion of automatic execution, and accelerating anomaly handling. The warning threshold is determined based on the statistical distribution of historical risk prediction data and is set as the mean of the risk score plus one standard deviation. Remediation strategies are divided into three levels—low risk, medium risk, and high risk—based on the overall risk score. Thresholds for each level are set: a risk score below the first threshold is low risk, between the first and second thresholds is medium risk, and above the second threshold is high risk.

[0065] Step 7.2, Automatic Execution Queue Management of Low-Risk Strategies: Based on the remediation strategies assessed as low-risk in Step 7.1, an automatic execution queue scheduling method is adopted to obtain an automatic execution task queue sorted by priority. Low-risk strategies meet the conditions of reversible operation, high historical success rate, and minimal business impact, and can be executed automatically without manual intervention. All low-risk strategies are added to the automatic execution queue, and the execution priority is determined according to the expected remediation effect and urgency of the strategy. Strategies with good expected remediation effect and high anomaly urgency have higher priority and are executed first. The urgency is assessed based on factors such as the scope of the anomaly's impact, business importance, and duration. The execution queue adopts a priority queue data structure, supporting the retrieval of strategies to be executed in priority order. The queue scheduler retrieves the highest priority strategy from the queue and submits it to the execution engine for automatic execution. The execution engine converts the strategy into specific automated operation and maintenance instructions, calls the API interface of the underlying operation and maintenance tools, and executes the remediation operation on the target server node. The execution engine has been specifically optimized for ARM architecture CPU servers and domestic operating system environments to ensure the compatibility and execution efficiency of the remediation instructions on domestic platforms such as Galaxy Kylin server operating system and UnionTech UOS.

[0066] Step 7.3, Manual Approval Process for Medium-to-High Risk Strategies: Based on the remediation strategies assessed as medium-to-high risk in Step 7.1, a remediation suggestion generation and approval process mechanism is adopted to obtain a remediation suggestion report awaiting manual confirmation. Medium-to-high risk strategies are not executed automatically; a detailed remediation suggestion report needs to be generated and submitted to the operations and maintenance personnel for manual approval. The remediation suggestion report includes detailed diagnostic information of the anomaly, the operational steps of the remediation strategy, risk assessment results, expected remediation effects, possible negative impacts, and alternative solutions. The report is presented in a structured and visual manner, clearly displaying key information. For medium-risk strategies, a semi-automatic execution mode is adopted. After reviewing the remediation suggestion, if the operations and maintenance personnel approve the solution, they click the confirmation button to authorize the system to execute automatically; if they do not approve, they can choose an alternative solution or handle it manually. For high-risk strategies, a suggestion reference mode is adopted. The system only provides remediation suggestions for the operations and maintenance personnel to refer to, without providing an automatic execution option. The operations and maintenance personnel perform manual operations based on the suggestion content and their own experience. An approval process management mechanism is established to record the submission time, approver, approval result, and execution status of each remediation suggestion, forming an approval log. Send reminders to suggestions that have not been approved for a long time to avoid delays in handling exceptions.

[0067] Step 7.4, State Machine and Atomic Operation Decomposition: Based on the repair strategy to be executed, state machine control and operation decomposition techniques are used to obtain a controllable repair execution process. The repair strategy typically includes multiple repair actions that need to be executed sequentially in a specific order. A repair execution state machine is designed, with states including pending execution, executing, successful execution, failed execution, rolling back, and rolled back. The initial state of the state machine is pending execution. The execution engine transitions from the pending execution state to the executing state to begin executing the repair actions. The action sequence of the repair strategy is decomposed into atomic operation steps. An atomic operation is the smallest indivisible execution unit, such as stopping a process, starting a process, modifying a configuration file, or restarting a service. For each atomic operation, the preconditions, execution method, postconditions, and expected result are defined. Atomic operations are executed one by one in the order of the action sequence. Before each atomic operation is executed, the preconditions are checked; if satisfied, the operation method is executed. After execution, the postconditions and expected result are checked. If an atomic operation is executed successfully, the state machine transitions to the execution of the next atomic operation. If an atomic operation fails, the state machine transitions to the failure state, terminating the execution of subsequent operations.

[0068] Step 7.5: Status Monitoring and Effect Evaluation During Execution; Based on the execution process of the repair operation, real-time monitoring and effect evaluation technologies are used to obtain system status change data and repair effect judgment results after execution. Before each atomic operation is executed, a snapshot of the current system status is collected. The snapshot includes information such as key performance indicators, service running status, and configuration file content. The snapshot is used for subsequent rollback and recovery. During the execution of the atomic operation, the system status changes are monitored in real time, and data such as performance indicators, log output, and process status are collected after the operation is executed. After the atomic operation is completed, the system status before and after execution is compared to evaluate whether the operation has produced the expected effect. The judgment criteria for the expected effect are determined according to the repair goal. If the repair goal is to restart the faulty service, the judgment criterion is that the service process status changes to running. If the repair goal is to reduce CPU utilization, the judgment criterion is that the CPU utilization drops to the normal range. A multi-indicator comprehensive evaluation method is adopted to check not only whether the direct repair goal has been achieved, but also whether side effects have occurred, such as whether other performance indicators have deteriorated, whether new alarms have appeared, and whether related services have been affected. If the comprehensive evaluation shows that the repair effect is good and there are no negative impacts, the operation is judged to be successful, and subsequent atomic operations continue to be executed. If the comprehensive assessment shows that the repair effect is not good or has a negative impact, the operation is judged to have failed or posed a risk, and the rollback process is triggered.

[0069] Step 7.6, Failure Rollback Mechanism and State Recovery: Based on the execution effect evaluation results output in Step 7.5, when a repair operation fails or causes a negative impact, a rollback recovery mechanism is adopted to restore the system to its pre-execution state. The goal of the rollback mechanism is to quickly restore the system to its stable pre-execution state when a repair operation fails, avoiding exacerbating anomalies or triggering new faults. The state machine transitions from the execution failure state to the rollback in progress state, and the execution engine initiates the rollback process. The rollback process proceeds in reverse order of atomic operations. For each executed atomic operation, its corresponding inverse operation is executed. The inverse operation is the reverse of the atomic operation and can undo the effect of the atomic operation. For example, if the atomic operation is to start a process, the inverse operation is to stop the process; if the atomic operation is to modify the configuration, the inverse operation is to restore the configuration. Using the state snapshot saved before execution, the critical state of the system is restored to the value recorded in the snapshot. For file modification operations, the original file content is restored using file backup. For configuration change operations, the configuration management interface is called to revert the configuration version. For service state change operations, the service management interface is called to restore the service to its original state. After performing all reverse operations, verify whether the system state has been successfully restored to the state before execution. Compare the current state with the snapshot before execution and check whether key indicators are consistent. If the rollback is successful, the state machine transitions to the rolled-back state, the rollback log is recorded, and the operations and maintenance personnel are notified to repair the failed rollback. If the rollback fails, an alarm is triggered, requesting manual intervention.

[0070] Step 7.7, Recording and Analysis of Execution Logs: Based on the entire repair execution process, log recording and post-event analysis techniques are used to obtain a complete execution process log and performance analysis report. The execution log records information throughout the entire repair execution process, including the execution start time, execution end time, execution strategy content, list of decomposed atomic operations, execution time and result of each atomic operation, status monitoring data, effect evaluation results, rollback operation records, etc. The log is stored in a structured format for easy subsequent retrieval and analysis. The execution log includes fields such as timestamp, log level, event type, detailed description, and associated data. An execution report is generated for each repair execution, summarizing key execution information, including anomaly description, diagnostic conclusions, the repair strategy used, execution results, execution time, and impact on business operations. The execution report is provided to operations personnel for review, helping them understand the working status of the self-healing system. Statistical analysis is performed on the execution logs to calculate performance indicators such as repair success rate, average repair time, and rollback occurrence rate for different anomaly types. Cases of repair failures are analyzed to identify common causes of failure, providing a basis for optimizing repair strategies and improving execution mechanisms. The execution logs and analysis results are used as inputs for knowledge evolution, which are then used to update the knowledge base and train the model.

[0071] A self-healing system for operational anomalies in server clusters, as shown in Figure 3, is used to execute the aforementioned self-healing method for operational anomalies in server clusters. It includes: a data acquisition module for collecting multi-source operational data from the server cluster, performing data preprocessing and multi-modal feature fusion to obtain a comprehensive representation vector sequence; a topology learning module for acquiring service call logs and network traffic data from the server cluster, combining the comprehensive representation vector sequence to extract topological relationships and learn structures, identifying changes in topological dependencies and incrementally updating to obtain a dynamic topology graph sequence; and an anomaly detection module for predicting server node states based on the comprehensive representation vector sequence and the dynamic topology graph sequence, identifying and initially classifying abnormal nodes to obtain the abnormal node sequence. The system comprises the following modules: a preliminary classification of the set and anomaly types; a root cause localization module, which performs causal inference and anomaly root cause localization based on the dynamic topology graph sequence and the set of anomaly nodes, and obtains root cause node identifiers; a diagnostic reasoning module, which performs rule reasoning based on the root cause node identifiers and the preliminary classification of anomaly types, identifies the anomaly types of root cause nodes, and obtains diagnostic identifiers; a policy generation module, which acquires historical operation and maintenance data of the server cluster, constructs a policy network, verifies and deploys the policy network, makes decisions based on the diagnostic identifiers, and obtains a set of repair policy candidates; and a self-healing execution module, which performs multi-dimensional risk assessment and graded processing based on the set of repair policy candidates, monitors the execution effect in real time and performs failure rollback, and obtains a repair execution result report and a status recovery record.

[0072] In one embodiment of the present invention, a specific example is provided: During a business promotion activity, an e-commerce platform experienced a significant increase in server cluster load, resulting in performance anomalies and service response timeouts on multiple service nodes. The application process of the self-healing system for operational anomalies of the present invention in this scenario is as follows.

[0073] The system's data acquisition agent continuously collects operational data from various server nodes in the server cluster. 120 seconds after the start of the promotional activity, performance metrics on multiple nodes exhibited abnormal fluctuations. Table 1 shows examples of some of the abnormal node performance metrics collected by the system.

[0074] Table 1: Example of performance index data for abnormal nodes;

[0075] The system's cross-modal fusion module integrates multi-source data such as performance metrics, log events, and alarm information to generate a comprehensive representation vector for each node. The dynamic graph learning module constructs a real-time topology graph based on service call relationships, identifying Node-B as the core node of the order processing service, with Node-A, Node-C, and Node-D as its downstream nodes. The graph spatiotemporal network module performs spatiotemporal correlation analysis on node states, completing anomaly detection 3 seconds after an anomaly occurs and identifying Node-B, Node-A, Node-C, and Node-E as anomalous nodes. The causal inference module uses Granger causality testing to determine that Node-B is the root cause node, and the anomaly propagates from Node-B to downstream Node-A and Node-C. The hybrid diagnostic module diagnoses the root cause node Node-B. The rule reasoning module identifies that Node-B's CPU and memory usage are consistently high, accompanied by a large number of thread creation logs. The pattern recognition module, through a deep learning model, outputs a diagnostic conclusion that the resource bottleneck is caused by connection pool exhaustion. After evidence fusion, the anomaly type is determined to be improper application-layer connection pool configuration, with a confidence level of 0.89. The strategy learning module generates a candidate set of repair strategies, including adjusting connection pool parameters, restarting application services, and horizontally scaling node capacity. After risk assessment, the system selects adjusting connection pool parameters as the preferred strategy. The execution engine automatically adjusts the maximum number of connections in Node-B's connection pool from 200 to 500 and the connection timeout from 5 seconds to 10 seconds. Fifteen seconds after the repair is executed, Node-B's CPU utilization drops to 0.68, memory utilization drops to 0.61, service response time returns to normal, and the anomaly score of downstream nodes also drops to the normal range. The system evaluates the repair effect as successful, constructs this anomaly handling process as a case and stores it in the case library, extracts diagnostic rules and adds them to the knowledge base, and uses this case for incremental training to improve recognition capabilities.

[0076] The entire anomaly handling process, from detection to repair completion, took only 18 seconds. Compared to the traditional manual handling method with an average response time of 15 minutes, this improved the speed and efficiency of anomaly handling and effectively ensured business continuity and user experience during promotional activities.

[0077] Table 2 shows some examples of abnormal node log event data collected by the system.

[0078] Table 2: Example of abnormal node log event data;

[0079] As can be seen from this application example, the operation and maintenance anomaly self-healing method and system provided by the present invention can quickly and accurately detect anomalies, locate root causes, intelligently diagnose, and automatically repair them in complex microservice cluster environments, achieving end-to-end automated operation and maintenance self-healing, improving the reliability and operation and maintenance efficiency of server clusters, and has obvious practical value and promotion prospects.

[0080] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A self-healing method for operational anomalies in server clusters, characterized in that, Includes the following steps: Multi-source operation and maintenance data of the server cluster are collected, and data preprocessing and multimodal feature fusion are performed to obtain a comprehensive representation vector sequence. Service call logs and network traffic data of the server cluster are obtained, and combined with the comprehensive representation vector sequence, topological relationship extraction and structure learning are performed to identify changes in topological dependencies and update incrementally to obtain a dynamic topology graph sequence. Based on the comprehensive representation vector sequence and the dynamic topology graph sequence, server node status is predicted, abnormal nodes are identified and preliminarily classified, and the abnormal node set and preliminary classification results of abnormal types are obtained. Based on the dynamic topology graph sequence and the set of abnormal nodes, causal inference and root cause localization are performed to obtain root cause node identifiers; Based on the root cause node identifier and the preliminary classification results of the anomaly type, rule reasoning is performed to identify the anomaly type of the root cause node and obtain the diagnostic identifier; historical operation and maintenance data of the server cluster is obtained, a policy network is constructed, the policy network is verified and deployed, decisions are made based on the diagnostic identifier, and a set of repair policy candidates is obtained; based on the set of repair policy candidates, a multi-dimensional risk assessment is performed and graded processing is carried out, the execution effect is monitored in real time and failure rollback is performed, and a repair execution result report and status recovery record are obtained.

2. The self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The data preprocessing includes: running a data acquisition agent on each server node and automatically adjusting the acquisition frequency according to CPU utilization; obtaining the performance index time series based on the original performance index data stream using a sliding window algorithm; obtaining the structured log event sequence based on the original log event stream using template matching technology; and obtaining the deduplicated alarm list based on the original alarm message stream using a deduplication algorithm.

3. The self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The multimodal feature fusion includes: using a temporal convolutional network to encode features based on a performance index matrix to obtain a performance modality feature vector; performing semantic encoding based on a structured log event sequence to obtain a log modality feature vector, and calculating the attention weight between each event in the structured log event sequence and other events through a self-attention module; using a graph convolutional network to encode a deduplicated alarm list to obtain an alarm modality feature vector, and constructing an alarm association graph based on the spatiotemporal proximity and type correlation of alarms; using a recurrent neural network to encode a configuration modality feature vector based on configuration change history; and using a multi-head attention mechanism to calculate cross-modal attention weights, and performing weighted fusion based on the attention weights and the performance modality feature vector, log modality feature vector, alarm modality feature vector, and configuration modality feature vector to obtain a comprehensive representation vector sequence.

4. A self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The steps for obtaining the dynamic topology graph sequence include: based on service call log data in the server cluster, statistically analyzing the call frequency and average response time between each pair of services within a fixed time window to obtain a service dependency graph; based on network traffic data in the server cluster, using traffic statistics and pattern recognition algorithms to obtain the communication relationships and traffic characteristics between server nodes, identifying the application layer protocol of communication based on port number and protocol type to obtain a node communication graph; based on the service dependency graph, node communication graph, and comprehensive representation vector sequence, using a graph structure learning algorithm to obtain a complete topology graph and infer potential implicit dependencies; and using a sliding window mechanism and incremental update algorithm to obtain a dynamically updated topology graph sequence, comparing the topology graph of the new window with the topology graph of the previous window to identify changes in the topology graph.

5. A self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The steps for obtaining the set of abnormal nodes and the preliminary classification results of abnormal types include: constructing a graph spatiotemporal convolutional network based on the comprehensive representation vector sequence and the dynamic topology graph sequence; using the graph spatiotemporal convolutional network and the node representation vectors and topology graph at the current and historical moments, using a sequence prediction method to obtain the state prediction value of each node; using the state prediction value and the actually observed node state value, using a deviation metric and scoring mapping method to obtain the abnormal score of each node; using the distribution of abnormal scores at historical moments and the current operating status of the cluster, using statistical analysis and adaptive adjustment algorithms to obtain an adaptive threshold; and using the node abnormal score and the adaptive threshold, using a threshold comparison and classification method to obtain the set of abnormal nodes and the preliminary classification results of abnormal types.

6. A self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The steps for obtaining the root cause node identifier include: based on the set of anomalous nodes and the historical state time series of the nodes, using the Granger causality test method to obtain the temporal causal relationship between anomalous node pairs; based on the causal relationship and significance level of the node pairs, using a comprehensive scoring method to obtain the causal strength score; based on the causal strength score of the anomalous node pairs, using a graph construction method to obtain a causal dependency graph describing the causal dependence relationship between anomalous nodes; based on the causal dependency graph, using a topology analysis method to obtain the starting node of the anomalous propagation chain as a candidate root cause node; and based on the priority list of candidate root cause nodes and the state anomality score, using a comprehensive judgment method to obtain the root cause node identifier and the anomalous propagation path.

7. A self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The steps for obtaining diagnostic identifiers include: collecting experience from operation and maintenance experts and converting it into IF-THEN rule form; matching abnormal features with rules to trigger diagnostic candidates; designing an anomaly classification model and using a fully connected neural network to obtain pattern recognition conclusions; fusing rule reasoning and pattern recognition conclusions; and using a confidence threshold to determine the diagnostic identifiers based on the fusion results.

8. A self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The steps for obtaining the repair strategy candidate set include: obtaining a digital twin simulation system based on historical operation and maintenance data and system configuration parameters of the server cluster environment using a data-driven modeling method; obtaining a formal description of self-healing strategy learning based on the digital twin simulation system using a Markov decision process framework; obtaining an optimal policy network based on the Markov decision process model and the digital twin simulation system using a deep reinforcement learning algorithm; obtaining a policy network that satisfies safety constraints using a constrained reinforcement learning method based on the risk characteristics of the repair operation; and obtaining the repair strategy candidate set by combining diagnostic identifiers and current cluster state information using a policy inference method.

9. A self-healing method for operational anomalies in server clusters according to claim 1, characterized in that, The steps for obtaining the repair execution result report and state recovery record include: based on each candidate strategy in the repair strategy candidate set, using a multi-dimensional risk assessment model to obtain the risk level classification of each strategy; based on the repair strategies assessed as low-risk, using an automatic execution queue scheduling method to obtain an automatic execution task queue sorted by priority; based on the repair strategies assessed as medium-risk and high-risk, using a repair suggestion generation and approval process mechanism to obtain a repair suggestion report awaiting manual confirmation; based on the repair strategies to be executed, using state machine control and operation decomposition technology to obtain the repair execution process, decomposing the action sequence of the repair strategy into atomic operation steps; based on the execution process of the repair operation, using real-time monitoring and effect evaluation technology to obtain the repair execution result report; and when a repair operation failure or negative impact is detected, using a rollback recovery mechanism to obtain the state recovery record.

10. A self-healing system for operation and maintenance anomalies in server clusters, characterized in that, A self-healing method for operational anomalies of a server cluster, as described in any one of claims 1-9, comprises: a data acquisition module for acquiring multi-source operational data of the server cluster, performing data preprocessing and multi-modal feature fusion to obtain a comprehensive representation vector sequence; a topology learning module for acquiring service call logs and network traffic data of the server cluster, combining the comprehensive representation vector sequence to extract topological relationships and learn structures, identifying changes in topological dependencies and incrementally updating to obtain a dynamic topology graph sequence; and an anomaly detection module for predicting server node states based on the comprehensive representation vector sequence and the dynamic topology graph sequence, identifying and initially classifying abnormal nodes to obtain an abnormal node set and an initial anomaly type. The system comprises the following modules: a classification module for initial steps; a root cause localization module, which performs causal inference and root cause localization based on the dynamic topology sequence and the set of abnormal nodes, and obtains root cause node identifiers; a diagnostic reasoning module, which performs rule reasoning based on the root cause node identifiers and the preliminary classification results of the anomaly types, identifies the anomaly types of the root cause nodes, and obtains diagnostic identifiers; a strategy generation module, which acquires historical operation and maintenance data of the server cluster, constructs a strategy network, verifies and deploys the strategy network, makes decisions based on the diagnostic identifiers, and obtains a set of candidate repair strategies; and a self-healing execution module, which performs multi-dimensional risk assessment and graded processing based on the set of candidate repair strategies, monitors the execution effect in real time and performs failure rollback, and obtains a repair execution result report and a status recovery record.

Images