System fault mining method for log data

Abstract

The invention provides a log data-oriented system fault mining method, which belongs to the technical field of information, and comprises the following steps: collecting original log data of a system, and preprocessing original system logs, including analysis, key information extraction, vectorization and generation of a structured log event sequence; processing the log event sequence by adopting a two-stage model fusing anomaly detection and mode clustering so as to distinguish accidental anomaly and a real fault mode; and for each candidate fault mode cluster, analyzing a time sequence causal relationship between internal events, and positioning a root cause event according to the causal flow score of the node. The method has the advantages that dependence on artificial experience is reduced through full-process automatic modeling and analysis; according to the method, log analysis, feature extraction, anomaly detection and root cause positioning are all automatically completed by adopting a preset algorithm process, and the detection effect is maintained through self-adjustment of the model, so that the flexibility and sustainability of fault mining are improved.

Description

Technical Field

[0001] This invention belongs to the field of information technology, specifically relating to a system fault mining method for log data. Background Technology

[0002] With the widespread adoption of internet services and the evolution of system architecture towards microservices and cloud-native architectures, modern software systems are becoming increasingly complex. During runtime, these systems generate massive amounts of log data, which are crucial sources of information recording system status, user behavior, and application errors. How to automatically, promptly, and accurately identify system failures, predict potential risks, and pinpoint root causes from this massive, unstructured log data has become a core challenge for intelligent operations and maintenance.

[0003] Existing log fault detection methods mainly rely on manual experience to set rules, which is difficult to adapt to dynamic system changes and massive logs, resulting in poor flexibility and scalability, and high false positive and false negative rates. Summary of the Invention

[0004] In view of this, the present invention addresses the shortcomings of the prior art by providing a system fault mining method for log data. To solve the above-mentioned technical problems, the technical solution adopted by the present invention includes: collecting the raw log data of the system, preprocessing the raw system logs, including parsing, extracting key information, vectorizing, and generating a structured log event sequence. A two-stage model integrating anomaly detection and pattern clustering is used to process the log event sequence to distinguish between occasional anomalies and real fault modes. The first anomaly detection stage is used to quickly filter out candidate anomaly events, and the second pattern clustering stage is used to cluster the candidate anomaly events to form candidate fault mode clusters. For each candidate fault mode cluster, the temporal causal relationship between its internal events is analyzed, a directed fault propagation graph is generated, and the root cause event is located based on the causal flow score of the nodes in the directed graph.

[0005] Furthermore, the raw log data of the collected system is preprocessed, including parsing, key information extraction, and vectorization, to generate structured log event sequences. Methods include: The system collects raw log data streams and parses each log entry using a parse tree template algorithm to extract constant event templates and variable parameter values. Assign a unique event type identifier to each event template Combined with timestamps and parameter dictionaries, a structured log event sequence is generated. ,in A dictionary containing key parameters corresponding to the event; The log event sequence is converted into a numerical feature vector; the conversion includes: converting timestamps into offsets relative to the start point of the sequence, performing one-hot encoding on the event type identifier EventID, and converting the parameter dictionary... The numerical parameters are directly introduced, and the classification parameters are introduced after being encoded, together forming the feature vector.

[0006] Furthermore, in the first anomaly detection stage, an improved isolated forest algorithm is used. The improved isolated forest algorithm based on dynamic branching factor and weighted path length is used to perform preliminary anomaly scoring on the numerical feature vector after the log event sequence is transformed, and to identify the abnormal event points. The second mode clustering stage employs a density-based clustering algorithm, whose clustering radius parameter is adaptively adjusted according to the distribution density of the candidate abnormal events.

[0007] Furthermore, the improved isolated forest algorithm processes the numerical feature vectors using the following methods: Dynamic branching factor adjustment: The numerical feature vector is divided into time windows of fixed duration, and the frequency of occurrence of each event type within each time window is statistically analyzed to form a multi-dimensional feature vector. When dividing each node in the isolation tree, if the event type corresponding to the selected dividing feature is a high-frequency event, the number of branches of that node is increased. The high-frequency event refers to its normalized frequency. Exceeding the preset threshold events; branching factors ,in, Set the number of basic branches to 2. The total number of training samples is γ, which is the adjustment slope and ranges from [0.5, 2]. Based on the isolation tree forest constructed by the dynamic branching factor adjustment, for any data point x in the numerical feature vector to be evaluated, anomaly evaluation is performed by calculating its path length in each isolation tree. The weighted path length includes: Calculate each event type based on historical fault data. weight The calculation formula is: ,in, It represents the conditional probability of this event occurring when a historical failure has occurred. δ is the prior probability of the event, and λ is a smoothing hyperparameter used to prevent calculation anomalies when the probability is zero; For any data point x in the numerical feature vector, its event type is: The weight is Its weighted path length is defined as an adjustment to the original average path length: ,in It is the average path length of all trees in the forest; Final weighted outlier score , where ψ is the sample size and c(ψ) is the standardization factor; This is a standardized anomaly probability index with a value range of (0,1). The closer the score is to 1, the higher the probability that the data point x is an anomaly.

[0008] Furthermore, the method for clustering the candidate abnormal events in the second pattern clustering stage includes: Candidate set construction: The anomaly scores Above the dynamic threshold The event is included in the candidate abnormal event set. ; Distance matrix calculation: to measure the set of candidate anomalies Similarity between events, in order to calculate The feature distance between all pairs of events, including timestamps, event type encoding, and embedded representations of key parameters; Density estimation: Calculate the distribution of all distance values ​​and take the p-th quantile as the neighborhood radius ε for the density clustering algorithm, i.e. , where D is the distance set; according to The size N is dynamically adjusted for quantile p: ,in The maximum quantile benchmark is used, and β is the attenuation coefficient, with a value range of [0.05, 0.2], to ensure that a smaller p value is used to obtain more refined clustering when outliers are dense; Fault mode cluster generation: Based on the neighborhood radius ε, a density-based clustering algorithm is used to cluster the candidate abnormal event set. Clustering is performed to divide the candidate abnormal event set into multiple density-connected subsets, and each subset constitutes a candidate fault mode cluster.

[0009] Furthermore, the method for calculating the distance matrix includes: The timestamp, event type encoding, and embedded representation of key parameters are standardized respectively; Assign a preset weight coefficient to each type of feature to construct a weighted composite feature vector; For the set of candidate abnormal events The weighted distance between any two events i and j in the dataset. Represented as: in, , , These represent the standardized timestamp feature, event type encoding vector, and normalized parameter embedding vector of event i, respectively. , , For the corresponding weight coefficients, satisfying .

[0010] Furthermore, the method for the clustering operation includes: The DBSCAN density clustering algorithm is used to cluster the candidate abnormal event set; With the neighborhood radius As a neighborhood radius parameter, set the minimum number of points. The Based on the size of the candidate abnormal event set Confirmed, indicated as: ; Iterate through each event point in the candidate abnormal event set, and if the neighborhood radius of that point... The neighborhood contains at least If a point is identified, then that point is marked as the core point. Starting from any unvisited core point, find all points that are density reachable from that point through density reachability, forming a new cluster of candidate failure modes; Points that do not belong to any cluster are marked as noise points.

[0011] Furthermore, for each candidate fault mode cluster, the method of analyzing the temporal causal relationships between its internal events, generating a directed fault propagation graph, and locating the root cause event based on the causal flow score of the nodes in the directed graph includes: Sequence extraction: For each candidate fault mode cluster, sort all its events by timestamp, extract the event type sequence, ignoring specific parameters; Cause-effect graph construction: A constraint-based PC algorithm is used, with event type as nodes and time lag as constraints, to perform conditional independence tests; the time lag is determined based on the autocorrelation function of the event sequence. For any two event types X and Y, given other event types and their lag values, if the lag value of X has a significant predictive power for Y, then a causal edge from X to Y is established, and the weight of this edge is determined by the significance measure of the lag term of X in the conditional independence test. Root cause determination: Calculate the causal flow score of each node in the causal graph. The causal flow score is the sum of the weights of all directed edges originating from that node minus the sum of the weights of all directed edges ending at that node. The node with the highest causal flow score is determined as the root cause event.

[0012] Furthermore, it also includes a failure impact surface analysis: In the generated directed graph of fault propagation, starting from the root cause event node, a depth-first or breadth-first traversal is performed to extract all reachable nodes. The event types corresponding to the extracted nodes are summarized into the potential impact chain of this failure, and an impact chain report is output.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By automating the entire modeling and analysis process, the reliance on human experience is reduced. This application uses a preset algorithm process to automatically complete everything from log parsing and feature extraction to anomaly detection and root cause localization, avoiding the subjectivity and limitations brought about by manually setting static rules. This automated processing method gives the method stronger adaptability, enabling it to maintain the detection effect through model self-adjustment when the system state or log pattern changes, thereby improving the flexibility and sustainability of fault discovery.

[0014] 2. When dealing with massive, high-speed generated log data, this application employs an improved isolated forest algorithm and an adaptive density clustering algorithm, which balance processing efficiency and model sensitivity in their design. The improved isolated forest algorithm enhances its ability to process high-dimensional data through mechanisms such as dynamic branching factors, while the adaptive clustering algorithm dynamically adjusts parameters based on the distribution characteristics of anomalous events. This design enables the method to complete analysis with acceptable time complexity when processing large-scale data, demonstrating good scalability.

[0015] 3. By integrating anomaly detection and pattern clustering into a two-stage model, this solution enhances the ability to identify real failure modes. The first stage broadly screens for anomalous events, while the second stage clusters these events to distinguish between sporadic anomalies and persistent failure modes. This helps reduce the misjudgment of isolated, harmless anomalies as failures, and also prevents real group failure events from being missed by discovering potential failure mode clusters, thereby improving the overall accuracy and reliability of failure detection. Attached Figure Description

[0016] The present invention will now be described in further detail with reference to the accompanying drawings.

[0017] Figure 1 : A schematic diagram of the process in this invention. Detailed Implementation

[0018] To better understand the present invention, the content of the invention is further clearly illustrated below with reference to embodiments and accompanying drawings. However, the scope of protection of the present invention is not limited to the embodiments described below. Numerous specific details are set forth in the following description to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the present invention can be practiced without one or more of these details.

[0019] Example 1: See Figure 1 This embodiment of a system fault mining method based on log data includes, S1. Collect raw log data from the system, preprocess the raw system logs, including parsing, key information extraction, vectorization, and generating structured log event sequences; including: The system collects raw log data streams and parses each log entry using a parse tree template algorithm to extract constant event templates and variable parameter values. The parse tree template algorithm analyzes the syntactic structure of log messages to parse unstructured raw logs into structured data. Its purpose is to extract constant event templates and variable parameter values, thereby classifying logs with different content but the same pattern into the same event type and achieving preliminary standardization of log data. Assign a unique event type identifier to each event template Combined with timestamps and parameter dictionaries, a structured log event sequence is generated. ,in By constructing a structured sequence containing a dictionary of key parameters corresponding to events, the messy raw log stream is transformed into a standard data unit with a unified timestamp, event type, and parameter dictionary, providing a consistent and well-organized data foundation for subsequent machine learning-based analysis. The log event sequence is converted into a numerical feature vector. This conversion includes: converting timestamps to offsets relative to the start of the sequence to eliminate interference from excessively large absolute timestamp values ​​on the model, while preserving the relative temporal information of the events; performing one-hot encoding on the event type identifier (Event ID) to convert discrete categorical variables into binary vectors, preventing the model from mistakenly believing there is a numerical order relationship between event type IDs; and converting the parameter dictionary... Numerical parameters are directly introduced, and classification parameters are encoded and then introduced to form a feature vector. The structured event sequence containing heterogeneous data types (timestamps, categorical IDs, numerical / classification parameters) is finally transformed into a pure numerical feature vector, thereby meeting the input data format requirements of subsequent machine learning algorithms such as Isolation Forest, and ensuring that the algorithm can be effectively applied.

[0020] S2. The log event sequence is processed using a two-stage model that integrates anomaly detection and pattern clustering to distinguish between occasional anomalies and real fault modes. The first anomaly detection stage is used to quickly filter out candidate anomaly events, and the second pattern clustering stage is used to cluster the candidate anomaly events to form candidate fault mode clusters. S21. The first anomaly detection stage employs an improved isolated forest algorithm. This algorithm, based on dynamic branching factors and weighted path lengths, performs preliminary anomaly scoring on the feature vectors transformed from the log event sequence, identifying anomalous event points. This algorithm is suitable for high-dimensional data and has high detection efficiency. Its purpose is to perform preliminary anomaly scoring on the feature vectors transformed from the log event sequence, achieving rapid and unsupervised screening of candidate anomalous event points. This stage focuses on efficiently identifying anomalous points deviating from conventional patterns from massive amounts of normal events, providing a high-quality candidate set for subsequent analysis; including: S211, Dynamic Branching Factor Adjustment: The numerical feature vector is divided into fixed-duration time windows, and the frequency of occurrence of each event type within each time window is statistically analyzed to form a multi-dimensional feature vector. When dividing each node in the isolation tree, if the selected partition feature corresponds to a high-frequency event, the number of branches of that node is increased. By creating a more refined partition for high-frequency event types, the algorithm's sensitivity and ability to detect and distinguish subtle abnormal patterns lurking in normal high-frequency traffic are improved. The high-frequency event refers to its normalized frequency. Exceeding the preset threshold events; branching factors ,in, Set the number of basic branches to 2. The total number of training samples is γ, which is the adjustment slope and ranges from [0.5,2]. By increasing the branching factor of high-frequency events, the detection sensitivity of small abnormal patterns lurking in normal high-frequency flow is improved. S212. Based on the isolation tree forest constructed by adjusting the dynamic branching factor, for any data point x in the numerical feature vector to be evaluated, anomaly evaluation is performed by calculating its path length in each isolation tree. The weighted path length includes: Weight calculation: Calculate the weight for each event type based on historical fault data. weight The calculation formula is: ,in, It represents the conditional probability of this event occurring when a historical failure has occurred. δ and λ are the prior probability of the event, and smoothing hyperparameters are used to prevent the calculation of anomalies when the probability is zero. The introduction of weights based on historical probabilities has the benefit of integrating domain prior knowledge into the model, enabling the algorithm to focus more on event types that are more correlated with historical failures, thereby improving the interpretability and relevance of anomaly scoring. Weighted path calculation: For any data point x in the numerical feature vector, its event type is The weight is Its weighted path length is defined as an adjustment to the original average path length: ,in It is the average path length of all trees in the forest. For events with high weight (i.e., high correlation with faults), their weighted path length will be shortened to highlight their anomalous probability in the final score. Outlier score calculation: final weighted outlier score , where ψ is the sample size and c(ψ) is the standardization factor; The standardized anomaly probability index has a value range of (0,1). The closer the score is to 1, the higher the probability that the data point x is an anomaly. The standardized score provides a unified anomaly measurement standard, laying a reliable foundation for setting dynamic thresholds for candidate event screening in subsequent stages.

[0021] S22. The second mode clustering stage employs a density-based clustering algorithm, whose clustering radius parameter is adaptively adjusted according to the distribution density of the candidate anomaly events. Its advantage lies in its ability to discover clusters of arbitrary shapes without pre-setting the number of clusters, making it suitable for mining fault modes with unknown morphologies. The purpose is to cluster the candidate anomaly events, with the clustering radius parameter adaptively adjusted according to the distribution density of the candidate anomaly events. Clustering the candidate anomaly events includes: S221. Candidate set construction: The abnormal scores are... Above the dynamic threshold The event is included in the candidate abnormal event set. The dynamic threshold Take the q quantile of all window anomaly scores, where q ranges from [0.85, 0.98]. The threshold can dynamically change with the overall distribution of anomaly scores within different time windows, avoiding the possibility of false screening or missed screening when the number of anomalies fluctuates due to a fixed threshold, thereby ensuring the stability of the candidate set quality. S222, Distance matrix calculation: to measure the set of candidate abnormal events Similarity between events, in order to calculate The distance between all pairs of events is calculated using features including timestamps, event type encoding, and embeddings of key parameters. These embeddings are generated using a pre-trained word vector model or trained using word vector training methods based on the contextual co-occurrence of parameter values ​​in the log sequence. By comprehensively considering multi-dimensional features such as the time, type, and parameter context of events, this calculation aims to provide a comprehensive and accurate similarity metric for subsequent density clustering, thereby more accurately reflecting the inherent relationships between anomalous events. The distance matrix calculation includes: The timestamps, event type codes, and embedding representations of key parameters are standardized. Standardization eliminates the influence of dimensional differences between different feature dimensions, preventing certain features from dominating the overall distance calculation due to their large numerical range, and ensuring that each feature contributes comparablely to the similarity metric. The timestamps are converted into offsets relative to the start of the sequence and Z-score standardized. This conversion preserves the relative temporal information of the events while eliminating the numerical scale effect of absolute timestamps. The event type codes are treated as one-hot encoded vectors. This ensures the orthogonality between different event types and avoids spurious order relationships that may be introduced by numerical encoding. The embedding representations of key parameters are L2 normalized. This operation unifies the scale of the embedding vectors to the unit norm, making the distance metric focus more on semantic differences in the vector direction rather than the magnitude. Preset weight coefficients are assigned to each type of feature to construct a weighted composite feature vector. By introducing configurable weight coefficients, this application can differentiate the importance of different feature dimensions in the fault mode distinction according to the specific application scenario, thereby enhancing the adaptability of distance metric to domain knowledge. For the set of candidate abnormal events The weighted distance between any two events i and j in the dataset. Represented as: in, , , These represent the standardized timestamp feature, event type encoding vector, and normalized parameter embedding vector of event i, respectively. , , For the corresponding weight coefficients, satisfying It provides a multi-dimensional similarity evaluation mechanism that comprehensively considers the timing of event occurrence, semantic type, and parameter context, which can more comprehensively capture the potential correlation between abnormal events and lay the foundation for generating high-quality distance matrices for subsequent density clustering algorithms.

[0022] S223, Density Estimation: Calculate the distribution of all distance values ​​and take the p-th quantile as the neighborhood radius ε for the density clustering algorithm, i.e. , where D is the distance set; the neighborhood radius ε is determined based on the actual spatial distribution characteristics of the current candidate abnormal event set, rather than a preset fixed value, which enables the clustering algorithm to adapt to abnormal event sets of different sizes and densities; according to The size N is dynamically adjusted for quantile p: ,in The maximum quantile is set as the reference value, with a range of [0.8, 0.95], and β is the attenuation coefficient, with a range of [0.05, 0.2]. This ensures that a smaller p value is used to obtain finer clustering when outliers are dense. When the number of candidate outliers N increases (outliers are dense), the quantile p is automatically reduced, thereby selecting a smaller neighborhood radius ε. This enables finer clustering granularity in high-density areas and effectively distinguishes multiple possible coexisting fault modes. S224. Fault mode cluster generation: Based on the neighborhood radius ε, a density-based clustering algorithm is used to cluster the candidate abnormal event set. Clustering is performed to divide the candidate abnormal event set into multiple density-connected subsets, each subset constituting a candidate fault mode cluster. Through the above adaptive parameter adjustment and clustering operation, discrete abnormal events can be effectively organized into fault mode clusters with clear outlines, providing a structured analysis object for subsequent root cause localization. Density-based clustering algorithms are used to analyze the candidate abnormal event set. Clustering operations include the following methods: The DBSCAN density clustering algorithm is used to cluster the candidate abnormal event set. The advantage of choosing the DBSCAN algorithm is that it can automatically discover clusters of arbitrary shapes and does not require pre-specifying the number of clusters, which is particularly suitable for fault mode mining scenarios with unknown shapes. With the neighborhood radius As a neighborhood radius parameter, set the minimum number of points. The Based on the size of the candidate abnormal event set Confirmed, indicated as: The technical effect of this parameter setting rule is that it enables MinPts to adaptively adjust with the size of the abnormal event set, avoids setting too high a threshold in a small set which would prevent effective clusters from forming, and prevents setting too low a threshold in a large set which would misclassify noise as clusters, thereby achieving a balance between clustering sensitivity and reliability. Iterate through each event point in the candidate abnormal event set, if the point The neighborhood contains at least one If a point is identified, it is marked as a core point. The identification of core points is the basis for the DBSCAN algorithm to build clusters. The purpose is to ensure that each cluster contains a sufficient density of core members, thereby ensuring the correlation strength of events within the cluster. Starting from any unvisited core point, all points that are density reachable from that point are found through density reachability, forming a new candidate fault mode cluster. The cluster expansion method based on density reachability can effectively aggregate discrete but density-connected abnormal events, naturally forming a cluster structure corresponding to the potential fault modes. By marking points that do not belong to any cluster as noise points and processing them separately, this application can effectively distinguish between fault events with group characteristics and true sporadic isolated anomalies, thereby further improving the accuracy of fault mode detection.

[0023] S3. For each candidate fault mode cluster, analyze the temporal causal relationships between its internal events, generate a directed fault propagation graph, and locate the root cause event based on the causal flow scores of the nodes in the directed graph; including: Constructing a directed graph of fault propagation using a time-series causal discovery algorithm, specifically including: Sequence Extraction: For each candidate fault mode cluster, all events contained therein are sorted by timestamp, and the event type sequence is extracted. Specific parameters are ignored, and specific fault instances are abstracted into generalized event type sequences, thereby eliminating noise caused by parameter differences and better focusing on the temporal correlation patterns between event types. Cause-effect graph construction: A constraint-based PC algorithm is used to distinguish between true causal relationships and simple temporal correlations through the system's conditional independence test. It does not require a pre-set data generation model and is suitable for complex system log data. The conditional independence test is performed using event type as a node and temporal lag as a constraint. The temporal lag is determined based on the autocorrelation function of the event sequence. Specifically, it is the lag order corresponding to the first time the autocorrelation coefficient falls below a preset threshold or the first local maximum value occurs. This determination method can capture the most significant time dependencies between events and provide reasonable temporal constraints for causal discovery. For any two event types X and Y, given other event types and their lag values, if the lag value of X has significant predictive power for Y, then a causal edge from X to Y is established. The weight of this edge is determined by the statistical significance measure of the lag term of X in the conditional independence test. For example, the negative logarithm of the p-value of the conditional independence test can be used as the weight, i.e.: By introducing weights based on statistical significance, the strength of causal edges can be quantified, providing a reliable basis for subsequent root cause determination. Root cause determination: Calculate the causal flow score of each node in the causal graph. The causal flow score is the sum of the weights of all directed edges originating from that node minus the sum of the weights of all directed edges ending at that node. This metric effectively measures the net influence of a node in the causal network, considering both its influence as a cause and the impact it receives as a result. The node with the highest causal flow score is determined as the root cause event. This determination criterion can identify the node with the largest net driving force on the entire fault propagation process from a complex causal network, thereby achieving accurate and interpretable root cause localization.

[0024] Fault impact analysis: In the generated directed graph of fault propagation, starting from the root cause event node, a depth-first or breadth-first traversal is performed to extract all reachable nodes. The event types corresponding to the extracted nodes are summarized into the potential impact chain of this failure, and an impact chain report is output. Technical effects of this embodiment: 1. This application improves the accuracy of system fault identification by integrating anomaly detection and pattern clustering in a two-stage processing model. The first stage of anomaly detection focuses on quickly filtering out potential anomalies from massive log events, while the second stage aggregates these isolated anomalies into fault event clusters with common patterns through clustering. This phased processing approach can effectively distinguish between occasional random anomalies and real, persistent fault patterns, thereby reducing the false alarm rate.

[0025] 2. The improved algorithm adopted in this application helps to enhance the detection capability of faults in complex operation and maintenance environments. For example, by introducing dynamic branching factors and weighted path lengths, the sensitivity of the isolated forest algorithm to weak abnormal signals lurking in high-frequency normal events is improved. At the same time, the adaptive adjustment mechanism of the clustering radius parameter enables the method to dynamically optimize the clustering granularity according to the distribution characteristics of the abnormal event set, which helps to improve the adaptability to diverse fault modes.

[0026] 3. By using time-series causal analysis to locate the root cause of a failure and assess its impact, more actionable diagnostic information can be provided. By constructing a directed graph of failure propagation and calculating the causal flow score of nodes, the root cause can be identified from the interconnected events, rather than just at the level of phenomena. In addition, by analyzing the failure propagation path, it helps maintenance personnel to quickly understand the failure impact chain, providing support for the formulation of precise handling measures, which is expected to shorten the system failure recovery time.

[0027] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Any other modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention, as long as they do not depart from the spirit and scope of the technical solutions of the present invention, should be covered within the scope of the claims of the present invention.

Claims

1. A system fault mining method based on log data, characterized in that, include, The system collects raw log data and preprocesses the raw system logs, including parsing, extracting key information, vectorizing, and generating structured log event sequences. A two-stage model integrating anomaly detection and pattern clustering is used to process the log event sequence to distinguish between occasional anomalies and real fault modes. The first anomaly detection stage is used to quickly filter out candidate anomaly events, and the second pattern clustering stage is used to cluster the candidate anomaly events to form candidate fault mode clusters. For each candidate fault mode cluster, the temporal causal relationship between its internal events is analyzed, a directed fault propagation graph is generated, and the root cause event is located based on the causal flow score of the nodes in the directed graph.

2. The system fault mining method for log data as described in claim 1, characterized in that, Methods for collecting raw log data from the system, preprocessing the raw system logs (including parsing, key information extraction, vectorization), and generating structured log event sequences include: The system collects raw log data streams and parses each log entry using a parse tree template algorithm to extract constant event templates and variable parameter values. Assign a unique event type identifier to each event template Combined with timestamps and parameter dictionaries, a structured log event sequence is generated. ,in A dictionary containing key parameters corresponding to the event; The log event sequence is converted into a numerical feature vector; the conversion includes: converting timestamps into offsets relative to the start point of the sequence, performing one-hot encoding on the event type identifier EventID, and converting the parameter dictionary... The numerical parameters are directly introduced, and the classification parameters are introduced after being encoded, together forming the feature vector.

3. The system fault mining method for log data as described in claim 1, characterized in that, The first anomaly detection stage employs an improved isolated forest algorithm. The improved isolated forest algorithm, based on dynamic branching factor and weighted path length, performs preliminary anomaly scoring on the numerical feature vector after the log event sequence is transformed, and identifies the abnormal event points. The second mode clustering stage employs a density-based clustering algorithm, whose clustering radius parameter is adaptively adjusted according to the distribution density of the candidate abnormal events.

4. The system fault mining method for log data as described in claim 3, characterized in that, The improved isolated forest algorithm processes the numerical feature vectors using the following methods: Dynamic branching factor adjustment: The numerical feature vector is divided into time windows of fixed duration, and the frequency of occurrence of each event type within each time window is statistically analyzed to form a multi-dimensional feature vector. When dividing each node in the isolation tree, if the event type corresponding to the selected dividing feature is a high-frequency event, the number of branches of that node is increased. The high-frequency event refers to its normalized frequency. Exceeding the preset threshold events; branching factors ,in, Set the number of basic branches to 2. The total number of training samples is γ, which is the adjustment slope and ranges from [0.5, 2]. Based on the isolation tree forest constructed by the dynamic branching factor adjustment, for any data point x in the numerical feature vector to be evaluated, anomaly evaluation is performed by calculating its path length in each isolation tree. The weighted path length includes: Calculate each event type based on historical fault data. weight The calculation formula is: ,in, It represents the conditional probability of this event occurring when a historical failure has occurred. δ is the prior probability of the event, and λ is a smoothing hyperparameter used to prevent calculation anomalies when the probability is zero; For any data point x in the numerical feature vector, its event type is: The weight is Its weighted path length is defined as an adjustment to the original average path length: ,in It is the average path length of all trees in the forest; Final weighted outlier score , where ψ is the sample size and c(ψ) is the standardization factor; This is a standardized anomaly probability index with a value range of (0,1). The closer the score is to 1, the higher the probability that the data point x is an anomaly.

5. The system fault mining method for log data as described in claim 4, characterized in that, The method for clustering the candidate abnormal events in the second pattern clustering stage includes: Candidate set construction: The anomaly scores Above the dynamic threshold The event is included in the candidate abnormal event set. ; Distance matrix calculation: to measure the set of candidate anomalies Similarity between events, in order to calculate The feature distance between all pairs of events, including timestamps, event type encoding, and embedded representations of key parameters; Density estimation: Calculate the distribution of all distance values ​​and take the p-th quantile as the neighborhood radius ε for the density clustering algorithm, i.e. , where D is the distance set; according to The size N is dynamically adjusted for quantile p: ,in The maximum quantile benchmark is used, and β is the attenuation coefficient, with a value range of [0.05, 0.2], to ensure that a smaller p value is used to obtain more refined clustering when outliers are dense; Fault mode cluster generation: Based on the neighborhood radius ε, a density-based clustering algorithm is used to cluster the candidate abnormal event set. Clustering is performed to divide the candidate abnormal event set into multiple density-connected subsets, and each subset constitutes a candidate fault mode cluster.

6. The system fault mining method for log data as described in claim 5, characterized in that, The method for calculating the distance matrix includes: The timestamp, event type encoding, and embedded representation of key parameters are standardized respectively; Assign a preset weight coefficient to each type of feature to construct a weighted composite feature vector; For the set of candidate abnormal events The weighted distance between any two events i and j in the dataset. Represented as: in, , , These represent the standardized timestamp feature, event type encoding vector, and normalized parameter embedding vector of event i, respectively. , , For the corresponding weight coefficients, satisfying .

7. The system fault mining method for log data as described in claim 5, characterized in that, The clustering operation method includes: The DBSCAN density clustering algorithm is used to cluster the candidate abnormal event set; With the neighborhood radius As a neighborhood radius parameter, set the minimum number of points. The Based on the size of the candidate abnormal event set Confirmed, indicated as: ; Iterate through each event point in the candidate abnormal event set, and if the neighborhood radius of that point... The neighborhood contains at least If a point is identified, then that point is marked as the core point. Starting from any unvisited core point, find all points that are density reachable from that point through density reachability, forming a new cluster of candidate failure modes; Points that do not belong to any cluster are marked as noise points.

8. The system fault mining method for log data as described in claim 5, characterized in that, For each candidate fault mode cluster, the method of analyzing the temporal causal relationships between its internal events, generating a directed fault propagation graph, and locating the root cause event based on the causal flow scores of the nodes in the directed graph includes: Sequence extraction: For each candidate fault mode cluster, sort all its events by timestamp, extract the event type sequence, ignoring specific parameters; Cause-effect graph construction: A constraint-based PC algorithm is used, with event type as nodes and time lag as constraints, to perform conditional independence tests; the time lag is determined based on the autocorrelation function of the event sequence. For any two event types X and Y, given other event types and their lag values, if the lag value of X has a significant predictive power for Y, then a causal edge from X to Y is established, and the weight of this edge is determined by the significance measure of the lag term of X in the conditional independence test. Root cause determination: Calculate the causal flow score of each node in the causal graph. The causal flow score is the sum of the weights of all directed edges originating from that node minus the sum of the weights of all directed edges ending at that node. The node with the highest causal flow score is determined as the root cause event.

9. The system fault mining method for log data as described in claim 8, characterized in that, It also includes a failure impact surface analysis: In the generated directed graph of fault propagation, starting from the root cause event node, a depth-first or breadth-first traversal is performed to extract all reachable nodes. The event types corresponding to the extracted nodes are summarized into the potential impact chain of this failure, and an impact chain report is output.

Images