System log anomaly detection method based on weighted directed graph and isolated forest

Abstract

The invention belongs to the technical field of system log detection, and particularly relates to a system log anomaly detection method based on a weighted directed graph and an isolated forest. In order to enable the detection method to efficiently process high-dimensional log data, capture a time sequence dependency relationship and have generalization ability, log events are mapped into nodes in a directed graph, and a transfer relationship between the events is mapped into weighted directed edges, so that a causal relationship, transfer frequency and time dependency between the events are captured. By extracting multi-dimensional features including node degree, edge frequency, time deviation and centrality indexes of a graph, the capability of anomaly detection is enhanced. The isolated forest algorithm utilizes the characteristics to efficiently identify abnormal log modes in an unsupervised learning mode.

Description

Technical Field

[0001] The present invention belongs to the technical field of system log detection, and in particular relates to a system log anomaly detection method based on a weighted directed graph and an isolation forest. Background Art

[0002] System logs are crucial records of the operational status of modern complex software systems, containing a wealth of operational information such as process calls, node communications, resource allocation, and error events. This information provides diverse features for anomaly detection, helping to identify potential system issues such as distributed denial of service (DDoS) attacks, system failures, or unauthorized access. However, execution path-based anomaly detection methods typically rely on meticulous log classification, segmenting them according to dimensions such as processes, nodes, and jobs. Notably, only jobs provide a complete execution path, which generates corresponding log sequences. However, in practical applications, log classification becomes particularly challenging due to complex factors such as process calls and inter-node communication. Even if we can correctly classify processes by tracking process identifiers (PIDs), we cannot guarantee that the resulting logs cover the entire execution path. This is because logs between processes may be distributed across different files, and in a multi-tasking environment, process call log information may be written to different text files in parallel, increasing the complexity of log classification and anomaly detection.

[0003] In real-world distributed systems or highly concurrent environments, log data often exhibits high dimensionality, temporal order, and heterogeneity. For example, in distributed file systems (such as Hadoop HDFS) or supercomputer systems (such as Blue Gene / L), log records not only involve a large number of system operations but also include cross-node interactions and dynamically changing operating states. These characteristics make it difficult for traditional anomaly detection methods, such as rule-based pattern matching or simple statistical analysis, to effectively address abnormal patterns in complex scenarios. In addition, abnormal events are often sparse and hidden, and may only appear in specific time windows or specific execution paths, which further increases the difficulty of detection. Therefore, developing an anomaly detection method that can efficiently process high-dimensional log data, capture temporal dependencies, and possess generalization capabilities has become an important research direction in the field of software reliability engineering.

[0004] In software reliability engineering, anomaly detection often relies on continuously acquired log information to monitor system status in real time and identify abnormal events. Traditional log analysis methods often use timestamps as the axis and perform anomaly detection based on statistical features or sequential patterns in logs. However, when faced with complex system logs, these methods are often limited by the singleness of feature extraction and the model's insufficient generalization ability to unknown anomalies. For example, anomaly detection methods based on deep neural networks, while demonstrating good performance in specific scenarios (such as DDoS attack detection), are inefficient in processing large-scale log data and require a large amount of labeled data for supervised training, which is often difficult to meet in practical applications. To this end, it is hoped that a universal anomaly detection paradigm can be constructed using continuous log sequences or batch log information with timestamps as the axis to reduce the reliance on labeled data and improve the robustness of detection. Summary of the Invention

[0005] In response to the above problems, the present invention provides a system log anomaly detection method based on weighted directed graph and isolation forest.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0007] The present invention provides a system log anomaly detection method based on a weighted directed graph and an isolation forest, comprising the following steps:

[0008] Parse the log sequence to obtain the log key;

[0009] Represent the log keys by a weighted directed graph G and the adjacency matrix M;

[0010] Extract multi-dimensional features based on the weighted directed graph G and the adjacency matrix M;

[0011] Based on the extracted multi-dimensional features, the isolation forest algorithm is used to complete anomaly detection.

[0012] Furthermore, the log key is represented by a weighted directed graph G, specifically:

[0013] Assume that the set of log keys is L = {L1, L2, ..., L n}, where L n Representing a specific log key, the directed graph G = (V, E) consists of the following elements:

[0014] Node set V: represents all elements in the log key set, that is, each log key can correspond to a node in the graph. The expression is as follows:

[0015] L={L1,L2,…,L n};

[0016] Edge set E: represents the transfer relationship between log keys. If a log key L i Followed by another log key L j occurs, then there will be a line from node L in the graph i To node L j Directed edges, using a two-tuple (L i ,L j ) to represent this directed edge, the expression is as follows:

[0017] E={(L i ,L j )|If event L i After the incident L j}.

[0018] Furthermore, the log key is represented by the adjacency matrix M, specifically:

[0019] Assuming there are n log keys, that is, n nodes, construct an n×n matrix M, which is expressed as follows:

[0020]

[0021] in, ω ij Indicates that from L i To L j The transfer weight is determined by the frequency of transfer.

[0022] Furthermore, the extracted multi-dimensional features include:

[0023] In-degree: all points to node v i The sum of the transfer weights of the edges is expressed as follows:

[0024]

[0025] Out-degree: v i The sum of the transfer weights of the edges pointing to other nodes is expressed as follows:

[0026]

[0027] Node frequency: the sum of the node's in-degree and out-degree, expressed as follows:

[0028]

[0029] Edge frequency: edge (v i ,v j ) is the transfer weight, which is expressed as follows:

[0030]

[0031] Time deviation: node v i and node v j The difference in timestamps is expressed as follows:

[0032]

[0033] Average in-degree of adjacent nodes: The average of the in-degree and out-degree of adjacent nodes. The expression is as follows:

[0034]

[0035] Connected components: the number of independent subgraphs in the graph, expressed as follows:

[0036]

[0037] Cosine similarity: used to measure the similarity between two nodes. The expression is as follows:

[0038]

[0039] Degree centrality: reflects the number of connections with other nodes, and the expression is as follows:

[0040] C degree (v i )=d in (v i )+d out (v i );

[0041] Closeness centrality: measures the average distance from a node to all other nodes in the graph, expressed as follows:

[0042]

[0043] Among them, d(v i ,v j ) represents node v i and v j The shortest path distance between them;

[0044] Betweenness centrality: measures the degree to which a node acts as an intermediary between other pairs of nodes, and is expressed as follows:

[0045]

[0046] Among them, σ(s,t) is the number of shortest paths between nodes s and t, σ(s,t|v i ) is through node v i The number of shortest paths;

[0047] Diameter of a graph: The maximum distance between any two nodes in the graph, expressed as follows:

[0048]

[0049] Average path length: The average value of the shortest path length between any two nodes. The expression is as follows:

[0050]

[0051] Furthermore, the use of the isolation forest algorithm to complete anomaly detection is divided into two stages:

[0052] (1) Training phase: Randomly sample from the training set and construct an isolation tree. Several isolated trees form an isolation forest. Randomly select n samples from the sampled sample points as the root nodes of the isolation tree; randomly select a feature dimension and select a partition value p between the maximum and minimum values of the feature dimension; divide the samples into the left subtree and the right subtree according to the p value, and recursively perform this operation on the left subtree and the right subtree until the preset height of the tree is reached or each leaf node has only one sample;

[0053] (2) Anomaly detection stage: traverse each isolated tree in the isolation forest, obtain the expected depth of the test sample, and calculate the anomaly score. The anomaly score formula is as follows:

[0054]

[0055] Where s(x,n) represents the anomaly score of sample x, n represents the number of samples, h(x) represents the path length of sample x from the root node to the leaf node, E[h(x)] represents the expected value of h(x), and c(n) is the normalization constant. The formula is:

[0056]

[0057] H(k)≈ln(k)+γ

[0058] Where H(k) is the harmonic number and γ represents the Euler constant;

[0059] The value of s(x,n) is in the range of (0,1]. If s(x,n)≈1, it means that sample x is likely to be an outlier. If s(x,n)≈0.5, it means that sample x behaves similarly to a normal point. If s(x,n)<<0.5, it means that sample x is very likely to be a normal point.

[0060] Compared with the prior art, the present invention has the following advantages:

[0061] The present invention introduces directed graphs into the log anomaly detection task to overcome the limitations of traditional methods. As a powerful mathematical tool, directed graphs have a wide range of applications in representing systems with directional relationships, such as network traffic analysis, social network modeling, and program execution process analysis. Log sequences are naturally temporal and causal. The transition relationship between events can be modeled as an edge in a directed graph, and the events themselves correspond to nodes in the graph. This representation method can intuitively capture the execution path and event dependency of log sequences, providing a new perspective for anomaly detection. In particular, a log sequence representation method based on a weighted directed graph is proposed. By assigning weights to the edges of the graph (such as transition frequency or time deviation), the correlation strength between events is further quantified, thereby enhancing the richness of feature expression.

[0062] Compared with traditional log analysis methods, the weighted directed graph representation method has the following significant advantages: First, it can comprehensively capture the temporal characteristics, causal relationships, and dynamic changes in log sequences, rather than relying solely on a single statistical indicator. Second, by extracting multi-dimensional graph features (such as node degree, centrality metrics, and graph structural features), it can reveal deep patterns and abnormal behaviors in log data. Finally, combined with the isolation forest algorithm of unsupervised learning, this method can achieve efficient anomaly detection without relying on labeled data, making it particularly suitable for processing unknown abnormal patterns and sparse abnormal events. Compared with anomaly detection methods based on deep neural networks, this method has significant advantages in feature extraction diversity and computational efficiency, and can achieve better classification results and generalization capabilities, especially when processing large-scale and complex log data.

[0063] Experimental results on public datasets such as HDFS and BGL demonstrate that this method demonstrates superior accuracy, robustness, and generalization when detecting known and unknown anomaly patterns compared to baseline methods such as SVM, LogRobust, and DeepLog. This method is suitable for real-time system monitoring and security applications involving complex log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 This is a framework diagram of the system log anomaly detection method of the present invention;

[0065] Figure 2 Schematic diagram of the directed graph representation of log sequences. DETAILED DESCRIPTION

[0066] In order to further illustrate the technical solution of the present invention, the present invention is further described below through examples.

[0067] Example 1

[0068] This embodiment is a system log anomaly detection method based on weighted directed graph and isolation forest, such as Figure 1 As shown, the following steps are included:

[0069] Parse the log sequence to obtain the log key;

[0070] Represent the log keys by a weighted directed graph G and the adjacency matrix M;

[0071] Extract multi-dimensional features based on the weighted directed graph G and the adjacency matrix M;

[0072] Based on the extracted multi-dimensional features, the isolation forest algorithm is used to complete anomaly detection.

[0073] A directed graph (digraph) is a graph structure consisting of a set of nodes (also called vertices) and a set of directed edges. In a directed graph, each edge is an ordered pair (u, v), pointing from node u to node v, where u and v are nodes in the graph. The set of nodes is represented by V, and the set of edges is represented by E. Unlike undirected graphs, the edges in a directed graph are directional, meaning that the starting and ending points of the edge are distinct. Each edge can be a transfer, connection, or dependency from one node to another, and the direction of the edge generally indicates the order of events or causal relationships.

[0074] In log analysis, weighted directed graph representations provide an intuitive and effective framework for modeling relationships between events. Specifically, when mapping log sequences into a weighted directed graph, each log event can be considered a node in the graph, while the transition relationships between events are represented by directed edges. Each directed edge not only reflects the sequence and causal relationships of events but also quantifies the strength or frequency of transitions between events through the edge's weight. This weight can be obtained by calculating the frequency or time deviation of event transitions.

[0075] Figure 2 This diagram shows a directed graph representation of different log sequences. In this graph, each node represents a specific log key, and these nodes are connected by directed edges, representing the causal relationships between log keys. The paths in the graph represent the execution order and flow of the program, clearly showing the execution trajectory from one log key to another.

[0076] A normal log sequence usually contains a series of ordered operations, which are often closely related to the core business processes of the program. Figure 2Route ① in the figure shows a normal log sequence. The transitions between each node along the path are reasonable and smooth, reflecting the program's operational logic. Log keys in this log sequence have a clear causal relationship, indicating that operations occur in the expected order. For example, there might be a login operation, followed by a data read, then a data write, and so on, without any abnormal interruptions or unexpected jumps.

[0077] Compared with normal log sequences, abnormal log sequences show some obvious abnormal characteristics. Figure 2 Route ② in Figure 2 is a typical example of an abnormal log sequence. As can be seen in the figure, path ② contains some illogical jumps, which may include jumping from a log key to a completely unrelated operation or jumping to a state that the program should not enter. This abnormal sequence often reflects abnormal transitions during program execution, which may be caused by program errors, unhandled exceptions, or unexpected branches.

[0078] Furthermore, orphan nodes are a key feature of abnormal log sequences. These nodes lack a causal relationship with other log keys. They represent operations that failed to trigger normally due to program logic errors, resource contention, and other reasons.

[0079] In the mathematical framework of graph theory, log sequences can be represented by directed graphs. Assume that the set of log keys is L = {L1, L2, ..., L n}, where L n Representing a specific log key, the directed graph G = (V, E) consists of the following elements:

[0080] Node set V: represents all elements in the log key set, that is, each log key can correspond to a node in the graph.

[0081] L={L1,L2,…,L n}

[0082] Each node L n Corresponds to a log key, such as login events, file read and write operations, etc.

[0083] Edge set E: represents the transfer relationship between log events. If a log key L i Followed by another log key L j occurs, then there will be a line from node L in the graph i To node L j Directed edges. Using a two-tuple (L i ,L j ) to represent this edge.

[0084] E={(L i ,L j )|If event L i After the incident L j}

[0085] This represents a transfer relationship from a log event.

[0086] To further describe the transfer relationship, we can use the adjacency matrix to represent the log sequence. Assuming there are n log keys (i.e. n nodes), we can construct an n×n matrix M:

[0087]

[0088] in:

[0089]

[0090] In this matrix, ω ij Indicates that from L i to L j The transfer weight is determined by the frequency of transfer.

[0091] Based on a weighted directed graph and its adjacency matrix, we extract multidimensional features. These rich features include not only statistical features but also log sequence features. The extraction of multidimensional features helps improve the analyzability of the graph, revealing deeper structures and relationships, thereby providing stronger support for subsequent machine learning classification tasks. Feature statistics are shown in Table 1:

[0092] Table 1 Feature statistics

[0093]

[0094]

[0095] Among them, closeness centrality measures the average distance from a node to all other nodes in the graph. The smaller the distance, the higher the closeness centrality. i , its closeness centrality is:

[0096]

[0097] d(v i ,v j ) represents node v i and v j The shortest path distance between them.

[0098] Betweenness centrality measures the degree to which a node acts as an intermediary between other pairs of nodes. It reflects the role of the node as a bridge in the graph. For node v i , its betweenness centrality is:

[0099]

[0100] Among them, σ(s,t) is the number of shortest paths between nodes s and t, σ(s,t|v i ) is through node v i The number of shortest paths.

[0101] Based on abnormal behavior of nodes in the graph (such as a sudden increase in indegree or abnormal transfer frequency), malicious attacks and abnormal logins can be detected. Combining time deviations with historical records of network behavior enables earlier identification of potential security issues. By integrating different types of features (such as statistical features and log sequence features), the accuracy and interpretability of the model are enhanced, helping to discover more potential patterns and abnormal behaviors. Comprehensive feature extraction is particularly important for complex weighted directed graphs or event sequence data, especially when dealing with scenarios with time dependencies and dynamic changes.

[0102] Isolation Forest is a machine learning algorithm based on decision trees. It is an unsupervised learning algorithm specifically designed for anomaly detection. The core idea behind Isolation Forest is that because outliers tend to deviate from the majority data, the path length (from the root node to the leaf node) in the decision tree building them is typically shorter, making them easier to segment than normal points. Because it is an unsupervised learning algorithm, it does not require labeled data and relies entirely on the distribution characteristics of the data.

[0103] The task of anomaly detection using isolation forests is divided into two stages:

[0104] (1) Training phase: First, randomly sample from the training set to construct an isolation tree. Several isolated trees form an isolation forest. Randomly select n samples from the sampled sample points as the root nodes of the isolation tree. Second, randomly select a feature dimension and select a partition value p between the maximum and minimum values of the feature dimension; divide the samples into the left subtree and the right subtree according to the p value, and recursively perform this operation on the left subtree and the right subtree until the preset height of the tree is reached or each leaf node has only one sample.

[0105] (2) Anomaly detection stage: traverse each isolated tree in the isolation forest, obtain the expected depth of the test sample, and calculate the anomaly score; the anomaly score formula is as follows:

[0106]

[0107] Where s(x,n) represents the anomaly score of sample x, n represents the number of samples, h(x) represents the path length from the root node to the leaf node of sample x, and E[h(x)] represents the expected value of h(x). The value of s(x,n) is in the range (0,1). If s(x,n)≈1, it means that sample x is likely an anomaly. If s(x,n)≈0.5, it means that sample x behaves similarly to normal points (the path length is close to the average). If s(x,n)<<0.5, it means that sample x is very likely a normal point.

[0108] The normalization constant c(n) is used to normalize the path length to reflect the expected path length of random segmentation. Its formula is as follows:

[0109]

[0110] H(k)≈ln(k)+γ

[0111] Where H(k) is the harmonic number and γ is the Euler constant.

[0112] Algorithm 1 gives the pseudo code of the isolation forest based anomaly detection method.

[0113]

[0114]

[0115] Example 2

[0116] The present embodiment is carried out on a local computer of Windows 11 operating system, and the hardware configuration includes Intel Core i7-8750H processor, 32GB memory, 512GB SSD hard disk, and GPU acceleration is not used. In terms of software environment, Python 3.7 is used as the main programming language, and development and debugging are carried out based on Jupyter Notebook. The core dependency library includes Scikit-learn 1.2.0 (for isolation forest algorithm implementation), NumPy 1.22, Pandas 1.4 (for data processing), and Matplotlib 3.5 (for result visualization). The number of isolated trees in the isolation forest is 100, the subsampling size is 256, the abnormal sample accounts for 0.001, and the sample is not sampled with replacement.

[0117] The datasets used in this example come from the open-source log data platform LogHub, and two representative datasets, HDFS and BGL, were selected. The HDFS dataset, derived from the Hadoop distributed file system's operational logs, records detailed system operational information under both normal and abnormal conditions and is suitable for anomaly detection research in distributed systems. The BGL dataset, derived from the system logs of the BlueGene / L supercomputer, covers a variety of key events, including node status and system failures, and is widely used for fault detection and log analysis in large-scale parallel systems. Both datasets contain anomaly labels, making them suitable for training and evaluating log anomaly detection models.

[0118] HDFS: A distributed file system with approximately 11,175,629 log entries, covering various system operations, file reading and writing, block allocation, and block reporting.

[0119] BGL: Operational logs from the IBM Blue Gene / L supercomputer system, approximately 4,747,963 entries. Log types include system-level events such as node status, task scheduling, and error reports.

[0120] Evaluation metrics: Precision, recall, and F1 score are used as evaluation metrics for anomaly detection tasks:

[0121] Precision:

[0122]

[0123] Recall:

[0124]

[0125] F1 Score:

[0126]

[0127] In order to evaluate the performance of the proposed log anomaly detection algorithm, the following method is selected for comparative experiments.

[0128] SVM-based methods: Support Vector Machine (SVM) is a classic and powerful machine learning algorithm that is widely used in classification, regression, and anomaly detection. It is a supervised learning method that performs particularly well when dealing with high-dimensional datasets and nonlinear problems.

[0129] LogRobust: Traditional anomaly detection methods often perform poorly when faced with high-dimensional features and imbalanced data distribution. To address these issues, LogRobust proposes a robust anomaly detection framework based on Word2Vec, aiming to improve detection accuracy and adaptability.

[0130] DeepLog: DeepLog is a deep learning-based log anomaly detection method. Unlike LogRobust, DeepLog leverages the powerful capabilities of recurrent neural networks (RNNs) to model the temporal characteristics of log data. DeepLog analyzes the patterns of log sequences and detects abnormal behavior that deviates from normal patterns, making it an outstanding performer in log sequence anomaly detection.

[0131] Log anomalies often manifest as patterns or behaviors that have never occurred before. Therefore, to comprehensively evaluate the accuracy and effectiveness of various methods, this embodiment analyzes two scenarios: one is for abnormal patterns that have already appeared in the training data, and the other is for new abnormal patterns that have not appeared in the training data.

[0132] In the task of log anomaly detection, detecting previously unseen anomalous patterns is crucial. These anomalies often represent unknown risks or threats to system operations, such as new cyberattacks, unforeseen system failures, or unusual behavior caused by environmental changes. Because these anomalies are not present in the training data, traditional supervised learning methods often struggle to effectively address them. Therefore, developing detection methods that generalize to unknown anomalies has become a core challenge in the field of anomaly detection. To accurately evaluate the model's anomaly recognition capabilities, the test set is divided into observed and unobserved parts during training.

[0133] Table 2 Performance on observed datasets

[0134]

[0135] Table 2 shows the anomaly detection performance of our proposed method and baseline methods (SVM, Logrobust, and DeepLog) on observed datasets (HDFS and BGL). This evaluation evaluates each method's ability to detect known anomaly patterns in the training set. On the HDFS dataset, our proposed method performs best with a recall of 1.00 and an F1 score of 0.98, outperforming the other methods. On the BGL dataset, our proposed method significantly outperforms the baseline methods in precision (0.96), recall (0.97), and F1 score (0.97), demonstrating its superior accuracy and robustness when faced with complex anomaly patterns. This demonstrates its effectiveness and accuracy in detecting known anomalies.

[0136] Table 3 Performance on unobserved datasets

[0137]

[0138] Table 3 shows the anomaly detection performance of our method and baseline methods on unobserved datasets, aiming to evaluate their generalization ability to novel anomaly patterns not included in the training set. On the HDFS dataset, our method and DeepLog both achieved the best F1 score (0.97), but our method had a higher recall (0.99), indicating that it was more accurate in detecting unobserved anomalies. On the BGL dataset, our method outperformed other methods in precision (0.82), recall (0.92), and F1 score (0.87). Compared to the performance on the observed dataset in Table 2, our method's performance on the unobserved dataset declined slightly, but it still maintained a high recall and F1 score. This indicates that our method generalizes better for anomaly detection.

[0139] Ablation study is an experimental method commonly used in the fields of machine learning and deep learning. By intentionally removing (or replacing) a certain part and observing the changes in model performance, the importance and role of this part in the model can be analyzed.

[0140] The goal of the feature ablation experiment is to evaluate the contribution of each feature or feature group to the model performance, thereby verifying the effectiveness of the feature design and providing a basis for subsequent optimization. In order to evaluate the contribution of each feature group extracted based on the weighted directed graph to the model performance, we conducted a feature ablation experiment. By removing the feature groups listed in Table 1 one by one, we observed their impact on the performance of the model on the HDFS and BGL datasets. The experiment still uses precision, recall, and F1 score as evaluation indicators. All experiments are conducted under the same training and testing conditions. The parameter settings of the isolation forest are consistent with the baseline experiment (the number of isolated trees is 100, the subsampling size is 256, the abnormal sample ratio is 0.001, and there is no replacement sampling).

[0141] Due to the strong correlation between some features (for example, in-degree and out-degree may be highly correlated), removing a feature alone may not fully reflect its importance. In this case, consider ablating by feature group. The features are divided into the following six groups according to category: node-level features, edge-level features, neighbor-level features, graph structure features, node similarity features, and centrality features. Node-level features include in-degree, out-degree, and node frequency; edge-level features include edge frequency and time deviation; neighbor-level features include the average in-degree of adjacent nodes and the average out-degree of adjacent nodes; graph structure features include connected components, graph diameter, and average path length; node similarity features include cosine similarity; and centrality features include degree centrality, closeness centrality, and betweenness centrality. During the experiment, each time a group of features is removed, the isolation forest model is retrained, and the performance is evaluated.

[0142] Table 4 Feature ablation experiment results

[0143]

[0144] Table 4 shows the results of the feature ablation experiments. The results show that on the HDFS dataset, removing centrality features (degree centrality, closeness centrality, and betweenness centrality) has the greatest impact on performance, with the F1 score dropping from 0.97 to 0.70, indicating that these features play a key role in capturing anomalous patterns. On the BGL dataset, removing graph structure features (connected components, graph diameter, and average path length) leads to a significant performance degradation, with the F1 score dropping from 0.87 to 0.68, demonstrating that global graph structure information is crucial for detecting complex anomalous patterns. In contrast, removing node similarity features and neighbor node features has a smaller impact on performance, indicating that these features contribute relatively limited contributions, but still help improve the overall performance of the model.

[0145] The foregoing shows and describes the principal features and advantages of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present invention is defined by the appended claims, not the foregoing description, and all variations that come within the meaning and range of equivalents of the claims are intended to be embraced therein.

[0146] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A system log anomaly detection method based on weighted directed graph and isolation forest, characterized in that: The following steps are involved: Parse the log sequence to obtain the log key; Represent the log keys by a weighted directed graph G and the adjacency matrix M; Extract multi-dimensional features based on the weighted directed graph G and the adjacency matrix M; Based on the extracted multi-dimensional features, the isolation forest algorithm is used to complete anomaly detection.

2. The system log anomaly detection method based on weighted directed graph and isolation forest according to claim 1 is characterized in that: The log key is represented by a weighted directed graph G, specifically: Assume that the set of log keys is L = {L1, L2, ..., L n }, where L n Representing a specific log key, the directed graph G = (V, E) consists of the following elements: Node set V: represents all elements in the log key set, that is, each log key can correspond to a node in the graph. The expression is as follows: L={L1,L2,…,L n }; Edge set E: represents the transfer relationship between log keys. If a log key L i Followed by another log key L j occurs, then there will be a line from node L in the graph i To node L j Directed edges, using a two-tuple (L i ,L j ) to represent this directed edge, the expression is as follows: E={(L i ,L j )|If event L i After the incident L j }.

3. The system log anomaly detection method based on weighted directed graph and isolation forest according to claim 1 is characterized in that: The log key is represented by the adjacency matrix M, specifically: Assuming there are n log keys, that is, n nodes, construct an n×n matrix M, which is expressed as follows: in, ω ij Indicates that from L i To L j The transfer weight is determined by the frequency of transfer.

4. The system log anomaly detection method based on weighted directed graph and isolation forest according to claim 1 is characterized in that: The extracted multi-dimensional features include: In-degree: all points to node v i The sum of the transfer weights of the edges is expressed as follows: Out-degree: v i The sum of the transfer weights of the edges pointing to other nodes is expressed as follows: Node frequency: the sum of the node's in-degree and out-degree, expressed as follows: Edge frequency: edge (v i ,v j ) is the transfer weight, which is expressed as follows: Time deviation: node v i and node v j The difference in timestamps is expressed as follows: Average in-degree of adjacent nodes: The average of the in-degree and out-degree of adjacent nodes. The expression is as follows: Connected components: the number of independent subgraphs in the graph, expressed as follows: Cosine similarity: used to measure the similarity between two nodes. The expression is as follows: Degree centrality: reflects the number of connections with other nodes, and the expression is as follows: C degree (in i )=d in (in i )+d out (in i ); Closeness centrality: measures the average distance from a node to all other nodes in the graph, expressed as follows: Among them, d(v i ,v j ) represents node v i and v j The shortest path distance between them; Betweenness centrality: measures the degree to which a node acts as an intermediary between other pairs of nodes, and is expressed as follows: Among them, σ(s,t) is the number of shortest paths between nodes s and t, σ(s,t|v i ) is through node v i The number of shortest paths; Diameter of a graph: The maximum distance between any two nodes in the graph, expressed as follows: Average path length: The average value of the shortest path length between any two nodes. The expression is as follows:

5. The system log anomaly detection method based on weighted directed graph and isolation forest according to claim 1 is characterized in that: The use of the isolation forest algorithm to complete anomaly detection is divided into two stages: (1) Training phase: Randomly sample from the training set and construct an isolation tree. Several isolated trees form an isolation forest. Randomly select n samples from the sampled sample points as the root nodes of the isolation tree; randomly select a feature dimension and select a partition value p between the maximum and minimum values of the feature dimension; divide the samples into the left subtree and the right subtree according to the p value, and recursively perform this operation on the left subtree and the right subtree until the preset height of the tree is reached or each leaf node has only one sample; (2) Anomaly detection stage: traverse each isolated tree in the isolation forest, obtain the expected depth of the test sample, and calculate the anomaly score. The anomaly score formula is as follows: Where s(x,n) represents the anomaly score of sample x, n represents the number of samples, h(x) represents the path length of sample x from the root node to the leaf node, E[h(x)] represents the expected value of h(x), and c(n) is the normalization constant. The formula is: H(k)≈ln(k)+γ Where H(k) is the harmonic number and γ represents the Euler constant; The value of s(x,n) is in the range of (0,1]. If s(x,n)≈1, it means that sample x is likely to be an outlier. If s(x,n)≈0.5, it means that sample x behaves similarly to a normal point. If s(x,n)<<0.5, it means that sample x is very likely to be a normal point.

Images