A system log detection method based on a self-attention mechanism graph network

Abstract

A system log detection method based on a self-attention mechanism graph network first collects log text data and system hardware data from the system's main log file to establish a standard multi-sequence data source. A spatiotemporal graph neural network model is then established, and inter-sequence graph relationships and intra-sequence temporal relationships are built based on the standard multi-sequence data, training optimal parameters. Based on the optimal parameter model, new standard multi-sequence data is used for model inference, and a criterion for judging the inference results is established. Based on the inference results, the region and time of anomaly occurrence are located, and the cause of the anomaly is further analyzed based on hardware data. This invention exhibits good stability and high detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer system security detection technology, specifically to a system log detection method based on a graph network with a self-attention mechanism. Background Technology

[0002] Today, data services typically operate in data centers comprised of thousands of servers, and the quality of these services depends on the reliability and security of the big data systems. Especially in enterprise data centers, it's crucial to detect anomalies proactively to prevent significant losses. As systems become more complex than ever before, they are exposing more vulnerabilities. Deploying fine-grained detection tools is essential, designed to identify potentially overlooked clues in the data. Logs record important events triggered during system runtime, forming a sequence based on timestamps. Effectively utilizing these log sequence data for event detection is of great significance.

[0003] System logs record system status and prohibited events to aid in debugging performance issues and failures, and to perform root cause analysis. This log data is universally available in almost all computer systems and is a valuable resource for understanding system status. Furthermore, because system logs record noteworthy events occurring in running processes, they are an excellent source of information for online monitoring and anomaly detection.

[0004] As systems grow in size and complexity, detecting anomalies through manual log checks becomes increasingly difficult. Over the years, numerous automated log-based methods have been proposed for anomaly detection. These works retrieve useful information from logs and employ data mining and machine learning techniques to analyze log data and detect the occurrence of system anomalies.

[0005] While existing log-based anomaly detection methods are effective, they are not robust enough in practice. To detect anomalies, almost all existing methods require building a detection model using known log events (i.e., templates for log messages) and log sequences (i.e., a series of log events recording a specific execution flow) extracted from the training data. Significant efforts have been made to improve log-based anomaly detection. Existing anomaly detection methods can be broadly categorized into two types based on their machine learning models: classic machine learning-based methods and deep learning-based methods. Although existing methods are successful in some cases, none are universal anomaly detection methods capable of detecting a wide variety of anomalous events.

[0006] Therefore, in today's complex system environment, there is an urgent need to provide an effective security detection method based on multi-source logs. Summary of the Invention

[0007] In view of the fact that current research focuses on specific attack scenarios and lacks research on detection based on multi-source log sequences, this invention proposes a system log detection method based on spatiotemporal graph neural networks with high reliability and detection accuracy.

[0008] The technical solution adopted by this invention to solve its technical problem is as follows:

[0009] A system log detection method based on a self-attention mechanism graph network includes the following steps:

[0010] S1. Collect log text data and system hardware data from the system's main log file to establish a standard multi-sequence data source;

[0011] S2. Establish a self-attention graph network model, and establish inter-sequence graph relationships and intra-sequence temporal relationships based on standard multi-sequence data sources;

[0012] S3. Perform intra-sequence feature learning on the log sequences respectively to establish a frequency domain time series model;

[0013] S4. By combining the self-attention graph network model and the frequency domain time series model, a self-attention spatiotemporal model is established to learn the relationship between multiple sequence logs from both spatial and temporal dimensions.

[0014] S5. Adjust the self-attention spatiotemporal model according to the discrimination criteria, and use multi-sequence training data as input to train the model and obtain the optimal parameter model.

[0015] S6. Based on the optimal parameter model, the processed multi-sequence logs to be detected are used as output. The inference results are used to locate the area and time of the abnormal event, and the cause of the abnormality is analyzed based on the hardware data.

[0016] Preferably, S1 specifically includes the following steps:

[0017] S1.1 Collect system log text sequences, extract log template numbers based on the log parser, and obtain log number sequences;

[0018] S1.2 Collect hardware sequence information, including system CPU, RAM, buffer, and disk information, at a frequency of 1 message / second, and write it to a file containing information on each indicator and the corresponding time.

[0019] S1.3 Establish a standard multi-sequence data source in seconds, match log text data and system hardware data according to the time dimension, and each log data corresponds to an underlying hardware sequence relationship to obtain standard multi-sequence data;

[0020] S1.4. Normalize the hardware data. The specific processing method is as follows:

[0021] (1)

[0022] in, For standard data results, and These are the maximum and minimum values ​​in the hardware data sample. This is hardware data.

[0023] Preferably, the specific steps for establishing the self-attention graph network model in S2 include:

[0024] S2.1. Obtain the weight matrix using the self-attention mechanism and establish a graph structure learning layer. The calculation method is as follows:

[0025] (2)

[0026] in, , and Represents the query, key, and embedding of all nodes, and the weight matrix. This is the result of the graph structure learning layer;

[0027] 2.2 Based on the weight matrix, the weight relationship between different sequences at the current time is obtained. The weight matrix is ​​repeated for different times to obtain the spatial relationship of all times in the sequence time.

[0028] Preferably, the specific steps for establishing the frequency domain time series model in S3 include:

[0029] S3.1 Construct intra-sequence features using spectral domain graph convolution, and convert the input to the spectral domain using graph Fourier transform, as shown in the following formula:

[0030] (3)

[0031] in, It is the identity matrix. yes diagonal matrix, It is the eigenvector matrix of the Laplacian matrix of the normalized graph. It is defined as The diagonalized matrix of eigenvalues;

[0032] S3.2. Based on establishing the graph relationships between sequences, the temporal relationships within sequences are learned. One-dimensional convolution is used to capture temporal features, and the calculation formula is as follows:

[0033] (4)

[0034] in, It is the input to a one-dimensional convolution. It is a convolution kernel. It is a non-linear sigmoid function. It is the element-wise product of matrices.

[0035] Preferably, the specific steps for establishing a self-attention spatiotemporal model in S4 include:

[0036] S4.1. Capture the features between and within the sequence by transforming each spectral time map convolutional unit in the spectral domain and frequency domain, and connect the two units in the form of residuals.

[0037] S4.2 In the self-attention spatiotemporal model, all module parameters are trained uniformly using data, and all parameters are updated in a data-driven manner. The target loss function is set as follows:

[0038] (5)

[0039] in, It is standard input. The predicted value of the model at time t;

[0040] S4.3 sets the training strategy, dividing the collected data into training data, validation data, and test data in a 7:2:1 ratio, and using the validation data to evaluate whether it is the optimal parameter model.

[0041] Preferably, the criteria for judging the reasoning result in S5 specifically include:

[0042] The self-attention spatiotemporal model evaluates the effectiveness of this anomaly detection method using accuracy, recovery rate, and F1 score, labeling the results as TP, TN, FP, and FN. TP represents true positives in the HDFS dataset, indicating that normal samples are predicted as normal by the model; TN represents true negatives in the HDFS dataset, indicating that anomalous samples are predicted as anomalous by the model; FP represents false positives in the HDFS dataset, indicating that anomalous samples are predicted as normal by the model; and FN represents false negatives in the HDFS dataset, indicating that normal samples are predicted as anomalous by the model.

[0043] The accuracy index is calculated as follows:

[0044] (6)

[0045] The recovery rate is calculated as follows:

[0046] (7)

[0047] The F1 score is calculated as follows:

[0048] (8).

[0049] Preferably, in S1.3, matching log text data and system hardware data according to the time dimension specifically involves matching an N*1 log number sequence with an N*m hardware sequence to form a standard multi-sequence data block of N*(1+m), where m is the number of indicators collected by the system hardware.

[0050] Compared with the prior art, this application has at least the following obvious advantages and effects:

[0051] This invention proposes a detection method based on multi-source log sequences. Considering other log sequences related to the system, it introduces a graph neural network method to construct relationships between multiple log sequences, thereby performing log detection. A workflow sequence based on time-based data matching and template indexing is constructed. Unlike previous log sequence feature detection methods, this method improves the model's sensitivity to anomalous sequences by matching logarithmic hardware data. Through multiple data sequences, we can identify a lower proportion of anomalous tasks. This invention exhibits good stability, higher accuracy, and greater sensitivity to small-scale anomalous events. Attached Figure Description

[0052] Figure 1 This is an overall schematic diagram of the method of the present invention.

[0053] Figure 2 This is a schematic diagram of multi-sequence data matching according to the present invention.

[0054] Figure 3 This is a schematic diagram of log data parsing in this invention.

[0055] Figure 4 This is a schematic diagram of the temporal prediction algorithm based on spectral domain graph convolution of the present invention. Detailed Implementation

[0056] To further understand the content of this invention, the invention will be described in detail with reference to the embodiments. The following embodiments are used to illustrate the invention, but are not intended to limit the scope of the invention.

[0057] Reference Figures 1 to 4 This embodiment relates to a system log detection method based on a graph network with a self-attention mechanism, the steps of which are as follows:

[0058] S1. Collect log text data and system hardware data from the system's main log file to establish a standard multi-sequence data source. The process is as follows:

[0059] S1.1 Collect system log text sequences, extract log template numbers based on the log parser, and obtain log number sequences;

[0060] S1.2 Collect hardware sequence information, mainly including system CPU, RAM, buffer, and disk information. Collect at a frequency of 1 message / second and write to a file. The content includes information on each indicator and the corresponding time.

[0061] S1.3. Due to the variable frequency of log generation, with a minimum unit of seconds, to establish a standard multi-sequence data source, it is necessary to match the log text data and system hardware data according to the time dimension. That is, to match the N*1 log number sequence with the N*m hardware sequence to form a standard multi-sequence data block of N*(1+m), where m is the number of indicators collected by the system hardware, which is m=4 here. Each log data corresponds to an underlying hardware sequence relationship, thus obtaining the standard multi-sequence data.

[0062] S1.4 To eliminate the influence of dimensions on the results, the hardware data also needs to be normalized. The specific processing method is as follows:

[0063]

[0064] in, For standard data results, and These are the maximum and minimum values ​​in the hardware data sample. This is hardware data.

[0065] S2. Establish a self-attention graph network model, and build inter-sequence graph relationships and intra-sequence temporal relationships based on standard multi-sequence data. The process of building a self-attention graph network model is as follows:

[0066] S2.1 A primary objective of the model is to learn the relationships between logs and m hardware components in the form of a graph structure. To achieve this, we construct the graph in a data-driven manner, where nodes represent levels and edges represent dependencies between nodes. Therefore, the first part of the model needs to be established: the graph structure learning layer. Here, a self-attention mechanism is used to obtain the weight matrix. The calculation method is as follows:

[0067]

[0068] in, , and Represents the query, key, and embedding of all nodes. Matrix This is the result of the graph structure learning layer.

[0069] S2.2, Based on weight matrix This allows us to obtain the weight relationships between different sequences at the current time. By repeating this process at different times, we can obtain the spatial relationships of all times in the sequence.

[0070] S3. Perform intra-sequence feature learning on the log sequences respectively to establish a frequency domain time series model. The process of establishing the frequency domain time series model is as follows:

[0071] S3.1 To better utilize the obvious periodicity in the data, this method uses spectral domain graph convolution to construct intra-sequence features. First, the input is converted to the spectral domain using graph Fourier transform, as shown in the following formula:

[0072]

[0073] in, It is the identity matrix. yes A diagonal matrix. It is the eigenvector matrix of the Laplacian matrix of the normalized graph. It is defined as The diagonalized matrix of eigenvalues.

[0074] S3.2. Building upon the inter-sequence graph relationships, the model also learns the temporal relationships within sequences. Here, the model uses one-dimensional convolution to capture temporal features, calculated using the following formula:

[0075]

[0076] in, It is the input to a one-dimensional convolution. It is the convolution kernel, and the kernel size here is 2. It is a non-linear sigmoid function. It is the element-wise product of matrices.

[0077] S4. Combining the graph network model and the frequency domain time series model, a self-attention spatiotemporal model is formed. This model learns the relationships between multiple sequence logs from both spatial and temporal dimensions. The process of establishing the self-attention spatiotemporal model is as follows:

[0078] S4.1 Each spectral time-map convolutional unit captures features between and within sequences through spectral and frequency domain transformations, and connects the two units in the form of residuals.

[0079] S4.2. All module parameters of the model are trained using unified data, and all parameters are updated in a data-driven manner. The target loss function is set as follows:

[0080]

[0081] in, It is standard input. This represents the model's predicted value at time t.

[0082] S4.3. Set a training strategy, dividing the collected data into training data, validation data, and test data in a 7:2:1 ratio. The validation data is used to evaluate whether the model has the optimal parameters.

[0083] S5. Train the optimal parameter model. Use multi-sequence training data as input to train the model. Adjust the model based on the discrimination criteria. The specific discrimination criteria for inference results include:

[0084] The self-attention spatiotemporal model evaluates the effectiveness of this anomaly detection method by accuracy, recovery rate, and F1 score, and labels the results as TP, TN, FP, and FN.

[0085] Where TP is a true positive in the HDFS dataset, indicating that a normal sample is predicted as normal by the model; TN is a true negative in the HDFS dataset, indicating that an abnormal sample is predicted as abnormal by the model; FP is a false positive in the HDFS dataset, indicating that an abnormal sample is predicted as normal by the model; and FN is a false negative in the HDFS dataset, indicating that a normal sample is predicted as abnormal by the model.

[0086] The precision metric is calculated as follows:

[0087]

[0088] The recovery rate is calculated as follows:

[0089]

[0090] The F1 score is calculated as follows:

[0091] .

[0092] S6. Based on the optimal parameter model, the processed multi-sequence logs to be detected are used as output. The inference results are used to locate the area and time of the abnormal event, and the cause of the abnormality is further analyzed based on hardware data. Although the specific embodiments of the present invention have been described in detail above, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention. Modifications or variations that do not involve creative effort are still within the protection scope of the present invention.

Claims

1. A system log detection method based on a graph network with a self-attention mechanism, characterized in that, It includes the following steps: S1. Collect log text data and system hardware data from the system's main log file to establish a standard multi-sequence data source; S2. Establish a self-attention graph network model, and establish inter-sequence graph relationships and intra-sequence temporal relationships based on standard multi-sequence data sources; S3. Perform intra-sequence feature learning on the log sequences respectively to establish a frequency domain time series model; S4. By combining the self-attention graph network model and the frequency domain time series model, a self-attention spatiotemporal model is established to learn the relationship between multiple sequence logs from both spatial and temporal dimensions. S5. Adjust the self-attention spatiotemporal model according to the discrimination criteria, and use multi-sequence training data as input to train the model and obtain the optimal parameter model. S6. Based on the optimal parameter model, the processed multi-sequence logs to be detected are used as output. The inference results are used to locate the area and time of the abnormal event, and the cause of the abnormality is analyzed based on the hardware data.

2. The system log detection method based on self-attention mechanism for graph networks according to claim 1, characterized in that, S1 specifically includes the following steps: S1.1 Collect system log text sequences, extract log template numbers based on the log parser, and obtain log number sequences; S1.2 Collect hardware sequence information, including system CPU, RAM, buffer, and disk information, at a frequency of 1 message / second, and write it to a file containing information on each indicator and the corresponding time. S1.3 Establish a standard multi-sequence data source in seconds, match log text data and system hardware data according to the time dimension, and each log data corresponds to an underlying hardware sequence relationship to obtain standard multi-sequence data; S1.

4. Normalize the hardware data. The specific processing method is as follows: ； in, This is a sample of actual hardware data. For standard data results, and These are the maximum and minimum values ​​in the hardware data sample. This is hardware data.

3. The system log detection method based on self-attention mechanism for graph networks according to claim 1, characterized in that, The specific steps for building a self-attention graph network model in S2 include: S2.

1. Obtain the weight matrix using the self-attention mechanism and establish a graph structure learning layer. The calculation method is as follows: ； in, 、 and These represent the query, key, and embedding of all nodes, respectively. The dimension of the key, the weight matrix This is the result of the graph structure learning layer. The number of data points collected; S2.2 Based on the weight matrix, obtain the weight relationship between different sequences at the current time. Repeat the weight matrix for different times to obtain the spatial relationship of all times in the sequence time.

4. The system log detection method based on self-attention mechanism for graph networks according to claim 3, characterized in that, The specific steps for establishing a frequency domain time series model in S3 include: S3.1 Construct intra-sequence features using spectral domain graph convolution, and convert the input to the spectral domain using graph Fourier transform, as shown in the following formula: ； in, It is the identity matrix; yes diagonal matrix, It is a weight matrix The element at position (i,j); It is a nearest neighbor matrix; It is the eigenvector matrix of the Laplacian matrix of the normalized graph; It is the characteristic diagonal matrix; It is defined as The diagonalized matrix of eigenvalues, yes Location feature values; S3.

2. Based on establishing graph relationships between sequences, learn the temporal relationships within sequences and use one-dimensional convolution to capture temporal features. H The calculation formula is as follows: ； in, It is the input to a one-dimensional convolution. It is a convolution kernel. It is a non-linear sigmoid function. It is the element-wise product of matrices.

5. The system log detection method for graph networks based on self-attention mechanism according to claim 1, characterized in that, The specific steps for establishing a self-attention spatiotemporal model in S4 include: S4.

1. Each spectral time-map convolutional unit captures features between and within sequences through spectral domain transformation and frequency domain transformation. The input and output features of the current spectral time-map convolutional unit are added element by element and used as the input of the next spectral time-map convolutional unit. S4.2 In the self-attention spatiotemporal model, all module parameters are trained uniformly using data, and all parameters are updated in a data-driven manner. The target loss function is set as follows: ； in, It is standard input. for The predicted value of the time-matter model, for The actual value at time, The termination time; S4.3 sets the training strategy, dividing the collected data into training data, validation data, and test data in a 7:2:1 ratio, and using the validation data to evaluate whether it is the optimal parameter model.

6. The system log detection method for graph networks based on self-attention mechanism according to claim 1, characterized in that, The specific criteria for judging inference results in S5 include: The effectiveness of the self-attention spatiotemporal model's prediction results was evaluated using accuracy, retrieval rate, and F1 score, and the results were labeled as TP, TN, FP, and FN. TP represents a true positive instance in the HDFS dataset, indicating that a normal sample was predicted as normal; TN represents a true negative instance in the HDFS dataset, indicating that an abnormal sample was predicted as abnormal; FP represents a false positive instance in the HDFS dataset, indicating that an abnormal sample was predicted as normal; and FN represents a false negative instance in the HDFS dataset, indicating that a normal sample was predicted as abnormal. accuracy The indicator is calculated as follows: ； Recovery rate The indicator is calculated as follows: ； The F1 score is calculated as follows: 。 7. The system log detection method for graph networks based on self-attention mechanism according to claim 2, characterized in that, In S1.3, matching log text data and system hardware data according to the time dimension specifically involves matching an N×1 log number sequence with an N×m hardware sequence to form a standard multi-sequence data block of N×(1+m), where m is the number of indicators collected by the system hardware and N is the number of data points collected.

Images