Server anomaly detection method based on time sequence log

Abstract

The invention discloses a server anomaly detection method based on a time sequence log. The method comprises the following steps: S1, collecting a server performance log; s2, data preprocessing; s3, an anomaly detection model is constructed, BiLSTM is introduced into an encoder layer based on an auto-encoder, and the encoder comprises an input layer, a forward LSTM and a backward LSTM; introducing an attention mechanism into a decoder to construct a one-way LSTM; s4, performing model training and reconstruction error calculation, inputting the training set into an AE-BiLSTM-Attn model for forward propagation, and optimizing the performance of the model by calculating a loss function and performing back propagation to update parameters W and b of the model; and S5, performing model evaluation, and calculating an error MSE between the reconstruction output and the original input of the AE-BiLSTM-Attn model on the test set for measuring the reconstruction precision of the model. According to the method, the data dimension reduction and feature extraction capabilities of the auto-encoder, the long-term dependency relationship capture capability of the bidirectional LSTM network on the time series data and the attention mechanism are combined, so that the accuracy and efficiency of the features are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of text analysis and anomaly detection, and in particular relates to a server anomaly detection method based on time series logs. Background Art

[0002] In the daily operation of information systems, servers generate a large amount of status indicator data, such as CPU usage, memory occupancy, disk I / O rate, network bandwidth, etc. These data not only reflect the current operating status of the server, but also contain the future change trend of the system. However, due to the interference of various internal and external factors, such as hardware failures, software vulnerabilities, network attacks, etc., the server status indicator data may fluctuate abnormally, thus affecting the normal operation of the information system.

[0003] Traditional anomaly detection methods, such as threshold judgment and statistical analysis, often rely on manually set rules and thresholds, and are difficult to adapt to the complex and changing server operating environment. In addition, these methods have limitations when processing time series data and cannot effectively capture the time dependency and long-term trends between data. Therefore, there is an urgent need for an intelligent method that can automatically and accurately identify anomalies in server status indicator data.

[0004] In information systems, for anomaly detection of server status indicator data, the prior art uses an anomaly detection method based on a statistical model, which mainly uses statistical models to capture trends and periodic changes in the data, thereby predicting the normal range of the server status. A threshold is set, and when the actual data deviates from the predicted value by more than the threshold, it is considered that an anomaly has occurred. The system will send an abnormal alarm message to the administrator based on the preset alarm mechanism so that timely measures can be taken to deal with it. However, there are two problems: first, the statistical model has poor adaptability to the complex and changeable server operating environment, and it is difficult to accurately capture the nonlinear relationship and long-term dependency in the data; second, the setting of the threshold depends on manual experience and lacks an automatic adjustment mechanism, which may lead to false positives or false negatives. Summary of the invention

[0005] The purpose of the present invention is to solve the problem that the statistical model has poor adaptability to the complex and changeable server operating environment, and it is difficult to accurately capture the nonlinear relationship and long-term dependence in the data and the setting of thresholds.

[0006] To achieve the above object, the present invention provides a server anomaly detection method based on time series logs, comprising the following steps:

[0007] S1. Obtain server performance log data, that is, obtain server performance log data in real time, including CPU usage, memory usage, and disk I / O key indicator data;

[0008] S2, preprocessing the server performance log data in step S1; and 70% of the server performance log data is a training set and 30% is a test set;

[0009] S3. Taking the autoencoder module as the basic framework, the BiLSTM model is introduced into the encoder of the autoencoder module to construct a BiLSTM encoder, and the attention mechanism is introduced into the decoder to construct a unidirectional LSTM decoder, thereby constructing an anomaly detection model;

[0010] S4, input the training set in step S2 into the anomaly detection model in step S3 for forward propagation, optimize the performance of the model by calculating the MSE loss function and updating the parameters W and b of the model through back propagation; determine whether the server performance log data is abnormal data based on the relationship between the MSE loss function and the set threshold;

[0011] S5. Model evaluation: The quality of the model is evaluated through key indicators such as Precision, Recall, and F1-score, as well as ROC curve and AUC value.

[0012] Further, the specific method of preprocessing in step S2 is:

[0013] S201, filtering the data in step S1 according to mainstream server performance KPI;

[0014] S202, constructing time series data by arranging in chronological order according to the @timestamp field;

[0015] S203. Delete null values, negative values, and abnormal numbers that have not changed to ensure data quality and consistency;

[0016] S204, remove the data unit and convert it into float numerical data;

[0017] S205: Normalize the data using the StandardScaler library to make the mean of each feature 0 and the standard deviation 1.

[0018] Further, the specific implementation of step S3 is:

[0019] S301, construct an autoencoder module, the autoencoder consists of an encoder layer φ e :RF→Rf, hidden layer, decoder layer φ d :RF→Rf, where the hidden layer stores the low-dimensional feature Z;

[0020] The encoder maps the sample x of the feature space of high-dimensional F to the potential low-dimensional feature Z of low-dimensional f by compression, θ e Represents the weight parameter corresponding to the encoder, the specific formula is Z = φe (x;θ e );

[0021] Decoder φ d Reconstruct the latent low-dimensional feature Z and map it back to a sample x′, θ, which is similar to the original sample x and has the same feature dimension d Represents the weight parameter corresponding to the decoder, the specific formula is x′=φ d (Z;θ d );

[0022] S302, construct a BiLSTM encoder, and generate an internal representation containing the bidirectional dependency relationship in the input sequence through the BiLSTM encoder, that is, the hidden state h t ;

[0023] S303. Introduce the attention mechanism to construct a unidirectional LSTM decoder, and the LSTM decoder generates reconstructed data based on the unidirectional LSTM.

[0024] Furthermore, the specific implementation process in step S302 is as follows:

[0025] The BiLSTM encoder consists of three layers: input layer, forward LSTM and backward LSTM. The construction process of the BiLSTM encoder is as follows:

[0026]

[0027] The input layer accepts the original input sequence X = {x 1 ,x 2 ,x 3 ......x T}, where x t is the input vector at the tth time step, T is the maximum sequence length; the forward LSTM layer processes the input from the beginning to the end of the sequence, and obtains the hidden state of the forward LSTM The backward LSTM processes the input from the end of the sequence to the beginning, obtaining the hidden state of the backward LSTM The output h of the bidirectional LSTM t is the final hidden state of the decoder, representing the information "remembered" after processing the input sequence up to the current time step t, including time dependency, pattern, context, etc., where ";" represents the concatenation of two vectors, t = 1, 2, 3...T;

[0028] Output the latent low-dimensional feature representation Z and select the last hidden state h of the encoder T As a potential low-dimensional feature representation Z = h T .

[0029] Furthermore, the specific implementation process in step S303 is as follows:

[0030] (1) Calculate the decoder hidden state s t' : The LSTM layer will calculate the hidden state s of the decoder t' , for each hidden state s at time step t′ t' , according to the hidden state s of the previous time step t'-1 And the potential low-dimensional feature representation Z output by the encoder is calculated;

[0031] s t' =LSTM(Z,s t'-1 )

[0032] (2) Construct an attention mechanism: According to the hidden state s of the decoder at each time step t′ t' The final hidden state h of the encoder t The correlation between them is used to generate a context vector c′ t ; The details are as follows:

[0033] Calculate the attention score e t,t' =v T tanh(W 1 h t +W 2 s t' ), represents the attention of the decoder at time step t′ to the encoder at time step t; v,W 1 ,W 2 is the parameter of the attention mechanism;

[0034] Calculating attention weights The attention score is converted into a probability distribution through the softmax function so that all its weights and values ​​are 1;

[0035] Calculate the context vector Denotes the decoder at time step t′ for all hidden states h of the encoder t The weighted sum of

[0036] (3) Decoder output: Convert the hidden state of the encoder output back to an output sequence of the same or similar length as the input sequence; convert the context vector c t′ and the decoder hidden state s t' Combined, the output reconstructed data is generated through the fully connected layer W is the weight matrix and b is the bias term, which are randomly generated at the beginning of the model and the optimal parameter value is obtained through continuous training. The calculation formula for the reconstructed data is as follows:

[0037]

[0038] Furthermore, the calculation formula of the MSE loss function in step S4 is:

[0039]

[0040] The threshold is mainly based on statistical analysis of training data, and the calculation formula is as follows:

[0041] Threshold=μ+kσ

[0042] Where μ is the mean reconstruction error on the training set, σ is the standard deviation of the reconstruction error on the training set, and k is a constant representing the tolerable error range; when the MSE loss function exceeds the set threshold, it is judged as abnormal data; otherwise, it is judged as normal data.

[0043] Furthermore, the calculation methods of the key indicators of Precision, Recall and F1-score, as well as the ROC curve and AUC value in step S5 are as follows:

[0044] (1) Precision: The ratio of all samples correctly judged as abnormal to the samples judged as abnormal by the model, that is, the ratio of TP to (TP + FP). The calculation formula is:

[0045]

[0046] TP represents the normal data points that the model correctly identifies as abnormal; FP represents the normal data points that the model incorrectly identifies as abnormal;

[0047] (2) Recall: The ratio of all samples correctly judged as abnormal to the real abnormal samples, that is, the ratio of TP to (TP + FN), calculated as:

[0048]

[0049] Among them, FN represents that the model mistakenly judges abnormal data points as normal;

[0050] (3) F1-score: The harmonic mean of Precision and Recall, calculated as:

[0051]

[0052] ROC curve and AUC value: The ROC curve is a curve drawn with the model's true positive rate TPR, that is, the recall rate, as the vertical coordinate and the false positive rate FPR, that is, the ratio of all normal samples that are mistakenly judged as abnormal to the real normal samples as the horizontal coordinate; the AUC value is the area under the ROC curve, which is used to evaluate the comprehensive performance of the model; the AUC value is calculated according to the trapezoidal rule, that is, by dividing the area under the ROC curve into multiple trapezoids, and then calculating the areas of these trapezoids and summing them up.

[0053] Beneficial effects:

[0054] 1. Design of AE-BiLSTM-Attn model: To solve the problem of poor adaptability in complex environments, the present invention is based on an autoencoder and introduces a bidirectional LSTM model and an attention mechanism. This model combines the data dimension reduction and feature extraction capabilities of the autoencoder, the bidirectional LSTM network's ability to capture long-term dependencies of time series data, and the attention mechanism to improve the accuracy and efficiency of features. By encoding and decoding the server status indicator data through the autoencoder, a low-dimensional representation and reconstruction error of the data can be obtained. The bidirectional LSTM network can capture the time dependencies in the data, allowing the model to more accurately identify abnormal patterns in the data. This technical feature significantly improves the adaptability of the model in complex and changeable server operating environments.

[0055] 2. Automatic determination of abnormal threshold by reconstruction error: To address the problem that threshold setting depends on manual experience, the present invention uses the reconstruction error of the autoencoder LSTM model to automatically determine the abnormal threshold. During the model training process, a reasonable threshold can be set by calculating the reconstruction error distribution of normal data, so that when the reconstruction error of new data exceeds the threshold, it is considered that an abnormality has occurred. This technical feature realizes the intelligent setting and dynamic adjustment of the threshold, avoiding the subjectivity and uncertainty caused by manual intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of the anomaly detection method of time series log provided by the present invention;

[0057] Figure 2 It is a framework diagram of the AE-BiLSTM-Attn model provided by the present invention;

[0058] Figure 3 It is a framework diagram of the self-encoding module provided by the present invention;

[0059] Figure 4 This is a framework diagram of the autoencoder model that introduces LSTM and attention mechanism provided by the present invention;

[0060] Figure 5 It is a performance parameter comparison chart of four algorithms of the training set provided by the present invention;

[0061] Figure 6 It is a performance parameter comparison chart of four algorithms of the test set provided by the present invention;

[0062] Figure 7 This is the ROC comparative evaluation of the model provided by the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0064] The application principle of the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0065] Embodiment 1:

[0066] like Figure 1 As shown, a server anomaly detection method based on time series logs includes the following steps:

[0067] S1. Obtain server performance log data

[0068] The main data of the present invention comes from the performance log obtained by the LeXiang intelligent operation and maintenance management platform for the status monitoring of the virtual integrated machine of the information system, including key indicator data such as CPU usage, memory usage, disk I / O, etc. The performance logs of all servers in the information system are collected in real time through the deployment script for subsequent abnormal detection of server performance.

[0069] S2. Data preprocessing

[0070] First, the collected raw performance indicator data is filtered according to the mainstream server performance KPI. The specific features are shown in Table 1. According to the @timestamp field, the data is arranged in chronological order to construct time series data; the pandas library's dropna and other methods are used to delete the abnormal data containing null values, negative values, and unchanged data for data cleaning and denoising to ensure the quality and consistency of the data; the data units (such as IO / s, ms, MB, etc.) are removed and converted to float numerical data to facilitate model training; finally, the time series data is normalized and the StandardScaler library is used to standardize the data so that the mean of each feature is 0 and the standard deviation is 1, eliminating the dimensional differences between different performance indicators and improving the training efficiency and accuracy of the model. After processing, 70% of the data is used for the training set and 30% for the test set.

[0071] Table 1 Index data

[0072]

[0073]

[0074] S3. Build anomaly detection model

[0075] like Figure 2As shown in the figure, this model introduces the BiLSTM layer into the encoder and Attn (attention mechanism) into the decoder structure to form the AE-BiLSTM-Attn anomaly detection model. This model can not only capture the temporal characteristics of time series data and learn its potential representation, but also utilize the unsupervised learning characteristics of the autoencoder to reduce the workload of data set label extraction. The AE-BiLSTM-Attn network structure is an unsupervised learning neural network model. It performs anomaly detection by learning the low-dimensional representation of the input data and trying to reconstruct the original input data. The introduction of the attention mechanism enables the decoder to focus on different parts of the input sequence, thereby improving the detection accuracy of the model. The anomaly detection model mainly includes the input layer, the BiLSTM encoding layer, the Attn-LSTM decoding layer, and the output layer. The details are as follows:

[0076] S301, construct an autoencoder module. An autoencoder is a fully connected neural network model whose input and output data have the same dimension. The present invention uses an autoencoder as the basic architecture of the present method, which consists of an encoder layer φ e :RF→Rf and decoder layer φ d :RF→Rf consists of two parts, and there is a hidden layer in the middle to store the low-dimensional feature Z. The architecture of the autoencoder module is shown in the figure Figure 3 shown.

[0077] For an input sample x, the encoder φ e Aims to map the sample x of the feature space of high-dimensional F to the potential low-dimensional feature Z of low-dimensional f by compression, θ e Represents the weight parameter corresponding to the encoder, the specific formula is Z = φ e (x;θ e ).

[0078] With encoder φ e In contrast, the decoder φ d Responsible for reconstructing this potential low-dimensional feature Z and mapping it back to a sample x′ similar to the original sample x and with the same feature dimension, θ d Represents the weight parameter corresponding to the decoder, the specific formula is x′=φ d (Z;θ d ).

[0079] As shown in Figure 4, in the present application, a BiLSTM model is introduced into the encoder of the self-encoding module in step S301 to construct a BiLSTM encoder, and an attention mechanism is introduced into the decoder to construct a unidirectional LSTM decoder.

[0080] S302, introduce BiLSTM (bidirectional long short-term memory) to construct an encoder. The present invention introduces bidirectional LSTM in the encoder to better capture the context information in the sequence data. Bidirectional LSTM is an advanced variant of recurrent neural network. Compared with traditional unidirectional RNN or LSTM, it can use information from both the past and future directions at each time step, thereby improving the modeling ability of sequence data.

[0081] The main function of the BiLSTM encoder is to generate an internal representation that contains the bidirectional dependencies in the input sequence, namely the hidden state h t The BiLSTM encoder consists of three layers: input layer, forward LSTM and backward LSTM. The construction process of the BiLSTM encoder is as follows:

[0082]

[0083] The input layer accepts the original input sequence, X = {x 1 ,x 2 ,x 3 ......x T} represents the original input sequence, where x t is the original input vector at the tth time step, and T is the maximum sequence length; the forward LSTM layer processes the input from the beginning to the end of the sequence, and obtains the hidden state of the forward LSTM The backward LSTM processes the input from the end of the sequence to the beginning, obtaining the hidden state of the backward LSTM The output h of the bidirectional LSTM t is the final hidden state of the encoder, representing the information "remembered" after processing the input sequence up to the current time step t, including time dependency, pattern, context, etc., where ";" represents the concatenation of two vectors, where t = 1, 2, 3...T.

[0084] Output the latent low-dimensional feature Z (a fixed-length vector), select the last hidden state h of the encoder T (T is the length of the input sequence) as a potential low-dimensional feature representation Z = h T .

[0085] S303, introduce the attention mechanism to build a unidirectional LSTM decoder. The decoder uses a unidirectional LSTM to generate reconstructed data, and introduces an attention mechanism based on LSTM. This mechanism allows the decoder to dynamically focus on different parts of the input sequence when generating an output sequence, thereby enhancing the performance of the model. The specific steps are as follows:

[0086] (1) Calculate the decoder hidden state s t' : The LSTM layer will calculate the hidden state s of the decodert' , for each hidden state s at time step t′ t' , according to the hidden state s of the previous time step t'-1 And the latent low-dimensional feature representation Z of the encoder output is calculated.

[0087] s t' =LSTM(Z,s t'-1 )

[0088] (2) Constructing the attention mechanism: The core is to calculate the hidden state s of the current decoder at each time step t′ t' The final hidden state h of the encoder t The correlation between them is used to generate a context vector c′ t In this method, the attention mechanism formula is used for calculation, and the specific steps are as follows.

[0089] Calculate the attention score e t,t' =v T tanh(W 1 h t +W 2 s t' ), represents the attention of the decoder at time step t′ to the encoder at time step t. v,W 1 ,W 2 are the parameters of the attention mechanism.

[0090] Calculating attention weights The attention score is converted into a probability distribution through the softmax function so that all its weights and values ​​are 1.

[0091] Calculate the context vector c t′ Denotes the decoder at time step t′ for all hidden states h of the encoder t The weighted sum of .

[0092] (3) Decoder output: The main function of the decoder is to convert the hidden state output by the encoder back to an output sequence of the same or similar length as the input sequence. In this method, the context vector c t′ and the decoder hidden state s t' Combined, the output reconstructed data is generated through the fully connected layer W is the weight matrix and b is the bias term, which are randomly generated at the beginning of the model and the optimal parameter value is obtained through continuous training. is the data reconstructed by the decoder. Its reconstructed data is calculated as follows:

[0093]

[0094] S4. Model training and reconstruction error calculation

[0095] During the model training phase, the training set needs to be input into the AE-BiLSTM-Attn model for forward propagation. The model performance is optimized by calculating the loss function and performing backpropagation to update the model parameters W and b. The process of forward propagation, loss calculation, and backpropagation is repeated until the preset number of iterations is reached. During the training process, a dropout strategy is used to randomly set the output of some neurons to 0 with a certain probability, forcing the network to not rely on a specific combination of neurons for prediction, thereby reducing the co-adaptability between neurons and preventing the model from overfitting.

[0096] Reconstruction error calculation. During model training, the loss function is used to measure the size of the model reconstruction error. For the anomaly detection task of time series data, commonly used loss functions include mean square error (MSE). The MSE loss function calculates the average of the sum of the squares of the difference between the reconstructed data and the original data to evaluate the reconstruction performance of the model. The calculation formula of the MSE loss function is as follows:

[0097]

[0098] By comparing the MSE loss function with the set threshold, it is determined whether the data is abnormal.

[0099] The choice of threshold is usually based on the performance of the model on normal data. During the training process, the model is trained only on normal data, hoping that the model can reconstruct normal data well. In this method, based on the statistical analysis of the training data, the threshold calculation formula is as follows:

[0100] Threshold=μ+kσ

[0101] Where μ is the mean value of the reconstruction error on the training set, σ is the standard deviation of the reconstruction error on the training set, and k is a constant (such as 2 or 3) indicating the tolerance range of the error. When the MSE loss function exceeds the set threshold, it is judged as abnormal data. Otherwise, it is judged as normal data.

[0102] Optimizer selection. During the model training process, it is necessary to select an appropriate optimizer to adjust the model parameters to minimize the loss function. Commonly used optimizers include Adam, SGD, etc. The Adam optimizer combines the advantages of the momentum method and the RMSprop optimizer, and has a faster convergence speed and better robustness.

[0103] S5. Model evaluation

[0104] The error MSE between the reconstructed output and the original input of the AE-BiLSTM-Attn model on the test set is calculated to measure the reconstruction accuracy of the model. It is compared with the statistical-based method (Z-score method) and the traditional machine learning algorithm (a class of support vector machine algorithms OC-SVM and Isolation Forest), and the key indicators such as Precision, Recall, and F1-score of each method are calculated respectively. The advantages and disadvantages of each model are also comprehensively evaluated by drawing the ROC (receiver operating characteristic) curve and calculating the AUC (area under the curve).

[0105] (1) Precision: The ratio of all samples correctly judged as abnormal to the samples judged as abnormal by the model. It reflects the accuracy of the model in judging abnormal samples and avoids misjudging normal samples as abnormal. It is the ratio of TP to (TP + FP). The calculation formula is:

[0106]

[0107] TP represents the normal data points that the model correctly identifies as abnormal. In other words, the data points that the model predicts to be abnormal are actually abnormal. FP represents the normal data points that the model mistakenly identifies as abnormal. That is, the points that the model predicts to be abnormal are actually normal.

[0108] (2) Recall: The ratio of all samples correctly judged as abnormal to the real abnormal samples. It measures the ability of the model to identify the real abnormal samples. It is hoped that the model can detect as many abnormal samples as possible, that is, the ratio of TP to (TP + FN). The calculation formula is:

[0109]

[0110] FN means that the model mistakenly judges an abnormal data point as normal, that is, a point predicted by the model as normal is actually abnormal.

[0111] (3) F1-score: The harmonic mean of Precision and Recall. F1-Score can help comprehensively consider the precision and recall of the model and is a more comprehensive evaluation indicator. When both the precision and recall are high, the F1-Score will also be high, indicating that the model performance is good. The calculation formula is:

[0112]

[0113] (5) ROC curve and AUC value: The ROC curve is a curve drawn with the model's true positive rate (TPR, i.e., recall rate) as the ordinate and the false positive rate (FPR, i.e., the ratio of all normal samples that are incorrectly judged as abnormal to the true normal samples) as the abscissa. The AUC value is the area under the ROC curve and is used to evaluate the comprehensive performance of the model. The ROC curve can intuitively show the performance of the model at various thresholds, and the AUC value can comprehensively evaluate the performance of the model. The larger the AUC value, the better the model performance.

[0114] The false positive rate FPR calculation formula is as follows:

[0115]

[0116] The calculation formula for the true positive rate TPR is as follows:

[0117]

[0118] FN represents the number of positive samples that are mistakenly judged as negative examples.

[0119] In this method, the AUC value is calculated according to the trapezoidal rule, that is, the area under the ROC curve is divided into multiple trapezoids, and then the areas of these trapezoids are calculated and summed. Assume there are two points (FPR 1 ,TPR 1 )(FPR 2 ,TPR 2 ), the area of ​​the trapezoid can be calculated as:

[0120]

[0121] like Figure 5-Figure 7 As shown in the figure, the AUC values ​​of encoding LSTM, z-score, OC-SVM, and Isolation Forest are 0.98, 0.8, 0.92, and 0.95, respectively. The AUC values ​​of all methods are greater than 0.7, indicating that the prediction accuracy is high. Among them, the LSTM model shows the highest prediction accuracy, followed by the Isolation Forest model.

[0122] Embodiment 2:

[0123] In the actual application of the remote control system of groundwater circulation wells, the anomaly detection system of the AE-BiLSTM-Attn model proposed in this invention is deployed to verify its effectiveness and practicality in actual scenarios. The system involves the remote control of groundwater circulation wells, has a large number of servers and data resources, and has an urgent need for real-time monitoring and anomaly detection of server status indicator data.

[0124] In order to achieve this goal, a system architecture consisting of four levels, namely data collection, data processing, anomaly detection, and alarm and response, was constructed. The data collection layer collects server status indicator data in real time through various sensors and monitoring tools, such as CPU usage, memory occupancy, disk read and write rates, etc. The data processing layer pre-processes the collected data to ensure the accuracy and integrity of the data. Subsequently, the anomaly detection layer uses the auto-encoding LSTM model of the present invention to perform anomaly detection on the processed data and identify abnormal patterns in the data. Once an anomaly is detected, the alarm and response layer will automatically trigger the alarm mechanism, send abnormal alarm information to the administrator, and provide corresponding processing suggestions.

[0125] During the actual operation, the server status indicator data for one month, including normal data and abnormal data, were collected for training and testing the AE-BiLSTM-Attn model. The experimental results show that the anomaly detection system can accurately identify abnormal patterns in the data with low false positive and false negative rates. In particular, in the detection of multiple server failures and abnormal behaviors, the system successfully provided timely processing suggestions, effectively avoiding serious consequences such as system paralysis and data loss.

[0126] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A server anomaly detection method based on time series logs, characterized in that: The following steps are involved: S1. Obtain server performance log data, that is, obtain server performance log data in real time, including CPU usage, memory usage, and disk I / O key indicator data; S2, preprocessing the server performance log data in step S1; And 70% of the server performance log data is the training set and 30% is the test set; S3. Taking the autoencoder module as the basic framework, the BiLSTM model is introduced into the encoder of the autoencoder module to construct a BiLSTM encoder, and the attention mechanism is introduced into the decoder to construct a unidirectional LSTM decoder, thereby constructing an anomaly detection model; S4, input the training set in step S2 into the anomaly detection model in step S3 for forward propagation, optimize the performance of the model by calculating the MSE loss function and updating the parameters W and b of the model through back propagation; determine whether the server performance log data is abnormal data based on the relationship between the MSE loss function and the set threshold; S5. Model evaluation: The quality of the model is evaluated through key indicators such as Precision, Recall, and F1-score, as well as ROC curve and AUC value.

2. The anomaly detection method based on time series log according to claim 1 is characterized in that: The specific method of preprocessing in step S2 is: S201, filtering the data in step S1 according to mainstream server performance KPI; S202, constructing time series data by arranging in chronological order according to the @timestamp field; S203. Delete null values, negative values, and abnormal data that have not changed to ensure data quality and consistency; S204, remove the data unit and convert it into float numerical data; S205: Normalize the data using the StandardScaler library to make the mean of each feature 0 and the standard deviation 1.

3. According to claim 1, the server anomaly detection method based on time series logs is characterized in that: Specific implementation of step S3: S301, construct an autoencoder module, the autoencoder consists of an encoder layer φ e :RF→Rf, hidden layer, decoder layer φ d :RF→Rf, where the hidden layer stores the low-dimensional feature Z; The encoder maps the sample x of the feature space of high-dimensional F to the potential low-dimensional feature Z of low-dimensional f by compression, θ e Represents the weight parameter corresponding to the encoder, the specific formula is Z = φ e (x;θ e ); Decoder φ d Reconstruct the latent low-dimensional feature Z and map it back to a sample x′, θ, which is similar to the original sample x and has the same feature dimension d Represents the weight parameter corresponding to the decoder, the specific formula is x′=φ d (Z;θ d ); S302, construct a BiLSTM encoder, and generate an internal representation containing the bidirectional dependency relationship in the input sequence through the BiLSTM encoder, that is, the hidden state h t ; S303. Introduce the attention mechanism to construct a unidirectional LSTM decoder, and the LSTM decoder generates reconstructed data based on the unidirectional LSTM.

4. According to claim 3, the server anomaly detection method based on time series logs is characterized in that: The specific implementation process in step S302 is as follows: The BiLSTM encoder consists of three layers: input layer, forward LSTM and backward LSTM. The construction process of the BiLSTM encoder is as follows: The input layer accepts the original input sequence X = {x1, x2, x3...x T }, where x t is the input vector at the tth time step, T is the maximum sequence length; the forward LSTM layer processes the input from the beginning to the end of the sequence, and obtains the hidden state of the forward LSTM The backward LSTM processes the input from the end of the sequence to the beginning, obtaining the hidden state of the backward LSTM The output h of the bidirectional LSTM t is the final hidden state of the decoder, representing the information "remembered" after processing the input sequence up to the current time step t, including time dependency, pattern, context, etc., where ";" represents the concatenation of two vectors, t = 1, 2, 3...T; Output the latent low-dimensional feature representation Z and select the last hidden state h of the encoder T As a potential low-dimensional feature representation Z = h T .

5. According to claim 3, the server anomaly detection method based on time series logs is characterized in that: The specific implementation process in step S303 is as follows: (1) Calculate the decoder hidden state s t' : The LSTM layer will calculate the hidden state s of the decoder t' , for each hidden state s at time step t′ t' , according to the hidden state s of the previous time step t'-1 And the potential low-dimensional feature representation Z output by the encoder is calculated; s t' =LSTM(Z,s t'-1 ) (2) Construct an attention mechanism: According to the hidden state s of the decoder at each time step t′ t' The final hidden state h of the encoder t The correlation between them is used to generate a context vector c′ t ; The details are as follows: Calculate the attention score e t,t' =v T tanh(W1h t +W2s t' ), represents the attention of the decoder at time step t′ to the encoder at time step t; v, W1, W2 are the parameters of the attention mechanism; Calculating attention weights The attention score is converted into a probability distribution through the softmax function so that all its weights and values ​​are 1; Calculate the context vector Denotes the decoder at time step t′ for all hidden states h of the encoder t The weighted sum of (3) Decoder output: Convert the hidden state of the encoder output back to an output sequence of the same or similar length as the input sequence; convert the context vector c t′ and the decoder hidden state s t' Combined, the output reconstructed data is generated through the fully connected layer W is the weight matrix and b is the bias term, which are randomly generated at the beginning of the model and the optimal parameter value is obtained through continuous training. The calculation formula for the reconstructed data is as follows:

6. The server anomaly detection method based on time series log according to claim 1 is characterized in that: The calculation formula of the MSE loss function in step S4 is: The threshold is mainly based on statistical analysis of training data, and the calculation formula is as follows: Threshold=μ+kσ Where μ is the mean reconstruction error on the training set, σ is the standard deviation of the reconstruction error on the training set, and k is a constant representing the tolerable error range; when the MSE loss function exceeds the set threshold, it is judged as abnormal data; otherwise, it is judged as normal data.

7. The server anomaly detection method based on time series log according to claim 1 is characterized in that: The calculation method of the key indicators of Precision, Recall and F1-score, ROC curve and AUC value in step S5 is as follows: (1) Precision: The ratio of all samples correctly judged as abnormal to the samples judged as abnormal by the model, that is, the ratio of TP to (TP + FP). The calculation formula is: TP represents the normal data points that the model correctly identifies as abnormal; FP represents the normal data points that the model incorrectly identifies as abnormal; (2) Recall: The ratio of all samples correctly judged as abnormal to the real abnormal samples, that is, the ratio of TP to (TP + FN), calculated as: Among them, FN represents that the model mistakenly judges abnormal data points as normal; (3) F1-score: The harmonic mean of Precision and Recall, calculated as: (4) ROC curve and AUC value: The ROC curve is a curve drawn with the model's true positive rate TPR, that is, the recall rate, as the vertical axis and the false positive rate FPR, that is, the ratio of all normal samples that are incorrectly judged as abnormal to the true normal samples as the horizontal axis; the AUC value is the area under the ROC curve, which is used to evaluate the comprehensive performance of the model; the AUC value is calculated according to the trapezoidal rule, that is, by dividing the area under the ROC curve into multiple trapezoids, and then calculating the areas of these trapezoids and summing them up.

Images