Real-time log anomaly detection method based on deep reinforcement learning

Through deep reinforcement learning methods, the log data is processed, the semantic features are extracted and the deep Q network is used for detection, which solves the problems of poor adaptability and time-consuming retraining in the existing technology, and realizes efficient and accurate log anomaly detection.

CN118689740BActive Publication Date: 2024-12-03WUHAN JISHU YUNZHOU TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411157308.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The existing log anomaly detection methods rely on expert knowledge, lack adaptability and generalization capabilities, and are difficult to deal with complex anomaly patterns. Deep learning-based methods require time-consuming retraining of the model when system updates.

Method used

Using a real-time log anomaly detection method based on deep reinforcement learning, replacing log data through regular expressions, extracting numerical parameters and semantic vectors, using deep Q networks for training and detection, combining experience playback technology and natural language processing technology to achieve high quality Processing of dimensional state space.

Benefits of technology

It realizes more accurate log exception detection, reduces false positives, can quickly adapt to changes when system updates, and improves detection efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118689740B_ABST
    Figure CN118689740B_ABST
Patent Text Reader

Abstract

The present invention provides a real-time log anomaly detection method based on deep reinforcement learning, comprising the following steps: step S1: replacing the parameters of each log in the log data set by regular expression to obtain log text, and extracting numerical parameters; step S2: converting the replaced log text into a semantic vector, converting the numerical parameters into a numerical parameter vector, and splicing the semantic vector and the numerical parameter vector to obtain a state vector; step S3: taking each feature vector as a state, and all replaced logs as an action set, and training a deep Q network; step S4: using the trained deep Q network to perform log anomaly monitoring, and obtaining log anomaly monitoring results. Not only can the effect and accuracy of log analysis be improved, but also it can help users better understand and manage the system.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of log anomaly detection, and in particular to a real-time log anomaly detection method based on deep reinforcement learning. Background Art

[0002] With the innovation of information technology and the widespread application of modern computer systems, various security threats and attack methods have also evolved and strengthened. Ensuring system security and reliability has become a vital task. Anomaly detection is an important means of large-scale system event management, and its purpose is to detect abnormal behavior of the system in a timely manner. At present, almost all modern computer systems have a log module. The system log records the system runtime information in detail and supports various system management and diagnostic tasks, such as ensuring application security, identifying performance anomalies, diagnosing errors and crashes, etc. Logs related to abnormal system behavior often have fixed patterns, which can be used to diagnose the source of system problems and predict potential system problems. Therefore, in the development and maintenance of many systems, system logs are widely used as a data source for anomaly detection.

[0003] Existing log anomaly detection methods can generally be divided into three types: rule-based log anomaly detection methods, machine learning-based log anomaly detection methods, and deep learning-based log anomaly detection methods.

[0004] However, rule-based methods rely heavily on domain knowledge of experts, require manual definition of static rule sets, and lack adaptability and generalization capabilities. As systems and log data change, the maintenance and update of rules also become complex and time-consuming, making it difficult to cope with complex abnormal patterns, which can easily lead to a large number of false positives.

[0005] Traditional machine learning methods such as PCA, decision trees, and invariant mining require manual design of log features for further detection. Different types of log data may have different feature patterns, and these algorithms have difficulty capturing complex nonlinear relationships.

[0006] Log anomaly detection methods based on deep learning can be divided into supervised and unsupervised methods. Supervised methods require a large amount of labeled data, and unsupervised methods usually rely on set anomaly thresholds. In addition, existing methods often need to retrain the entire model in combination with historical data when the system is updated, which is very time-consuming. Summary of the invention

[0007] To solve the above technical problems, the present invention provides a real-time log anomaly detection method based on deep reinforcement learning, comprising the following steps:

[0008] Step S1: replace the parameters of each log in the log data set with a regular expression to obtain the log text, and extract the numerical parameters;

[0009] Step S2: converting the replaced log text into a semantic vector, converting the numerical parameter into a numerical parameter vector, and concatenating the semantic vector and the numerical parameter vector to obtain a state vector;

[0010] Step S3: taking each of the feature vectors as a state and all the replaced logs as an action set to train the deep Q network;

[0011] Step S4: Use the trained deep Q network to perform log anomaly detection to obtain log anomaly detection results.

[0012] Preferably, the timestamp data is excluded from the log text, and the numerical parameter includes timestamp interval data.

[0013] Preferably, the expression for converting the replaced log text into a semantic vector in step S2 is:

[0014] ;

[0015] ;

[0016] In the formula, represents the sentence vector before removing the principal component, represents the number of different logs after preprocessing, is a constant, It's a word The probability of appearing in a log statement, Representing words The word vector of express All vectors of are concatenated to get the first singular vector of the matrix, Represents the final semantic vector.

[0017] Preferably, the deep Q network includes an experience pool, a current value network, a target value network and an agent that takes actions for different states, and the agent is updated using experience replay technology.

[0018] Preferably, the method for training in step S3 includes:

[0019] Step S31: Initialize the experience pool;

[0020] Step S32: randomly sampling n samples from the experience pool for training;

[0021] Step S33: input the current state vector into the current value network to generate a predicted Q value, and calculate the target Q value;

[0022] Step S34: Calculate the error loss between the predicted Q value and the target Q value;

[0023] Step S35: Repeat steps S32 to S33 to perform back propagation on the current value network parameters, and optimize them through the optimizer. After each round, the parameter values ​​of the current value network are assigned to the target value network to complete the training.

[0024] Preferably, step S31 includes:

[0025] Step S311: Set the log reward of each session in the training set to 0, and initialize the structure of the experience pool to an empty list;

[0026] Step S312: traverse each state vector, and the agent selects the next action based on the ε-greedy strategy, that is, randomly selects an action with a probability of the greedy coefficient ε, inputs the state value into the current value network with a probability of 1-ε, outputs the score of each action and selects the action with the largest score;

[0027] Step S313: Read the next state value and set the action reward. If the next state is the last one in the log sequence, set the reward of the action to 50 and record the abort state value. Otherwise, set the reward to 1.

[0028] Step S314: according to the information obtained in step S312 and step S313, add a five-tuple in the format of (state, action, reward, next state, termination flag) to the experience pool;

[0029] Step S315: Repeat steps S312 to S314 until the entire training set is traversed to initialize the experience pool.

[0030] Preferably, the target Q value The expression is:

[0031] ;

[0032] In the formula, R, They are action reward, next state value and round termination flag respectively. is the discount factor for the next reward, , Represents the cumulative rewards of taking different actions when the next state value is known.

[0033] Preferably, the deep Q network further includes a parameter update step for the newly added logs:

[0034] Step S361: Processing the log data in adjacent windows of fixed time length to obtain a state vector;

[0035] Step S362: Calculate the standard mean difference of the two window data;

[0036] Step S363: continuously monitor the standard mean difference of the adjacent window data, and when it is greater than the threshold, collect the data of the second window and put it into the experience pool;

[0037] Step S364: Update parameters for the data in the experience pool.

[0038] Preferably, the expression of step S364 is:

[0039] ;

[0040] In the formula, is the learning rate, is the updated parameter, is the gradient of the deep Q network loss function with respect to the update data, and W represents the searchable parameter space.

[0041] Preferably, in step S35, the optimizer selects Adam for optimization.

[0042] The beneficial effects of the present invention include at least: extracting accurate semantic features and quantitative features of log sequences through sentence embedding technology of natural language processing and log parameter parsing. By using the deep Q network DQN, which uses a deep neural network to approximate the state function, the intelligent agent can process high-dimensional state space and achieve more accurate log anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of a data preprocessing process according to an embodiment of the present invention;

[0045] Figure 3 A framework diagram of a deep Q network model provided by an embodiment of the present invention;

[0046] Figure 4 The following is a flow chart of log anomaly detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0048] like Figure 1 As shown, an embodiment of the present invention provides a real-time log anomaly detection method based on deep reinforcement learning, comprising the following steps:

[0049] Step S1: Replace the parameters of each log in the log data set with a regular expression to obtain the log text, and extract the numerical parameters.

[0050] Specifically, the log information is mainly divided into two parts. One part is the information related to the device and time, such as process number, timestamp, device number, etc., and the other part is the main content of the log. All the characters in it are converted to lowercase, and regular expressions are used to extract the key parameters in the log information. The parameters include text parameters and numerical parameters. Text parameters include file path, IP address and port number, as well as log level, etc.

[0051] Therefore, in the embodiment of the present invention, the following regular matching formula is set, the parameter type is counted, and the corresponding position characters matched in the log are replaced with the type name of the parameter. The list of the corresponding content of the log and the parameter name replaced by it is shown in Table 1. The preprocessing process is as follows: Figure 2 shown.

[0052] Table 1

[0053]

[0054] Step S2: converting the replaced log text into a semantic vector, converting the numerical parameter into a numerical parameter vector, and concatenating the semantic vector and the numerical parameter vector to obtain a state vector.

[0055] Specifically, in the embodiment of the present invention, the log statements after the text parameters are replaced are converted into semantic vectors using sentence embedding technology. Log information can be regarded as a semi-structured text content, so the contextual semantic information can be extracted using natural language processing methods. First, the log text content is segmented, and then the pre-trained word vector is used to obtain the word vector representation of each word after segmentation. This example uses word vectors trained using subword information on Wikipedia in 2017, UMBC web-based corpus, and the statmt.org news dataset in FastText, and then uses the SIF algorithm to calculate the sentence vector for each log sentence. , the formula is:

[0056] ;

[0057] in Indicates the number of different logs after preprocessing. The essence of system logs is a combination of system print statements and placeholder parameters. Since print statements are limited and the corresponding numerical and text parameters are replaced after log preprocessing, is a finite constant. is a constant, It's a word The probability of appearing in the log statement is concatenated into a matrix by concatenating all vectors obtained by the left formula , This is the first singular vector of the matrix. Then, the word vectors are subtracted from their respective word vectors in the sentence vector matrix by the right formula. The final semantic vector is obtained by projecting it onto the first singular vector of . This method retains the important information of each word vector and is simple, efficient and robust.

[0058] Then, the numerical parameters of the log are converted into vectors of fixed dimensions, such as the time interval with the previous log and the quantity information appearing in the log entry. Then, the semantic vector and the numerical parameter vector are concatenated to obtain the high-dimensional feature vector corresponding to each log.

[0059] In order to reduce the influence of fixed information on the final vector in the embodiment of the present invention, the timestamp data in the original document, such as Figure 2 1118246890 is deleted and converted into numerical features for processing, such as 0, 2, and 5 in the parameters, which represent the time interval between the current log and the first log.

[0060] Step S3: Take each feature vector as a state and all replaced logs as an action set to train the deep Q network.

[0061] Specifically, in the embodiment of the present invention, the data set is first divided, and after the collected logs are converted into vectors according to the above steps, considering the imbalance of positive and abnormal logs, 80% of the normal execution logs are taken as the training set, and the other 20% of the normal execution logs and the logs containing abnormalities are taken as the test set.

[0062] The embodiment of the present invention will not be described in detail with respect to the traditional deep Q network, and only the improved deep Q network provided by the embodiment of the present invention will be described.

[0063] Its network model is as follows Figure 3As shown, it mainly includes an experience pool (Replay Buffer), a current value network (Q-Net), a target value network (QT-Net), and an agent that performs actions for different states. In this embodiment, a session log of normal execution of the system is regarded as a round, and the vector of each log after processing is regarded as a state (State), and different logs after preprocessing are regarded as all action (Action) sets. The current value network and the target value network have the same structure, consisting of four fully connected layers and three ReLU activation functions. The input is the state vector corresponding to the log, and the first linear layer expands the input dimension from Input_Size to 256. The subsequent linear layers gradually reduce the dimension, and finally output a vector with a dimension of all possible actions, indicating the cumulative reward corresponding to each possible action.

[0064] Initialize the experience pool:

[0065] The first step is to set the log reward of each session of the training set to 0 and initialize the structure of the experience pool to an empty list;

[0066] In the second step, the log vector of each log sequence is traversed as the state value. The agent selects the next action based on the ε-greedy strategy, that is, randomly selects an action with a probability of the greedy coefficient ε, inputs the state value into the current value network with a probability of 1-ε, outputs the score of each action and selects the action with the largest score. The value of ε is usually between 0 and 1, which determines the degree of exploration. In this embodiment, ε is set to 0.1 to balance the trade-off between exploration and the use of known information.

[0067] The third step is to read the next state value and set the action reward. If the next state is the last one in the log sequence, set the reward of the action to 50 and record the termination state value. Otherwise, set the reward to 1.

[0068] The fourth step is to add a five-tuple of the format (state, action, reward, next state, termination flag) to the experience pool based on the information obtained in the first two steps. For a log sequence with an input length of N, N-1 of the above tuples will be generated.

[0069] Repeat steps 2 to 4 until the entire training set is traversed. In the embodiment of the present invention, the experience pool can enable the agent to learn from past experience by storing and reusing experience data through the experience playback method, rather than just being limited to the current interaction, which can improve the utilization efficiency of samples and make training more stable and efficient. The experience pool has a fixed capacity, and the elements put in will overwrite the old values ​​from the front to the back if they exceed the capacity.

[0070] Train the model:

[0071] The first step is to randomly sample Batch_Size samples from the experience pool for training. Sampling converts all elements of the quintuple into floating-point tensors. Continuous samples are often highly correlated, and random sampling can break this correlation, making training more independent and stable.

[0072] In the second step, the current state vector is input into the current value network to generate the predicted Q value, and the target Q value is further calculated based on the other values ​​of the quintuple. The formula is as follows:

[0073] ;

[0074] Among them, R They are action reward, next state value and round termination flag respectively. , is the discount factor for the next reward, when When it is close to 0, the agent pays more attention to the immediate reward and pays less attention to the next reward. In this embodiment, it is set to 0.99. Different Q values ​​reflect the different cumulative rewards that can be obtained after performing a certain action in the current state. Represents the cumulative rewards of taking different actions when the next state value is known.

[0075] The third step is to calculate the mean square error (MSE) loss L between the predicted Q value and the target Q value (targets), the formula is,

[0076] ;

[0077] Repeat steps one to three to perform back propagation on the current value network parameters, and set the optimizer to Adam. This embodiment tries multiple sets of hyperparameter combinations to make the model converge to achieve better performance. During the training process, this embodiment executes 1000 epochs, and after each epoch, the parameter values ​​of the current value network are assigned to the target value network. After the training is completed, the predicted value network parameters and the termination state are saved to the file. The set of termination states can be regarded as a priori knowledge. When the termination state of a log sequence is abnormal, the log sequence is naturally abnormal.

[0078] After the anomaly detection model is deployed and launched, if a system update occurs, changes in the corresponding logs may have an adverse effect on the detection performance of the model. For large-scale systems, the long-term accumulated historical logs are huge, and retraining for historical logs is very time-consuming. Therefore, the following method is provided in the embodiment of the present invention to update parameters based on the changed data.

[0079] The first step is to preprocess the log data in adjacent windows of fixed time length to obtain vectors;

[0080] The second step is to calculate the standard mean difference (SMD) of the two window data. The formula is

[0081] ;

[0082] in represents the mean of the t-th window data, Represents the variance of the t-th window data.

[0083] The third step is to continuously monitor the standard mean difference of adjacent window data. When it is greater than the threshold, collect the data of the second window and put it into the experience pool. Since the size of the experience pool is fixed, the historical data in the experience pool will be continuously overwritten during continuous monitoring.

[0084] The fourth step is to update the parameters for the data in the experience pool. The formula is:

[0085] ;

[0086] in is the learning rate, is the updated parameter, This is the gradient of the DQN model loss function for the updated data. In order to prevent the model from being only sensitive to the current data after the parameter update and forgetting the historical data, an L2 penalty term for the parameter is added. , so that the parameters of the model do not change drastically during the update, resulting in unstable model prediction, where W represents the searchable parameter space.

[0087] Step S4: Use the trained deep Q network to perform log anomaly detection to obtain log anomaly detection results.

[0088] Specifically, the detection process is as follows Figure 4 As shown, iterate each log sequence in the test set, read the model parameters and termination state from the file, traverse the log vector of the log sequence as the state value and input it into the model to obtain the Q value of the state, that is, the corresponding cumulative reward under the state. If it is greater than the threshold 50 and the last log termination state in the sequence is normal, then the sequence is considered normal, otherwise it is abnormal. In order to evaluate the performance of the method, this embodiment uses a variety of evaluation indicators, including accuracy, recall rate and F1 value. The following are the definitions and calculation formulas of these evaluation indicators:

[0089] 1. Precision is the ratio of the number of correct predictions of the log to the total number of all predictions. The formula is:

[0090] ;

[0091] : True Positives, the number of samples where abnormal logs are correctly identified. : True Negatives, the number of samples where normal logs are correctly identified. : False Positives, the number of samples where the model incorrectly predicts normal logs as abnormal, that is, the number of false positives. : False Negatives, the number of samples where the model did not identify abnormal logs, that is, the number of missed reports.

[0092] 2. Recall describes the proportion of identified abnormal samples to all abnormal samples. The formula is:

[0093] ;

[0094] 3. The F1 value is the harmonic mean of the precision and recall, and the formula is:

[0095] ;

[0096] The following DeepLog method and QLLog method are selected as baselines and compared with the solution LogDRL proposed in this invention. The three log anomaly detection methods are compared on the public data sets HDFS and BGL. The detection results of the data set HDFS are shown in Table 2, and the detection results of the data set HDFS are shown in Table 3.

[0097] Table 2

[0098] Precision Recall F1-score DeepLog 95% 96% 96% QLlog 95% 98% 96% LogDRL 99% 99% 99%

[0099] Since the HDFS data set comes from a Hadoop system that has not been updated, and the interval for collecting the log data does not exceed 48 hours, the log statements are relatively stable, and the model in DRL does not update parameters. Since the traditional reinforcement learning method has limited state, and deep learning relies on long-term dependencies for prediction, the scheme LogDRL proposed in the present invention has higher precision, recall rate and F1 value than the deep learning-based and reinforcement learning-based methods DeepLog and QLlog on a stable HDFS data set.

[0100] Table 3

[0101] Precision Recall F1-score DeepLog 21% 95% 34% QLlog 76% 94% 84% LogDRL 82% 98% 89%

[0102] Since the interval between the BGL dataset is as long as 7 months, the system has been updated many times, and the log statements have changed greatly in different periods. The other two methods are prone to more false positive predictions. The method DRL proposed in this scheme can update parameters when the changes are large. Therefore, the accuracy, recall rate and F1 value of the relatively unstable BGL dataset are higher than those of the deep learning-based and reinforcement learning-based methods DeepLog and QLlog.

[0103] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0104] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A real-time log anomaly detection method based on deep reinforcement learning, characterized by: The following steps are involved: Step S1: replace the parameters of each log in the log data set with a regular expression to obtain the log text, and extract the numerical parameters; Step S2: converting the replaced log text into a semantic vector, converting the numerical parameter into a numerical parameter vector, and concatenating the semantic vector and the numerical parameter vector to obtain a state vector; Step S3: Take each state vector as a state, and all replaced logs as an action set to train the deep Q network; Step S4: Use the trained deep Q network to perform log anomaly detection and obtain the log anomaly detection result: the log vector of the traversal log sequence is input as the state value into the deep Q network to obtain the Q value of the state. If it is greater than the set threshold and the termination state of the last log in the sequence is normal, the sequence is considered normal, otherwise it is abnormal.

2. The real-time log anomaly detection method based on deep reinforcement learning according to claim 1 is characterized in that: The timestamp data is removed from the log text, and the numerical parameter includes timestamp interval data.

3. The real-time log anomaly detection method based on deep reinforcement learning according to claim 1 is characterized in that: The expression for converting the replaced log text into a semantic vector in step S2 is: ; ; In the formula, represents the sentence vector before removing the principal component, represents the number of different logs after preprocessing, is a constant, It's a word The probability of appearing in a log statement, Representing words The word vector of express All vectors of are concatenated to get the first singular vector of the matrix, Represents the final semantic vector.

4. The real-time log anomaly detection method based on deep reinforcement learning according to claim 1 is characterized in that: The deep Q network includes an experience pool, a current value network, a target value network and an intelligent agent that takes actions for different states, and the intelligent agent is updated using experience replay technology.

5. The real-time log anomaly detection method based on deep reinforcement learning according to claim 4 is characterized in that: The method for training in step S3 includes: Step S31: Initialize the experience pool; Step S32: randomly sampling n samples from the experience pool for training; Step S33: input the current state vector into the current value network to generate a predicted Q value, and calculate the target Q value; Step S34: Calculate the error loss between the predicted Q value and the target Q value; Step S35: Repeat steps S32 to S33 to perform back propagation on the current value network parameters, and optimize them through the optimizer. After each round, the parameter values ​​of the current value network are assigned to the target value network to complete the training.

6. The real-time log anomaly detection method based on deep reinforcement learning according to claim 5 is characterized in that: Step S31 includes: Step S311: Set the log reward of each session in the training set to 0, and initialize the structure of the experience pool to an empty list; Step S312: traverse the state vector of each log sequence, and the agent selects the next action based on the ε-greedy strategy, that is, randomly selects an action with a probability of the greedy coefficient ε, inputs the state value into the current value network with a probability of 1-ε, outputs the score of each action and selects the action with the largest score; Step S313: Read the next state value and set the action reward. If the next state is the last one in the log sequence, set the reward of the action to 50 and record the abort state value. Otherwise, set the reward to 1. Step S314: according to the information obtained in step S312 and step S313, add a five-tuple in the format of (state, action, reward, next state, termination flag) to the experience pool; Step S315: Repeat steps S312 to S314 until the entire training set is traversed to initialize the experience pool.

7. The real-time log anomaly detection method based on deep reinforcement learning according to claim 5 is characterized in that: The target Q value The expression is: ; In the formula, R, They are action reward, next state value and round termination flag respectively. is the discount factor for the next reward, , Represents the cumulative rewards of taking different actions when the next state value is known.

8. The real-time log anomaly detection method based on deep reinforcement learning according to claim 5 is characterized in that: The deep Q network also includes a parameter update step for the newly added logs: Step S361: Processing the log data in adjacent windows of fixed time length to obtain a state vector; Step S362: Calculate the standard mean difference of the two window data; Step S363: continuously monitor the standard mean difference of the adjacent window data, and when it is greater than the threshold, collect the data of the second window and put it into the experience pool; Step S364: Update parameters for the data in the experience pool.

9. The real-time log anomaly detection method based on deep reinforcement learning according to claim 8, characterized in that: The expression of step S364 is: ; In the formula, is the learning rate, is the updated parameter, is the gradient of the deep Q network loss function with respect to the update data, and W represents the searchable parameter space.

10. The real-time log anomaly detection method based on deep reinforcement learning according to claim 5, characterized in that: In step S35, the optimizer selects Adam for optimization.