A large-scale system log anomaly detection method based on attention mechanism
Through the generative adversarial network model based on attention mechanism, the problem of the inability to detect whether a log in a large-scale system log in the prior art is solved, and the fine detection and positioning of system abnormalities is realized, which improves the security of the system and reliability.
Patent Information
- Application Number
- CN202111128644.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-26
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2041-09-26
AI Technical Summary
The existing technology cannot accurately detect whether a log in large-scale system logs is abnormal, and cannot locate the specific reasons for system abnormality, resulting in the inability to effectively prevent system crashes and network security events.
Using a generative adversarial network model based on attention mechanism, through the alternating update mechanism between the generator and the discriminator, a model that fits the real subsequent normal events is generated, and the type of the real subsequent log is judged as to determine whether the real subsequent log is the same as the generated subsequent normal log, and then the system is judged. Is it abnormal?
It realizes fine abnormality detection of large-scale system logs, can accurately determine whether the log is abnormal, locate the cause of system abnormality, and improves the security and reliability of the system.
Smart Images

Figure CN113918367B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent operation and maintenance, and more specifically, relates to a large-scale system log anomaly detection method based on an attention mechanism. Background Art
[0002] Nowadays, large-scale systems are used in all walks of life, and the services they can provide have become diversified. The quality of services has greatly affected the user experience. However, due to the instability of computer hardware and software, various errors will occur during the operation of the system, resulting in system downtime. On the other hand, the existing network environment is extremely complex, and various new types of attacks on computer systems emerge in an endless stream. When attacked from the outside, network security personnel are unable to take effective preventive measures, resulting in system crashes. Both of these situations will cause immeasurable losses to service providers. More and more institutions and research are committed to detecting whether the existing information in the computer system is in an abnormal state, and effective measures can be taken in advance to avoid risks. Therefore, research on anomaly detection is very necessary.
[0003] A large number of log files are generated every day in large-scale systems, recording some important events in the system operation process, tracking and monitoring the operation status of the computer. Detecting whether the system is in an abnormal state by the output status of the log is a feasible solution. However, large systems and their underlying machines are usually monitored by many administrators, but compared with the number of millions of logs recorded every day, it is not feasible to simply filter out abnormal logs by manpower. At present, some machine learning-based methods determine whether a certain log sequence is abnormal based on the characteristics of the sequence, thereby determining whether the system is in an abnormal state. However, this coarse-grained session-level detection method cannot reflect whether a specific log is abnormal, and thus cannot more accurately locate the cause of the system abnormality.
[0004] Anomaly detection technology is widely used in some practical scenarios, such as financial fraud, post-disaster situation analysis, and social media event monitoring. At present, there are many studies on anomaly detection based on machine learning methods, and according to different learning categories, they are mainly divided into supervised learning tasks, semi-supervised learning tasks, and unsupervised learning tasks. In supervised learning tasks, the labels of samples are available and positive and negative samples are separable in a specific space. The paper "Anomaly Detection Method Based on Decision Tree" proposes a detection method based on decision trees, which uses server request sequences to detect failures of large-scale network sites. Decision trees can simultaneously process discrete attributes and continuous attributes of samples, so this method is widely used. The paper "Log Event Anomaly Detection in IBM System" uses multiple classifiers (such as support vector machine algorithms) to detect log anomalies in large-scale clusters. In addition, supervised learning models can quickly achieve the purpose of online detection, but this method is very dependent on the quality of data set labels, so the use scenario is greatly limited. The premise of using semi-supervised learning algorithms is that the distance between positive samples in a specific space is very close, and abnormal samples are far away from the normal sample group. The most representative of them is the nearest neighbor algorithm, which can be divided into the nearest neighbor algorithm using distance measurement and the nearest neighbor algorithm using density measurement according to different evaluation indicators. The paper "Finding Outlier Subspaces in High-Dimensional Data" proposed a high-dimensional outlier subspace detection method, which uses the nearest neighbor algorithm to find the optimal feature subspace that can represent outlier samples in high-dimensional space, so as to divide out abnormal samples. In addition, the paper "Deep Log: System Log Anomaly Detection Method Based on Deep Learning" proposed an anomaly detection framework DeepLog based on long short-term memory network, which uses log key values and parameters to help administrators diagnose system errors in workflows. However, in real scenarios, abnormal samples are often sparse and unlabeled, so unsupervised learning methods are more suitable for such scenarios. The paper "Isolation Tree" uses the concept of isolating abnormal samples and proposes an isolation forest model, which constructs a forest by randomly selecting attributes to achieve the purpose of distinguishing normal samples from abnormal samples. Although unsupervised learning methods do not rely on data labels, they require deep expert knowledge to solve problems in real scenarios.
[0005] The above existing methods can only solve the coarse-grained log anomaly detection task, that is, the model can only detect whether there is an anomaly in a log sequence, but cannot detect whether there is an anomaly in a log in the sequence. Summary of the invention
[0006] In view of the problem that the existing model can only detect whether a section of a log sequence has an anomaly, but cannot detect whether a certain log in the sequence has an anomaly, the present invention provides a large-scale system log anomaly detection method based on an attention mechanism, which adopts a generative adversarial network model structure. The model consists of two parts, a generator and a discriminator. Through the mechanism of continuous alternating update between the two, a model that generates and fits the actual subsequent normal events is obtained, and then the actual subsequent log is compared with the generated subsequent normal log type to determine whether the system is abnormal.
[0007] To solve the above problems, the present invention adopts the following technical solutions.
[0008] A large-scale system log anomaly detection method based on attention mechanism includes the following steps:
[0009] Step S1, data processing stage: extracting log information generated by the computer system, performing data preprocessing on the log information, and constructing a log context combination set C;
[0010] Step S2, adversarial training stage: input the log context combination set C constructed in step S1 into the neural network model to train the neural network model;
[0011] Step S3, detection phase: obtain the log information to be predicted, perform data preprocessing on the log information to be predicted, construct a log event sequence to be predicted, input the log event sequence to be predicted into the neural network model trained in step S2 to generate subsequent subsequence events, and compare the subsequent subsequence events with the real events. If the subsequent subsequence events are the same as the real events, the system is in a normal state, otherwise, the system is out of an abnormal state.
[0012] In a further technical solution, in step 1, the specific steps of preprocessing the log information are:
[0013] S101, collect sample data S = {e (1) , e (2) ,……,e (l)}, where e (l) represents each log, l represents the number of logs;
[0014] S102, use a sliding window of size n to perform sliding processing with a step length of 1 on the initial sequence data S to collect data, and obtain a context combination set C = {c 1 ,c 2 ,……,c i}, i represents the number of context combinations, c i represents a set of log sequences of length n {e (i) ,e (i+1) ,……,e(i+n)}.
[0015] According to a further technical solution, in step S2, the neural network model includes a generator model and a discriminator model, and the generator model and the discriminator model are trained using an alternating updating training mechanism. The specific training method is:
[0016] Step S201, inputting the log context combination set C constructed in step S1 into the generator to generate subsequent subsequence events;
[0017] Step S202, inputting the subsequent subsequence event and the real event into the discriminator model respectively, and the discriminator model determines whether the subsequent subsequence event is a real event;
[0018] Step S203: During the discriminator model's discriminative process, the loss function L in the discriminator model D The parameters of the discriminator model are continuously updated until the discriminator model can determine whether the subsequent subsequence events are real events. The training of the discriminator model is completed and the training of the generator model is carried out.
[0019] Step S204: The subsequent subsequence events generated by the generator model are discriminated by the trained discriminator model, and the discriminator model inputs the discrimination results into the generator model;
[0020] Step S205: Loss function L of the generator model G The model error is generated by combining the discrimination result of the discriminator model and the difference between the subsequent subsequence events generated by the generator model and the real events. The generator model adjusts its own network parameters according to the model error, so that the generator model continuously generates more realistic subsequent subsequence events.
[0021] Step S206: Input the subsequent subsequence events and real events generated by the generator model into the discriminator model for discrimination. When the discriminator model cannot distinguish whether the input subsequence event is a subsequent subsequence event generated by the generator model or a real event, the training of the generator model ends, and the process returns to step S201 to continue training the discriminator model.
[0022] Step S207: When the set number of training cycles is reached, the generator model and the discriminator model reach a balance, a mature generator model is obtained, and the training ends.
[0023] In a further technical solution, in step S203, the loss function L of the discriminator model D =min φ-(logD(e|c)+log(1-D(e|c))), where φ represents the parameters of the discriminator model, D represents the discriminator model, e is the vector representation of the real event, and c is the vector representation of the log context combination set input to the generator model.
[0024] In a further technical solution, in step S205, the loss function of the generator model is Among them, θ represents the parameters of the generator model, A vector representation of the subsequent subsequence events produced by the generator model.
[0025] According to a further technical solution, the generator model is composed of three layers of neural networks, namely: long short-term memory neural network layer, attention neural network layer and fully connected network layer.
[0026] According to a further technical solution, the discriminator model is composed of two layers of neural networks, namely: a long short-term memory neural network layer and a fully connected network layer.
[0027] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:
[0028] The present invention discloses a log anomaly detection method based on an attention mechanism, which adopts a generative adversarial network model structure. The model consists of two parts: a generator and a discriminator. The generator combining a long short-term memory network and an attention mechanism is used to learn the distribution of data samples and generate subsequent normal log events of a log sequence. The discriminator determines whether the data distribution learned by the generator conforms to the sample distribution of real log sequence data. Through a mechanism of continuous alternating updates between the two, a model that generates and fits real subsequent normal events is obtained, and then the real subsequent log is compared with the generated subsequent normal log to determine whether the type is the same to determine whether the system is abnormal. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] Figure 1 It is the overall flow chart of the present invention. DETAILED DESCRIPTION
[0030] The present invention is further described below in conjunction with specific embodiments and drawings.
[0031] Example
[0032] This embodiment provides a log anomaly detection method based on the attention mechanism. Figure 1 As shown in the figure, the log anomaly detection method based on the attention mechanism mainly includes three parts: log parsing, adversarial training, and anomaly detection.
[0033] Log analysis:
[0034] The log parsing module is mainly responsible for converting the original unstructured log file into a structured log information description as the input data of the generative adversarial network. The information of the log file mainly consists of three parts: timestamp, signature, and parameter. Since the parameter is a dynamic information related to the state of the system at the current moment of log output, it cannot provide effective help for parsing log events. Therefore, the parameter information will not be extracted during log parsing. Therefore, this work mainly focuses on the remaining two main contents: timestamp and signature. The timestamp records the time when the log occurs and can be used to provide a basis for the order arrangement in the log sequence. The signature is the information that remains unchanged during the output process of the same type of log, which is an important basis for judging log parsing. The original log file is very messy, so the log is matched by constructing a regular expression to format the information. Then the logs with similar signature information are classified as the same type of log events, and the number of types of all logs is counted, and the m-dimensional vector E is used to represent m types of log events.
[0035] Assume that the initial window size is n, n logs within the window range are used as sequence data, and the n+1th log is used as the subsequent log. The window slides on the sequence with a step size of 1, and each slide generates a training data. The idea of using a sliding window to construct log sequence data ensures the number of training data, so that the model can learn more comprehensive knowledge during the training process. The sequential characteristics of logs contain rich information, but large-scale systems are generally in a high-concurrency state when running, and can generate millions of log data per second. These logs will be disordered in the transmission process due to network delays and other problems, which is inconsistent with the real order. This method introduces the method of disordered event modeling to reorder the log sequence. This method increases the size of the data while solving the disorder problem. In addition, there is a common phenomenon of sample imbalance in anomaly detection data. The number of normal samples far exceeds the number of abnormal samples. The model is more biased towards the distribution of normal samples during the learning process, and the learning of the distribution of abnormal samples is not sufficient, which makes the model prone to overfitting during the training process. The negative sample sampling method is used to increase the number of negative samples and effectively balance the number of positive and negative samples in the training set.
[0036] The log sequence set needs to be embedded in the vector space. Here, the word2vec model is used to project the log sequence set into the n*s dimensional vector space to obtain the specific n-dimensional vector representation of the log event. The vector representation of each log event in the sequence is concatenated into an h*n dimensional log sequence vector representation.
[0037] Adversarial Training:
[0038] The log anomaly detection model is mainly built based on the generative adversarial network, which mainly includes two models, the generator model and the discriminator model. The generator model is mainly responsible for generating subsequent normal subsequence log events based on the input log sequence, and the discriminator model is responsible for determining whether the input normal log sequence event is a real subsequent subsequence log event or a subsequent normal subsequence log event generated by the generator. The two constantly compete to optimize their respective capabilities. The generator model consists of three layers of networks, namely: long short-term memory neural network layer, attention neural network layer, and fully connected network layer. The discriminator model is similar to it. To prevent the network from being too complex and difficult to converge, the attention neural network layer is removed, which is: long short-term memory neural network layer and fully connected network layer.
[0039] Given sample data S = {e (1) ,e (2) ,……,e (l)},e (l) represents each log, l represents the number of logs, and the log event encoding vector E obtained in the log parsing stage above is E={e 1 ,e 2 ,……,e m},e m Represents a log event, uses a sliding window of size n to perform sliding processing with a step size of 1 on the initial sequence data S to collect data, and obtains a context combination set C = {c 1 ,c 2 ,……,c i}, c i represents a set of log sequences of length n {e (i) ,e (i+1) ,……,e (i+n)}, i represents the number of context combinations, and in this process, the log sequences in the combination set C are labeled. If the next log in the log sequence is an abnormal log, the data label is 1, otherwise it is 0.
[0040] During the training process, in order to make the feature extraction of log sequences more refined, the attention neural network is introduced. The h*n-dimensional log sequence vector obtained in the log parsing stage is input into the attention neural network. First, the importance of each log event in the sequence to the other log events needs to be calculated. In the process of calculating the weight, each log event is further coded, and the activation function tanh is used to increase the nonlinearity of the model to obtain a log event vector with further extracted features and changed dimensions. The following is the formula of the attention neural network:
[0041] out i =tanh(W w h i +b a )
[0042] Where W w ,b a Denote weight and bias terms respectively, h i For each log vector representation, h i Encoded into a key-value vector out i , then perform pairwise matrix product operations between events to obtain the weights between events, and use the mean normalization method to normalize the weight values:
[0043]
[0044] Among them, the random vector u w Represents the characteristics of the entire sequence, α i It represents the importance of each log in the calculation sequence to the log context sequence. Then, the weighted summation method is used to obtain the vector representation of each event that integrates the features of other events in the sequence:
[0045]
[0046] Among them, s represents the vector representation of the log context sequence. Finally, each log event vector is concatenated to obtain the feature representation of the entire log sequence. The attention neural network does not rely too much on the order information of the sequence when processing serialized data, which reduces the impact of the order disorder caused by network delay mentioned above.
[0047] The log context sequence vector representation s is input into the fully connected network layer. The sequence vector is projected into the m-dimensional probability vector space through the fully connected layer to indicate the possible types of subsequent events. The top p event types with the highest probability are selected to form an m-dimensional 0-1 vector. 1 indicates that the subsequent event may be this event, and 0 indicates otherwise. The process is represented by the following formula:
[0048] E = sigmoid(ReLU(W f s+b f ))
[0049] Among them, W f ,b f They are the weights and bias items of the fully connected network layer respectively. The final output E is an m-dimensional vector. Each dimension represents the probability of a normal subsequent log. By setting the threshold, the most likely normal subsequent log is screened out.
[0050] The discriminator consists of a long short-term memory neural network layer and a fully connected network layer. The m-dimensional probability distribution vector E and the real subsequence event vector R generated by the generator are respectively input into the discriminator network, and the discriminator is trained using a supervised learning scheme. Among them, the E vector has a total of m dimensions, indicating that there are a total of m types of events. The number above each dimension is a probability value, indicating the probability that the subsequent subsequence event may be of this type. The final output is the probability result of whether the vector is a real subsequence event. The formula for this process is:
[0051] p=sigmoid(,E,R]W D +b D )
[0052] Among them, W D , b D Represent the weight and bias of the discriminator respectively. The ultimate goal of this training scheme is to increase the authenticity of the subsequent event sequence generated by the generator.
[0053] The generator and the discriminator optimize their respective network parameters through a game mechanism and an alternating update training mechanism. In the initial training phase of the entire model, the discriminator model is first trained, and the subsequent subsequence events generated by the generator and the real subsequence events are labeled and input into the discriminator model respectively, and the loss function L of the discriminator model is used to calculate the loss function L. D The model parameters are continuously updated until the training is stopped when the model has a good ability to judge whether the subsequent subsequence events are real subsequence events. The following is the formula of the discriminator's loss function.
[0054]
[0055] Among them, φ represents the parameters of the discriminator model, D represents the discriminator model, e is the vector representation of the real event, and c is the vector representation of the log context set input by the generator model.
[0056] Then the generator model is trained, and the generator continuously generates subsequent subsequence events. At the same time, the generated subsequent subsequence events are input into the discriminator model, and the discriminator model determines whether the subsequent subsequence event is a real subsequent subsequence event. The loss function L G The difference between the result judged by the comprehensive discriminator model and the generator result and the correct result generates the model error. The generator model adjusts its own network parameters according to the error, so that it continuously generates data that better fits the distribution of the actual subsequent subsequence log events. The discriminator cannot distinguish whether the generated data is generated by the generator. The following is the loss function of the generator (where θ represents the parameters of the generator model).
[0057]
[0058] Among them, θ represents the parameters of the generator model, is the vector representation of the subsequent subsequence events generated by the generator model, c is the vector representation of the log context set input to the generator model, and e is the vector representation of the real event.
[0059] Assume that é is used to represent the output of the generator. During the training process, o is an m-dimensional 0-1 vector, which indicates that the positions of m log events that have appeared in the subsequent logs are set to 1, and those that have not appeared are set to 0. The probability of events that have not occurred in the subsequent subsequence events is filtered out, and only the probability of events that have occurred is used to update the generator network. This can effectively avoid the misleading of network updates by events that have not occurred in the subsequent subsequence. Generator training is terminated when the discriminator cannot distinguish whether the input subsequence event is a real subsequent subsequence event or a subsequent subsequence event generated by the generator.
[0060] Afterwards, the discriminator uses the trained generator to update its own model parameters again to enhance its ability to distinguish input data. The two continue to compete, and after reaching the number of cyclic training times, the discriminator and generator finally reach a balance and obtain a mature generator model.
[0061] Anomaly Detection
[0062] The purpose of the discriminator model is to help the generator continuously optimize network parameters during the model training process. After the overall model training is completed, the discriminator loses its function.
[0063] In the log anomaly detection phase, the log information to be predicted is first parsed and processed to construct a set of log context combinations to be predicted. The generator generates an m-dimensional subsequent log vector by inputting a log sequence, indicating the probability that each log may be a normal log. A certain threshold is set in advance, and the log with a probability exceeding the threshold is a normal log. Then, the real log event is compared to see if it is the same as the normal log event generated by the generator. If they are the same, the log is a normal log, otherwise, it is an abnormal log.
[0064] The examples described in the present invention are merely descriptions of the preferred implementation modes of the present invention, and are not intended to limit the concept and scope of the present invention. Without departing from the design concept of the present invention, various modifications and improvements made to the technical solutions of the present invention by engineers and technicians in this field should all fall within the protection scope of the present invention.
Claims
1. A large-scale system log anomaly detection method based on attention mechanism, characterized in that: The steps include: Step S1, data processing stage: extracting log information generated by the computer system, performing data preprocessing on the log information, and constructing a log context combination set C; Step S2, adversarial training phase: input the log context combination set C constructed in step S1 and the real event into the neural network model to train the neural network model; Step S3, detection stage: obtaining the log information to be predicted, performing data preprocessing on the log information to be predicted, constructing a log context combination set to be predicted, inputting the log context combination set to be predicted into the neural network model trained in step S2 to generate subsequent subsequence events, and comparing the subsequent subsequence events with real events; In step S1, the specific steps of performing data preprocessing on the log information are: S101, collect sample data S = {e (1) ,e (2) ,……,e (l) }, where e (l) represents each log, l represents the number of logs; S102, use a sliding window of size n to perform sliding processing with a step size of 1 on the initial sequence data S to collect data, and obtain a context combination set C = {c1, c2, ..., c i }, i represents the number of context combinations, c i represents a log sequence of length n {e (i) ,e (i+1) ,……,e (i+n) }; In step S2, the neural network model includes a generator model and a discriminator model, and the generator model and the discriminator model are trained using an alternating update training mechanism. The specific training method is: Step S201, inputting the log context combination set C constructed in step S1 into a generator to generate a subsequent subsequence event vector; Step S202, inputting the subsequent subsequence event and the real event into the discriminator model respectively, and the discriminator model determines whether the subsequent subsequence event is a real event; Step S203: During the discriminator model's discriminative process, the loss function L in the discriminator model D The parameters of the discriminator model are continuously updated until the discriminator model can determine whether the subsequent subsequence events are real events. The training of the discriminator model is completed and the training of the generator model is carried out. Step S204: The subsequent subsequence events generated by the generator model are discriminated by the trained discriminator model, and the discriminator model inputs the discrimination results into the generator model; Step S205: The generator model is based on the loss function L G The resulting model error optimizes the subsequent subsequence events generated by the generator model; Among them, θ represents the parameters of the generator model, is the vector representation of the subsequent subsequence events generated by the generator model, c is the vector representation of the log context set input to the generator model, and e is the vector representation of the real event; Step S206: Input the subsequent subsequence event vector and the real event vector generated by the generator model into the discriminator model for discrimination. When the discriminator model cannot distinguish whether the input subsequence event is a subsequent subsequence event generated by the generator model or a real event, the training of the generator model ends, and returns to step S201 to continue training the discriminator model. Step S207: When the set number of training cycles is reached, the generator model and the discriminator model reach a balance, a mature generator model is obtained, and the training ends.
2. According to the large-scale system log anomaly detection method based on attention mechanism according to claim 1, it is characterized by: In step S203, the loss function L of the discriminator model is D =min φ -(logD(e|c)+log(1-D(e|c))), where φ represents the parameters of the discriminator model, D represents the discriminator model, e is the vector representation of the real event, and c is the vector representation of the log context combination set input to the generator model.
3. According to the large-scale system log anomaly detection method based on attention mechanism according to claim 1, it is characterized by: In step S205, the loss function of the generator model is Among them, θ represents the parameters of the generator model, A vector representation of the subsequent subsequence events produced by the generator model.
4. According to claim 3, a large-scale system log anomaly detection method based on attention mechanism is characterized in that: The generator model consists of three layers of neural networks, namely: long short-term memory neural network layer, attention neural network layer and fully connected network layer.
5. According to claim 4, a large-scale system log anomaly detection method based on attention mechanism is characterized in that: The discriminator model consists of two layers of neural networks, namely: long short-term memory neural network layer and fully connected network layer.
