A method and system for automatic identification of system log anomalies
By combining supervised and unsupervised learning methods, fasttext and word2vec models were constructed, which solved the problem of low accuracy in system log anomaly identification and achieved high-precision anomaly identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING MECHANICAL EQUIP INST
- Filing Date
- 2021-10-29
- Publication Date
- 2026-04-17
AI Technical Summary
In existing technologies, the accuracy of system log anomaly identification is low, especially when there are few anomaly samples, making it difficult to build a high-precision classification model.
Supervised and unsupervised learning methods are employed to construct fasttext and word2vec models respectively. By constructing vector representations of log templates and mapping them to the classification space, massive amounts of normal sample information are fused to improve recognition accuracy.
By constructing a mapping relationship between the content space and the classification space and utilizing massive amounts of normal sample information, the accuracy of anomaly identification in system logs has been significantly improved.
Smart Images
Figure CN116089132B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of system log anomaly identification technology, and in particular to an automatic identification method and system for system log anomalies. Background Technology
[0002] System logs record all system actions and express them according to a certain standard. System logs play a vital role in areas such as security, information statistics, and debugging.
[0003] In real-world production environments, anomalies are relatively rare. Therefore, the identification of abnormal logs is mostly achieved through methods such as clustering and outlier detection. These algorithms are unsupervised and cannot fully utilize the massive amount of information contained in normal samples, resulting in low anomaly detection accuracy. To achieve higher classification accuracy, training a precise classification model often requires a large number of samples. Due to the scarcity of anomalous samples, how to build a high-precision classification model based on a small number of anomalous samples has always been a major challenge in industry. Summary of the Invention
[0004] Based on the above analysis, the present invention aims to provide an automatic identification method and system for system log anomalies, in order to solve the problem of low accuracy in existing log anomaly identification.
[0005] On one hand, embodiments of the present invention provide an automatic identification method for system log anomalies, including the following steps:
[0006] Extract historical system log records, obtain the log template corresponding to each log record, and generate a log template sequence; construct a supervised training sample set and an unsupervised training sample set based on the template sequence;
[0007] Construct a fasttext model, train the fasttext model based on a supervised training sample set, obtain the trained fasttext model, and at the same time obtain the vector representation of each log template in the supervised training sample set to form a classification space;
[0008] Construct a word2vec model, train the word2vec model based on the unsupervised training sample set, and obtain the vector representation of each log template in the unsupervised training sample set to form the content space;
[0009] Map the vector representation of each log template in the content space to the classification space to obtain the vector representation of each log template in the classification space;
[0010] Obtain the log records to be identified, generate a log template sequence based on the log template corresponding to each log record, and obtain the anomaly identification result of the log template sequence based on the vector representation of the log template in the classification space and the trained fasttext model.
[0011] The beneficial effects of the above technical solution are as follows: This invention uses both supervised and unsupervised learning methods to construct vector representations of partially overlapping samples in two spaces, and constructs a mapping relationship between the two spaces, thereby integrating massive amounts of normal sample information into the abnormal log identification process, which greatly improves the identification accuracy.
[0012] Further, a supervised training sample set and an unsupervised training sample set are constructed based on the template sequence, including:
[0013] Using the sliding window method, with w as the window size, the log template sequence within the window is extracted sequentially as samples, and the (w+1)th log record is used as the label corresponding to the sample to construct training samples.
[0014] M samples are randomly selected from the normal samples and merged with the abnormal samples to form a supervised training sample set;
[0015] All normal samples constitute the unsupervised training sample set.
[0016] The beneficial effects of the above technical solution are as follows: by adopting the sliding window method, the log template is regarded as a time series, and the log template sequence can be quickly extracted to construct a training sample set. The method is simple and easy to implement.
[0017] Furthermore, the vector representation of each log template in the content space is mapped to the classification space using a least squares regression model, deep neural network, SVM, or logistic regression.
[0018] Furthermore, the least squares regression model is expressed as follows: Where x represents the vector representation of the content space. This represents the vector representation of x after mapping to the classification space, where the mapping parameter θ = (x... s T x s ) -1 x s T y s x s and y s The vector representations of the log templates that overlap in the supervised and unsupervised training sample sets are shown in the classification space and content space, respectively.
[0019] The beneficial effects of the above technical solution are as follows:
[0020] By constructing a model of the mapping relationship between a small number of overlapping samples in two representation spaces, it is possible to construct the relationship between all samples in the two spaces and realize the mutual conversion between feature representations.
[0021] By employing a least squares regression model to map the vector representations of log templates in all unsupervised training sets to the classification space, the vector representations of normal sample log templates in the classification space are obtained. The obtained vector representations have better expressive power and can distinguish well from abnormal samples in the classification space, thereby improving the accuracy of log anomaly identification.
[0022] Furthermore, the word2vec model adopts a single hidden layer neural network model with an embedding dimension of 100, a sliding window size of 5, and the optimization objective is to minimize cross-entropy. The CBOW algorithm is used to train the word2vec model.
[0023] The beneficial effects of the above technical solution are as follows: By adopting a single hidden layer neural network model as the network structure of the word2vec model, the structure is simple and easy to implement. The word2vec model is trained by the CBOW algorithm, thereby quickly predicting the log template through the context template of the log template.
[0024] This section provides the model parameters for the training content space. Based on these parameters, the wor2vec model can be trained. Once training is complete, the vector representation of each input vector in the content space can be obtained.
[0025] Furthermore, the fasttext model adopts a single hidden layer neural network model with an embedding dimension of 100, a sliding window size of 5, and the optimization objective is to minimize cross-entropy.
[0026] The beneficial effects of the above technical solution are as follows: By using a single hidden layer neural network model as the network structure of the fasttext model, the structure is simple and easy to implement. Based on these parameters, the fasttext model can be trained, thereby obtaining the vector representation of the classification space and the trained classification model.
[0027] On the other hand, embodiments of the present invention provide an automatic identification system for system log anomalies, including the following modules:
[0028] The training sample construction module is used to extract historical system log records, obtain the log template corresponding to each log record, and generate a log template sequence; based on the template sequence, a supervised training sample set and an unsupervised training sample set are constructed.
[0029] The classification space construction module is used to construct the fasttext model. The fasttext model is trained based on the supervised training sample set to obtain the trained fasttext model. At the same time, the vector representation of each log template in the supervised training sample set is obtained to form the classification space.
[0030] The content space construction module is used to construct the word2vec model. The word2vec model is trained based on the unsupervised training sample set to obtain the vector representation of each log template in the unsupervised training sample set, which constitutes the content space.
[0031] The mapping module is used to map the vector representation of each log template in the content space to the classification space, so as to obtain the vector representation of each log template in the classification space.
[0032] The log anomaly detection module is used to acquire log records to be identified, generate a log template sequence based on the log template corresponding to each log record, and obtain the anomaly detection result of the log template sequence based on the vector representation of the log template in the classification space and the trained FastText model.
[0033] Furthermore, the training sample construction module is used to construct a supervised training sample set and an unsupervised training sample set based on the template sequence, including:
[0034] Using the sliding window method, with w as the window size, the log template sequence within the window is extracted sequentially as samples, and whether the (w+1)th log record is abnormal is used as the label corresponding to the sample to construct training samples.
[0035] M samples are randomly selected from the normal samples and merged with the abnormal samples to form a supervised training sample set;
[0036] All normal samples constitute the unsupervised training sample set.
[0037] Furthermore, the mapping module is used to map the vector representation of each log template in the content space to the classification space using a least squares regression model, a deep neural network, an SVM, or logistic regression.
[0038] Furthermore, the least squares regression model is expressed as follows: Where x represents the vector representation of the content space. This represents the vector representation of x after mapping to the classification space, where the mapping parameter θ = (x... s T x s ) -1 x s T y s x s and y sThe vector representations of the log templates that overlap in the supervised and unsupervised training sample sets are shown in the classification space and content space, respectively.
[0039] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description
[0040] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.
[0041] Figure 1 This is a flowchart of the automatic identification method for system log anomalies according to an embodiment of the present invention;
[0042] Figure 2 This is a structural block diagram of the automatic identification system for system log anomalies according to an embodiment of the present invention. Detailed Implementation
[0043] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.
[0044] A specific embodiment of the present invention discloses an automatic identification method for system log anomalies, such as... Figure 1 As shown, it includes the following steps:
[0045] S1. Extract historical system log records, obtain the log template corresponding to each log record, and generate a log template sequence; construct a supervised training sample set and an unsupervised training sample set based on the template sequence.
[0046] The massive amounts of log data generated by a system are expanded from a few log templates. The detailed information in the logs generally consists of two parts: template terms and parameter terms. The content of the template terms does not change with the running state at the time of output, while the parameter terms change dynamically according to the program's running state at the time of output. Log templates can be extracted using methods such as tag recognition trees. In step S1, historical system log records are extracted, and the log template corresponding to each log record is matched based on the obtained log templates, thus forming a log template sequence.
[0047] Specifically, a supervised training sample set and an unsupervised training sample set are constructed based on the template sequence, including:
[0048] S11. Using the sliding window method, with w as the window size, the log template sequence within the window is extracted sequentially as samples, and the label corresponding to the sample is whether the (w+1)th log record is abnormal, thus constructing training samples.
[0049] For example, the template sequence is sequentially divided into time windows according to length w. The log template sequence {log1, log2, ..., logw} within the time window w is used as a sample, and the label of the (w+1)th log record is determined by whether it is abnormal. For instance, if the (w+1)th log record is normal, the label is 0; if it is abnormal, the label is 1. Multiple log template sequences and their corresponding labels are extracted sequentially using this method to construct a training sample set.
[0050] S12. Randomly extract M samples from the normal samples and merge them with the abnormal samples to form a supervised training sample set.
[0051] Specifically, since the number of abnormal samples is relatively small compared to the number of normal samples, directly classifying and training all normal and abnormal samples together would cause the abnormal samples to be easily overwhelmed by the normal samples, resulting in inaccurate training results. Therefore, M samples are randomly selected from the normal samples to form a supervised training sample set with the abnormal samples. In practice, M can preferably be 2 to 3 times the number of abnormal samples.
[0052] S13. All normal samples constitute the unsupervised training sample set.
[0053] All normal samples constitute an unsupervised (also known as self-supervised training) training sample set, which is used to predict template content based on the context of the template, thereby constructing a content space.
[0054] S2. Construct a fasttext model. Train the fasttext model based on the supervised training sample set to obtain the trained fasttext model. At the same time, obtain the vector representation of each log template in the supervised training sample set to form a classification space.
[0055] FastText can be used for both unsupervised training to learn word vectors and supervised learning for text classification. Therefore, this application uses FastText for supervised training and classification learning. In this application, unsupervised training samples are treated as sentences in text classification training, and log templates in the samples are considered as words in those sentences.
[0056] FastText maps each word in a sentence to a word vector through a lookup layer. These word vectors are then fed into a hidden layer, where they are summed and averaged to form the sentence's vector. A linear classifier is then used for classification. FastText lacks non-linear hidden layers, resulting in a simpler structure and faster training. Training with FastText not only yields a model for text classification but also additional results: the word vectors for each word in the text.
[0057] A supervised training sample set was constructed to train the fasttext model. Specifically, the fasttext model uses a single-hidden-layer neural network with an embedding dimension of 100, a sliding window size of 5, and the optimization objective is to minimize cross-entropy. In practice, the number of hidden layers, embedding dimension, and sliding window size can be modified according to actual training needs.
[0058] By training the fasttext model using a constructed supervised training sample set, we obtain not only a fasttext classification model for log anomaly detection, but also a vector representation of each log template in the supervised training sample set. The set of vector representations corresponding to each log template in the supervised training sample set constitutes the classification space.
[0059] S3. Construct a word2vec model. Train the word2vec model based on the unsupervised training sample set to obtain the vector representation of each log template in the unsupervised training sample set, which constitutes the content space.
[0060] Word2vec is a popular unsupervised training method that represents words as vectors. It includes two different algorithmic modes: CBOW and skip-gram. The CBOW algorithm uses the context of a word as input to predict the word itself. Skip-gram, on the other hand, uses a word as input to predict its surrounding context. This application adopts the CBOW algorithm.
[0061] An unsupervised training sample set was constructed and used to train the word2vec model. Specifically, the word2vec model adopted a single hidden layer neural network model with an embedding dimension of 100, a sliding window size of 5, and the optimization objective was to minimize cross-entropy. The CBOW algorithm was used to train the word2vec model. In implementation, the number of hidden layers, embedding dimension, and sliding window size of the neural network can be modified according to actual training needs.
[0062] This application treats unsupervised training samples as sentences used for word vector training, and log templates in the samples as words in those sentences. By employing word2vec, the vector representation corresponding to each word (i.e., each log template in the unsupervised training sample set in this application) can be obtained, i.e., word vector. The set of vector representations corresponding to each log template in the unsupervised training sample set constitutes the content space.
[0063] S4. Map the vector representation of each log template in the content space to the classification space to obtain the vector representation of each log template in the classification space.
[0064] In order to make full use of the massive amount of normal sample information, the vector representation of the content space is mapped to the classification space, thereby overcoming the problem of model inaccuracy caused by insufficient abnormal samples and greatly improving the recognition accuracy.
[0065] Specifically, the vector representation of each log template in the content space can be mapped to the classification space using a least squares regression model, deep neural network, SVM, or logistic regression.
[0066] Preferably, the least squares regression model is expressed as follows: Where x represents the vector representation of the content space. This represents the vector representation of x after mapping to the classification space, where θ represents the mapping parameters, and θ = (x... s T x s ) -1 x s T y s x s and y s The vector representations of the log templates that overlap in the supervised and unsupervised training sample sets are shown in the classification space and content space, respectively.
[0067] A least squares regression model is used to map the vector representations of log templates in all unsupervised training sets to the classification space, obtaining the vector representations of normal log templates in the classification space. The obtained vector representations have better expressive power and can distinguish them well from abnormal samples in the classification space. Therefore, abnormal logs can be identified online using the pre-trained FastText model.
[0068] S5. Obtain the log records to be identified, generate a log template sequence based on the log template corresponding to each log record, and obtain the anomaly identification result of the log template sequence based on the vector representation of the log template in the classification space and the trained fasttext model.
[0069] The specific identification method is as follows: obtain the current log record to be identified, generate a log template sequence according to the log template corresponding to each log record, and obtain the anomaly identification result of the template sequence based on the vector representation of the log template in the classification space and the trained fasttext model, thereby predicting whether an abnormal log has occurred.
[0070] Since the vector representation of each log template in the classification space has already been obtained, there is no need for the FastText lookup layer to map the log template into a vector. Instead, the lookup layer queries the vector representation of the log template in the classification space, inputs it into the hidden layer, and the classification result is obtained after processing by the hidden layer.
[0071] This invention addresses the practical needs of anomaly log identification, which is hampered by a scarcity of anomalous samples, a massive amount of normal samples, and the difficulty in achieving sufficient algorithm accuracy for real-world applications. By combining the content representation space generated through self-supervised learning with the classification space formed by training on a small number of samples, and constructing a mapping relationship between the two spaces, the invention obtains a representation of massive amounts of data in the classification space. This allows for direct classification using the trained classification model, significantly improving the accuracy of anomaly classification and solving the problem of anomaly identification in industry.
[0072] One specific embodiment of the present invention discloses an automatic identification system for system log anomalies, such as... Figure 2 As shown, it includes the following modules:
[0073] The training sample construction module is used to extract historical system log records, obtain the log template corresponding to each log record, and generate a log template sequence; based on the template sequence, a supervised training sample set and an unsupervised training sample set are constructed.
[0074] The classification space construction module is used to construct the fasttext model. The fasttext model is trained based on the supervised training sample set to obtain the trained fasttext model. At the same time, the vector representation of each log template in the supervised training sample set is obtained to form the classification space.
[0075] The content space construction module is used to construct the word2vec model. The word2vec model is trained based on the unsupervised training sample set to obtain the vector representation of each log template in the unsupervised training sample set, which constitutes the content space.
[0076] The mapping module is used to map the vector representation of each log template in the content space to the classification space, so as to obtain the vector representation of each log template in the classification space.
[0077] The log anomaly detection module is used to acquire log records to be identified, generate a log template sequence based on the log template corresponding to each log record, and obtain the anomaly detection result of the log template sequence based on the vector representation of the log template in the classification space and the trained FastText model.
[0078] Preferably, the training sample construction module is used to construct a supervised training sample set and an unsupervised training sample set based on the template sequence, including:
[0079] Using the sliding window method, with w as the window size, the log template sequence within the window is extracted sequentially as samples, and whether the (w+1)th log record is abnormal is used as the label corresponding to the sample to construct training samples.
[0080] M samples are randomly selected from the normal samples and merged with the abnormal samples to form a supervised training sample set;
[0081] All normal samples constitute the unsupervised training sample set.
[0082] Preferably, the mapping module is used to map the vector representation of each log template in the content space to the classification space using a least squares regression model, a deep neural network, an SVM, or logistic regression.
[0083] Preferably, the least squares regression model is expressed as follows: Where x represents the vector representation of the content space. This represents the vector representation of x after mapping to the classification space, where the mapping parameter θ = (x... s T x s ) -1 x s T y s x s and y s The vector representations of the log templates that overlap in the supervised and unsupervised training sample sets are shown in the classification space and content space, respectively.
[0084] The above-described method and system embodiments are based on the same principles, and their related aspects can be referenced from each other to achieve the same technical effects. For specific implementation details, please refer to the method embodiments; they will not be repeated here.
[0085] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0086] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.
Claims
1. A method for automatic identification of system log anomalies, characterized in that, Includes the following steps: Extract historical system log records, obtain the log template corresponding to each log record, and generate a log template sequence; Based on the template sequence, a supervised training sample set and an unsupervised training sample set are constructed. Construct a fasttext model, train the fasttext model based on a supervised training sample set, obtain the trained fasttext model, and at the same time obtain the vector representation of each log template in the supervised training sample set to form a classification space; Construct a word2vec model, train the word2vec model based on the unsupervised training sample set, and obtain the vector representation of each log template in the unsupervised training sample set to form the content space; Map the vector representation of each log template in the content space to the classification space to obtain the vector representation of each log template in the classification space; Obtain the log records to be identified, generate a log template sequence based on the log template corresponding to each log record, and obtain the anomaly identification result of the log template sequence based on the vector representation of the log template in the classification space and the trained fasttext model; The vector representation of each log template in the content space is mapped to the classification space using a least squares regression model, deep neural network, SVM, or logistic regression. The word2vec model adopts a single hidden layer neural network model with an embedding dimension of 100 and a sliding window size of 5. The optimization objective is to minimize the cross entropy, and the CBOW algorithm is used to train the word2vec model. The FastText model uses a single hidden layer neural network with an embedding dimension of 100, a sliding window size of 5, and the optimization objective is to minimize cross-entropy.
2. The automatic identification method for system log anomalies according to claim 1, characterized in that, Based on the template sequence, a supervised training sample set and an unsupervised training sample set are constructed, including: Using the sliding window method, with w as the window size, the log template sequence within the window is extracted sequentially as samples, and whether the (w+1)th log record is abnormal is used as the label corresponding to the sample to construct training samples. M samples are randomly selected from the normal samples and merged with the abnormal samples to form a supervised training sample set; All normal samples constitute the unsupervised training sample set.
3. The automatic identification method for system log anomalies according to claim 1, characterized in that, The least squares regression model is expressed as follows: , where x represents the vector representation of the content space. This represents the vector representation of x after mapping it to the classification space, with mapping parameters. , and The vector representations of the log templates that overlap in the supervised and unsupervised training sample sets are shown in the classification space and content space, respectively.
4. A system for automatic identification of system log anomalies, characterized by Includes the following modules: The training sample construction module is used to extract historical system log records, obtain the log template corresponding to each log record, and generate a log template sequence. Based on the template sequence, a supervised training sample set and an unsupervised training sample set are constructed. The classification space construction module is used to construct the fasttext model. The fasttext model is trained based on the supervised training sample set to obtain the trained fasttext model. At the same time, the vector representation of each log template in the supervised training sample set is obtained to form the classification space. The content space construction module is used to construct the word2vec model. The word2vec model is trained based on the unsupervised training sample set to obtain the vector representation of each log template in the unsupervised training sample set, which constitutes the content space. The mapping module is used to map the vector representation of each log template in the content space to the classification space, so as to obtain the vector representation of each log template in the classification space. The log anomaly detection module is used to acquire log records to be identified, generate a log template sequence based on the log template corresponding to each log record, and obtain the anomaly detection result of the log template sequence based on the vector representation of the log template in the classification space and the trained fasttext model. The vector representation of each log template in the content space is mapped to the classification space using a least squares regression model, deep neural network, SVM, or logistic regression. The word2vec model adopts a single hidden layer neural network model with an embedding dimension of 100 and a sliding window size of 5. The optimization objective is to minimize the cross entropy, and the CBOW algorithm is used to train the word2vec model. The FastText model uses a single hidden layer neural network with an embedding dimension of 100, a sliding window size of 5, and the optimization objective is to minimize cross-entropy.
5. The system for automatic identification of system log anomalies according to claim 4, characterized in that, The training sample construction module is used to construct supervised training sample sets and unsupervised training sample sets based on the template sequence, including: Using the sliding window method, with w as the window size, the log template sequence within the window is extracted sequentially as samples, and whether the (w+1)th log record is abnormal is used as the label corresponding to the sample to construct training samples. M samples are randomly selected from the normal samples and merged with the abnormal samples to form a supervised training sample set; All normal samples constitute the unsupervised training sample set.
6. The automatic identification system for system log anomalies according to claim 4, characterized in that, The least squares regression model is expressed as follows: , where x represents the vector representation of the content space. This represents the vector representation of x after mapping it to the classification space, with mapping parameters. , and The vector representations of the log templates that overlap in the supervised and unsupervised training sample sets are shown in the classification space and content space, respectively.
Citation Information
Patent Citations
Log anomaly detection method and device, training method and device and electronic equipment
CN113468035A
Method for classifying log files associated with a system
US20200349112A1