Template-oriented Word2vec-based log exception detection method and device

Abstract

The invention discloses a template-oriented Word2vec-based log anomaly detection method and device, and the method comprises the following steps: carrying out the preprocessing of an original log, obtaining a log template, and carrying out the segmentation of the log template, so as to obtain a log sequence; solving a feature vector of the log template based on Word2vec, wherein the ID serial number of the log template is used as the input of the Word2vec; solving a feature vector of the log sequence according to the feature vector of the log template; and performing machine learning on the feature vector of the log sequence to obtain an anomaly detection model, and performing detection according to the anomaly detection model. Starting from a Word2vec processing object as a template, thescale of training data can be reduced. Moreover, the original log is preprocessed, and the time consumed by log anomaly detection is reduced through preprocessing so as to avoid affecting the final anomaly detection result.

Description

technical field [0001] The invention relates to the technical field of log anomaly detection, in particular to a template-oriented and Word2vec-based log anomaly detection method and device. Background technique [0002] At present, the word is used as the log sequence of the processing object of Word2vec (a language representation model that generates word vectors in natural language processing) (expressed as log events generated by the system in chronological order within a period of time, and the original log is divided by the window And get) the steps of anomaly detection are as follows: First, the original log is used as input, and Word2vec is used to map each word in the original log to the vector space, so that each word has its corresponding coordinates, and then the log event (reflecting the system The coordinates of the operation message) are represented by the centroid of all word coordinates in the event, and the log sequence is represented by the centroid of all...

AI Technical Summary

Problems solved by technology

Therefore, there will be deviations in the representation of the log sequence feature vector of this method, which will affect the final result of anomaly detection; secondly, due to the large scale of the system log sequence, it needs to be performed for each word when training, so the computational complexity is relatively high. high

[0005] (2) The log sequence anomaly detection of words as the processing object of Word2vec, which directly takes the original log as input without preprocessing the original log

The disadvantages of directly using the original log as input are: firstly, when some data in the original log is lost, some log messages are incomplete and cannot fully reflect the content expressed by the event; secondly, there is some redundant information in the original log. Take the data set as an example. Each log message includes timestamp, date, node, time, repeated node, message type, component (where the message is generated), message level, statement content, etc. These incomplete log messages and redundant Information will affect the results of log anomaly detection

Images