System anomaly detection method and system based on depth log sequence analysis

Abstract

The invention provides a system anomaly detection method and system based on deep log sequence analysis. The method comprises the following steps: applying a sequence labeling model Bi-LSTM-CRF to logpath anomaly detection; applying normal distribution to log parameter anomaly detection, so that the BiLCN can automatically learn normal log modes including log execution paths and log event parameters, and log events deviating from a normal model can be accurately detected and marked as anomalies. Meanwhile, the system further comprises a log analyzer, a feature extractor and a log path flow model, the log path flow model is constructed through the detected log sequence, abnormal conditions are fed back to the user so that the user can diagnose the system in time, and experiments prove thatthe method has high accuracy and execution efficiency.

Description

technical field [0001] The disclosure belongs to the technical field of large-scale system log anomaly detection, and in particular relates to a system anomaly detection method and system based on deep log sequence analysis. Background technique [0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art. [0003] System anomaly detection is an essential and important task to detect system failures, debug machine performance, and maintain system security. more and more; thus, anomaly detection is also facing more and more challenges. System log files can be extracted from all computer systems, and record the operating status and events of the system at different times. They are important data for fault analysis, performance detection, and system anomaly detection. [0004] Due to the diversification of logs due to system differences and other reasons, it is a great challenge to ...

AI Technical Summary

Problems solved by technology

[0004] Due to the diversification of logs due to system differences and other reasons, it is a great challenge to detect anomalies in the system; in order to overcome this problem, many researchers have designed different log mining tools according to different systems, and used log mining method to detect anomalies in the system; although methods based on log mining have certain accuracy in detecting system anomalies, these methods are limited to specific scenarios, and a lot of professional knowledge is required in the process of log mining

Therefore, based on traditional data mining methods, they cannot play a better role in anomaly detection.

[0005] The inventors found that in the prior art, the method for anomaly detection of system logs mainly uses two data processes: log parsing and feature extraction. The original log events are converted into time-series log sequences through the log The parameters in the log-like event are constructed as a parameter vector, and the knowledge of natural language processing is used to detect the anomaly of the system by incorporating the attention mechanism into the RNN language model. Although this method has a certain effect on the anomaly detection of the system, However, it cannot effectively solve the problem of anomaly detection between different systems.

Images