Rapid log anomaly detection method and device based on low-dimensional multi-level features

Abstract

The invention discloses a quick log anomaly detection method and device based on low-dimensional multi-level features. The method comprises a log feature extraction step and a log anomaly detection step. In the log feature extraction step, a natural language processing technology is utilized to convert an unstructured log stream into a structured log, feature extraction is carried out, a dimension reduction technology is utilized to reduce feature dimensions, and the detection speed is improved; in the log anomaly detection step, anomaly detection is carried out according to the extracted features, wherein a detection model is obtained through online training according to the current to-be-detected log flow and an anomaly detection algorithm; according to the method, anomaly detection is carried out on the log stream, a periodic retraining strategy is designed to overcome the concept drift problem in log anomaly detection, the detection rate and the detection speed are improved by utilizing low-dimensional multi-level features, detection of network faults and network attacks is realized, and the security of a network environment is improved.

Description

technical field [0001] The present invention relates to the technical field of computer network security, in particular to a method and device for fast log anomaly detection based on low-dimensional and multi-level features. Background technique [0002] With the development of 5G technology, modern communication networks are becoming more and more complex, and network devices generate massive amounts of log data every day. These logs can be used to analyze the operating status of devices and realize fault detection or prediction. However, due to the large number of logs and inconsistent log formats, the cost of rule matching or manual detection methods continues to increase, and the efficiency gradually decreases. However, the existing log detection algorithms use high-dimensional log features for detection, which will make the unsupervised model difficult to converge and increase the detection time. In addition, these models often suffer from concept drift problems that l...

AI Technical Summary

Problems solved by technology

However, due to the large number of logs and inconsistent log formats, the cost of the method of rule matching or manual detection is increasing, and the efficiency is gradually decreasing.

However, the existing log detection algorithm uses high-dimensional log features for detection, which will make it difficult for the unsupervised model to converge and increase the detection time

In addition, these models often suffer from concept drift problems that cause model failure

Images