Anomaly detection method and system based on log information, and computer equipment

Abstract

The invention relates to an anomaly detection method and system based on log information, and computer device. The anomaly detection method based on the log information comprises the following steps: obtaining structured data: exporting a log, extracting attribute features of the log through a regular expression, and converting the attribute features into structured data; performing unsupervised detection model training: carrying out dimensionality reduction on the structured data, carrying out data clustering on the internal structure of the structured data by utilizing a clustering algorithm, and repeating the step to obtain an unsupervised recognition model; performing supervised detection model training: constructing time sequence feature data by using a timestamp according to the structured data, and training a supervised recognition model based on the time sequence feature data; and performing anomaly detection: importing a to-be-detected log into the unsupervised recognition model and the supervised recognition model, and performing anomaly detection. By using a supervised algorithm and an unsupervised algorithm, abnormal log recognition is carried out from different angles, and the log anomaly detection effect is greatly improved.

Description

technical field [0001] The present application relates to the technical field of machine learning, in particular to an anomaly detection method, system and computer equipment based on log information. Background technique [0002] Modern systems are moving to massive scale, scaling distributed systems across thousands of machines, scaling high-performance computing through these supercomputers. Most of these systems are designed to run 24 / 7, serving tens of millions of users around the world, so high availability and reliability are a must. Any incident on these systems, including service outages and degradations in service quality, can bring down applications and lead to huge revenue losses. Anomaly detection aims to detect abnormal system behavior in time and plays an important role in event management of large-scale systems. [0003] Log-based anomaly detection can reduce the workload of manual detection and reduce the subsequent impact of equipment anomalies. Existing...

AI Technical Summary

Problems solved by technology

[0004] The main defect of the method based on log information statistics is that the threshold judgment needs to accumulate a lot of rules. For log information, it has various forms and is constantly updated. In this case, a large number of rules need to be maintained to achieve better results. The workload of anomaly detection is huge and the cost is high

[0005] At present, no effective solution has been proposed for the balance between anomaly detection workload and anomaly detection results in related technologies.

Images