Large-scale anomaly detection system based on KPI clustering

Abstract

The invention belongs to the technical field of KPI anomaly detection, and particularly relates to a large-scale anomaly detection system based on KPI clustering. The system comprises the following modules: a preprocessing module, a baseline extraction module, a clustering module, a classification module and a log module; the preprocessing module uses a linear interpolation method to fill the percentage of KPI missing according to data points adjacent to KPI; the baseline extraction module removes extreme values which may be abnormal; the clustering module executes density-based clustering on a sampling KPI baseline based on shape similarity by using a density-based clustering method; the classification module calculates the distance between the new data and each previous centroid, sorts the distances, determines the minimum class as the classified class, and determines that the new time sequence curve does not belong to any previous class when the minimum distance is greater than a certain threshold value; and the log module records curves which do not belong to any category.

Description

technical field [0001] The invention belongs to the technical field of KPI abnormality detection, and in particular relates to a large-scale abnormality detection system based on KPI clustering. Background technique [0002] Internet-based service companies today monitor thousands to millions of KPIs across their applications and systems to maintain the reliability of their services. Anomalies on KPIs usually indicate potential failures of related applications, such as server failures, network overloads, external attacks, etc. Therefore, anomaly detection technology is widely used to detect abnormal events in time to reduce the losses caused by abnormal events. [0003] Why it is problematic or flawed: Most anomaly detection algorithms assume that each KPI requires a separate model. Therefore, large-scale anomaly detection on thousands to millions of KPIs is very challenging due to the huge overhead of model selection, parameter tuning, model training, or anomaly labeling....

AI Technical Summary

Problems solved by technology

[0004] Aiming at the above-mentioned problems that each KPI requires a separate model and holiday effects, the present invention provides a system that significantly reduces modeling costs and is highly efficient

Images