A method for generating KPI curves and labeling band features based on log event relationships

Abstract

The present invention discloses a method for generating KPI curves based on log event relationships and marking band characteristics. First, log KPI curves are generated according to the relationship of events in logs, and then the KPI curves are divided into several equal-length bands, which are aggregated according to the non-time dimension of the bands. Classify into multiple clusters, extract the fundamental wave of each cluster, compare the similarity between each band data of each cluster and the fundamental wave, find out the grouping boundary line of each cluster, group each band data of each cluster, and extract the continuous The total time length of similar bands, take the maximum value of the total time length as the sliding window width. This window is used to divide the KPI curve, so that the bands in each divided window are easy to cluster and classify, which is beneficial to quickly convert the entire KPI curve into a band chain composed of different types of bands, and then cycle the KPI curves of individual monitoring indicators Detection and type detection mark labels, and then use this window to segment a separate KPI curve, and use the bands in the fundamental KPI curve for grouping and labeling.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence, in particular to a method for generating KPI curves and marking band features based on log event relationships. Background technique [0002] Outlier detection (also known as outlier detection) is a detection process to find objects that behave differently than expected, and these objects are called outliers or outliers. Anomaly detection methods usually include statistical-based models, distance-based models, linear transformation models, nonlinear transformation models, and machine learning models. [0003] KPI (key performance indicators) refers to the monitoring indicators of services, systems and other objects (such as latency, throughput, etc. in the network). Its storage form is a sequence arranged in the order of time of its occurrence, which is what we usually call a time series. Anomaly detection of time series is to check whether the current data deviates significantl...

AI Technical Summary

Problems solved by technology

[0004] Real-time anomaly detection by setting thresholds on KPI data is very common, but real-time anomaly detection for system logs has not been publicly reported

[0005] In order to pursue effectiveness, traditional machine learning mostly adopts supervised learning methods. In practice, it is difficult to obtain abnormal labels in batches. The accuracy of model output is improved through massive labeled data samples. Therefore, a large number of business experts are required to manually label KPI curves, which often requires Repeated adjustments and corrections are time-consuming and labor-intensive. In practice, it may be necessary to start monitoring millions or tens of millions of KPIs at the same time. Therefore, it is often impossible to find a certain algorithm that can meet the above requirements at the same time in the actual anomaly detection practice. Unable to solve the above challenges at the same time; non-supervised learning commonly used clustering and other technologies are mainly used in feature discovery, data exploration and other scenarios, because of the lack of annotation, the results need to be interpreted by data scientists to be abstractly mapped to business models, and cannot be directly Functional results; in the specific implementation of weak supervision, because of the introduction of non-supervised / supervised methods in stages, the accuracy of circular recursion is too academic, and it is difficult to implement. On the other hand, in order to integrate specific methods, vector expressions need to be used to unify different The representation between methods, the result is not easy for the application personnel to understand

Images