Multi-level anomaly detection method based on exponential smoothing and integrated learning model

Abstract

A multi-level anomaly detection method based on exponential smoothing, sliding window distribution statistics and an integrated learning model comprises the following steps of a statistic detection stage, an integrated learning training stage and an integrated learning classification stage, wherein in the statistic detection stage, a, a key feature set is determined according to the application scene; b, for discrete characteristics, a model is built through a sliding window distribution histogram, and a model is built through exponential smoothing for continuous characteristics; c, the observation features of all key features are input periodically; d, the process is ended. In the integrated learning training stage, a, a training data set is formed by marked normal and abnormal examples; b, a random forest classification model is trained. The method provides a general framework for anomaly detection problems comprising time sequence characteristics and complex behavior patterns and is suitable for online permanent detection, the random forest model is used in the integrated learning stage to achieve the advantages of parallelization and high generalization ability, and the method can be applied to multiple scenes like business violation detection in the telecom industry, credit card fraud detection in the financial industry and network attack detection.

Description

technical field [0001] The invention relates to a detection method for abnormal pattern modeling, especially including exponential smoothing and statistical detection technology suitable for resident rapid screening in massive data scenarios and integrated learning classification technology suitable for parallel detection. Background technique [0002] Abnormal patterns in big data can be intelligently discovered using machine learning and data mining techniques. Anomaly detection is also a hot issue with a wide range of practical scenarios in data mining, such as intrusion detection in the network environment, credit card fraud detection in the financial industry, business violation detection in the telecommunications industry, and new epidemic detection in the medical and health industry. Effective application of anomaly detection can recover high financial losses and even protect human lives for relevant organizations or individuals. [0003] Commonly used anomaly detect...

AI Technical Summary

Problems solved by technology

Among them, the statistical method relies on the idealized probability distribution assumption. Although the calculation speed is the fastest, in many cases, because the assumption cannot describe the essential structure of the application problem well, the accuracy is not high, especially the proportion of false alarms. too high

Anomaly detection algorithms based on classification, clustering, information theory, etc. can use more complex machine learning models, and the accuracy of prediction is guaranteed by the generalization ability of the model. However, the computational complexity of most model training processes is high and cannot be applied to online real-time detection

Images