Abnormity detection method and device for parallelization of isolated forest algorithm based on Flink

Abstract

The invention discloses an abnormity detection method and a device for parallelization of an isolated forest algorithm based on Flink. The isolated forest algorithm is realized based on the real-timeprocessing platform Flink platform, the calculation speed of the isolated forest algorithm is greatly improved, and the problem that the exception detection is limited by the data volume when the isolated forest is used for exception detection in a stand-alone mode is solved. Transverse expansion is carried out by configuring and adding machines, and an abnormity detection task is also carried outin a real-time scene. The method is implemented by adopting the Flink, and abnormity detection is carried out on the data in real time by utilizing the real-time processing characteristic of the Flink. The parallelism degree of the algorithm can be transversely expanded by setting the parallelism degree of the operator of the Flink, so that the parallelism degree of the algorithm is not limited by the data volume when abnormity detection is carried out on mass data.

Description

technical field [0001] The invention relates to the field of anomaly detection calculations, in particular to an anomaly detection method and device based on Flink-based parallelization of the isolated forest algorithm. Background technique [0002] Anomaly detection usually refers to an unsupervised process of finding data that is far from the expected data. The detected data are usually called outliers or outliers. Anomaly detection has played an important role in many fields, such as: intrusion detection system, credit card anti-fraud, intelligent operation and maintenance, etc. [0003] Generally, there are several detection methods for anomaly detection: statistics-based methods, proximity-based methods, density-based methods, cluster-based methods, etc. However, they all have their own shortcomings. Statistics-based methods are more effective for single-dimensional anomaly detection, but for multiple dimensions, the detection effect and performance are relatively poo...

AI Technical Summary

Problems solved by technology

However, they all have their own shortcomings. Statistics-based methods are more effective for single-dimensional anomaly detection, but for multiple dimensions, the detection effect and performance are relatively poor; proximity-based methods cannot handle abnormalities with different densities. For regional data sets, because density is a local feature and distance is a global feature, it cannot take into account changes in local density; density-based methods overcome the problem of not being able to consider global density based on proximity, but they are not suitable for high-dimensional data. The computational complexity is also high; clustering-based methods need to select the appropriate number of clusters, the number of clusters has a great influence on the effect of anomaly detection, and it is difficult to choose it in actual use

Images