Distributed cross-dimensional abnormal data detection method in big data environment

Abstract

The invention provides a distributed interdimensional abnormal data detection method under big data environment. The method comprises the following steps of: data segmentation: segmenting an input data set on the basis of dimensionalities of the input data set so as to form a plurality of data buckets; forming a non-equilibrium binary coding tree according to the data bucket segmented from each dimensionality; distributing the plurality of data buckets segmented on the basis of the non-equilibrium binary coding tree onto each compute node; calculating and executing a relative outlier amount of each data point on each compute node; and screening the data points, the relative outlier amounts of which are greater than or equal to a set threshold value, so as to form an abnormal point set.

Description

technical field [0001] The invention relates to the technical field of big data processing, and in particular to a distributed cross-dimensional abnormal data detection method in a big data environment. Background technique [0002] With the continuous development of big data analysis and data mining technology, the volume of data is becoming larger and larger. In this large amount of data, the number of abnormal data also increases with the increase of data volume. Different from the conventional data in the data set, its characteristics deviate from the conventional data, so that the data analysis method will have obvious errors in the analysis results due to the existence of these data. Before traditional data mining activities, the data is carefully selected to ensure the integrity and consistency of the data. However, the massive data in the big data environment cannot be manually selected, so abnormal data detection plays a very important role. At the same time, abno...

AI Technical Summary

Problems solved by technology

However, these algorithms have the disadvantages of poor portability, inability to deal with abnormal data points of local data, cumbersome parameter setting, large influence of human factors, and inability to apply to multidimensional data sets.

However, the existing distributed abnormal data point detection technology has some problems in adapting to the heterogeneous distributed parallel computing environment. To a large extent, data segmentation is unreasonable, data allocation is not collapsed, disk I / O and The network I / O is high and the overhead is too large

Images