Abnormal data detection method and device and data pre-processing method and system

Abstract

The invention proposes an abnormal data detection method and device and a data pre-processing method and system. The abnormal data detection method comprises the following steps of performing dimension reduction processing on a data set to be detected by a principal component algorithm to form a first data set; reconstructing the first data set by the principal component algorithm to form a second data set, wherein the second data set has the same dimension as the data set to be detected; calculating correlation between corresponding data of the data set to be detected and corresponding data of the second data set; and acquiring abnormal data, having big difference with the corresponding data in the second data set, from the data to be detected. According to the invention, a hypothesis that a data set to be analyzed conforms to certain specific distribution is not needed, and reliability, universality and stability are high.

Description

technical field

[0001] The invention relates to the field of computer technology, in particular to an abnormal data detection method and device, and a data preprocessing method and system. Background technique

[0002] In the fields of image processing systems, credit card fraud detection systems, and credit early warning systems, the detection of outliers is often involved. Outlier detection (also called outlier detection) is to find out that its behavior is very different from the expected object. A detection process of , these points that are different from the expected object are called outliers or outliers. The most common outlier detection is based on statistical methods, which can be divided into unary and multivariate cases according to the number of processing variables, for example:

[0003] 1) Univariate outlier detection method based on normal distribution

[0004] Suppose there are n sample points (x 1 , x 2 ,...,x n ), then the mean μ and variance σ of the...

Images