Abnormal data detection method based on active learning

Abstract

The invention relates to an abnormal data detection method based on active learning, and the method comprises the steps: selecting a statistical and similarity-based model and an axis parallel subspace division-based unsupervised model as a basic learner according to the comparative analysis of various basic learners; merging the data of the outlier scores judged by the base learners at the outliers and the normal boundaries, and presenting the merged data to human experts for labeling; and sampling and training a supervised binary classification model from the labeled data set and the data set generated by voting of each base learner, and applying the model to the full data set to obtain a final mining result. The invention discloses an abnormal data detection method based on active learning. An outlier integrated mining method OMAL based on active learning is provided in combination with active learning and model integration, a supervised binary classification model is trained in combination with learning results of a plurality of unsupervised base learners and human expert knowledge, and high accuracy is achieved while workload is reduced and expansibility is improved.

Description

technical field [0001] The invention relates to a method for detecting abnormal data, in particular to a method for detecting abnormal data based on active learning. Background technique [0002] An outlier is a significant deviation from the rest of the data in the dataset that raises suspicions that these data points were produced by a different mechanism. Outlier Detection, also known as Outlier Mining, has received extensive attention and research because of its broad application prospects in financial fraud, network intrusion, fault detection, bioinformatics and other fields. [0003] Outlier detection tasks usually lack available labeled data, and outlier data only account for a small part of the entire data set. Therefore, compared with other data mining tasks, outlier detection is more difficult. At present, the research on outlier detection can be mainly divided into the following categories: (1) detection methods based on probability and statistics, including dete...

AI Technical Summary

Problems solved by technology

[0004] Due to the complexity of the outlier detection task, there is no single algorithm suitable for all scenarios, so the researchers proposed a detection method based on model integration to reduce the risk brought by a single algorithm

Images