Concept drift detection method based on classifier diversity and Mcdiarmid inequality

Abstract

The invention discloses a concept drift detection method based on classifier diversity and a Mcdiarmid inequality. The objective of the invention is to detect whether conceptual drift occurs in a datastream by combining the inconsistency of a plurality of classifiers and the Mcdiarmid inequality. The method comprises the following steps: 1, incrementally training two individual classifiers with relatively large divergence, monitoring the diversity of the pair of classifiers for a newly coming data stream, and calculating the difference measurement of prediction results between the classifiers; 2, setting the size of the sliding window h to be n, and if the sliding window h is not full, automatically adding the difference measurement result of the latest data stream into the sliding windowh; and if the sliding window h is full, moving the initial difference measurement result out of the sliding window, and adding the latest result; 3, giving a confidence coefficient, and solving a threshold value for judging drifting through the confidence coefficient and an Mcdiarmid inequality theory. 4, associating each element in the sliding window with a weight, calculating the difference value between the weighted average value of the sliding window at the current moment and the maximum weighted average value observed at present, and comparing the difference value with the previously obtained threshold value to judge whether drifting occurs or not, so that concept drifting can be effectively detected, a classifier is updated, and better classification performance and generalization ability are shown.

Description

technical field [0001] The invention relates to the technical field of data stream processing, in particular to a concept drift detection method based on classifier diversity and Mcdiarmid inequality. Background technique [0002] With the rapid development of information technology, new businesses such as web browsing, online shopping, and social networking continue to emerge, resulting in explosive growth of data. As a new data type, data flow has the characteristics of high dimensionality, high speed, dynamicity and continuity compared with traditional data. This makes traditional classification methods face severe challenges. Moreover, the data flow will inevitably lead to the concept drift problem in the rapidly changing real environment. [0003] Concept drift refers to the target concept, that is, the phenomenon that the statistical characteristics of the target variable change in an unpredictable way as the environment changes. After concept drift occurs, the pred...

AI Technical Summary

Problems solved by technology

Therefore, continuing to use the previous classifier to classify new samples will lead to a sharp decline in the recognition ability of the classification model

[0004] Due to the concept drift problem caused by the changing environment, the adaptability of the classifier to the new incoming data flow becomes poor, and it is difficult to maintain a good and stable performance.

Images