Double-window concept drift detection method based on sample distribution statistical test

Abstract

The invention discloses a double-window concept drift detection method based on sample distribution statistical test, and belongs to the field of machine learning. Aiming at the problem of concept drift of data streams along with time attributes, the method comprises the following steps: firstly, carrying out outlier detection in a fixed window by adopting support vector regression (SVR); then, for the detected outliers, calculating the Euclidean distance between new and old samples in a variable window, and according to the Euclidean distance, performing statistical analysis in combination with multiple distribution inspection methods to indirectly reflect whether data distribution is changed or not so as to determine whether drifting occurs or not; and finally, verifying the effectiveness of the method on a cement strength reference data set and an urban solid waste incineration (MSWI) outlet nitrogen oxide concentration data set.

Description

technical field [0001] The dual-window concept drift detection method based on the statistical test of sample distribution belongs to the field of machine learning. Background technique [0002] At present, the research work of machine learning mainly focuses on non-incremental batch learning. The learning method is to pack the collected data into data sets according to batches, and train the base learner intensively. With the massive growth of data, the use of traditional data sets to read and process data will increase the storage cost of data. At the same time, the way of centralized training makes data lag, which cannot reflect the current working conditions in a timely manner, nor can the data be updated at any time. Reasonable feedback on time changes. The online learning algorithm updates the learner based on a single sample or a batch of samples, and then expects to obtain a hypothesis based on all current samples, which is more suitable for practical problems. [...

AI Technical Summary

Problems solved by technology

[0018] To analyze whether the differences among the three test results are significant, the confidence levels of the three tests should be set in advance The confidence level represents the acceptable error range of the hypothesis test. When the confidence level is too small, the distribution test is sensitive to concept changes, and when there is a slight difference between samples, it is difficult to pass the test, resulting in drift and misjudgment, which increases the burden on the learner update; when the confidence level When it is too large, the distribution test is tolerant to concept changes, resulting in drift and missed judgments, and the prediction effect is reduced

Images