Method for classifying data streams under dynamic data environment

Abstract

The invention relates to the technical field of intelligent information processing and discloses a method for classifying data streams in a dynamic data environment. The method comprises the following steps: partitioning the data streams; establishing different classifiers for different concept drift; storing in a characteristic data pool of the classifiers; when a new data block arrives, judging whether the concept drift occurs or not by Kullback-Leibler (KL) divergence; if the concept drift does not occur, classifying by using the classifier at the last moment; if the concept drift occurs, seeking the proper classifiers from the characteristic data pool of the classifiers by the KL divergence and classifying; and if no coincident classifier exists, training a new classifier, adding the new classifier into the characteristic data pool of the classifiers and deleting the outdated classifiers. By the method, stable and mutational concept drift can be detected simultaneously; when the concept drift occurs, classification is performed by selecting the proper classifier to guarantee the efficiency of a model; and the performance of the model is guaranteed by deleting the outdated classifiers.

Description

technical field [0001] The invention relates to the technical field of intelligent information processing, in particular to a data stream classification method in a dynamic data environment, which is applicable to network intrusion detection, network security monitoring, sensor data monitoring, power grid power supply and the like. Background technique [0002] With the development of information technology, data stream, as a special data, has attracted more and more attention from the industry. Data stream refers to a huge sequence of data transmitted at high speed, and can only be read in a predetermined order. In real-world applications, since the data stream is usually transmitted at a very high speed, it will become very difficult to calculate and store the data stream data. Usually, there is only a chance to process the data when it first arrives, and it is difficult at other times. Then access the data. In addition, in the process of data flow generation, the data i...

AI Technical Summary

Problems solved by technology

The method of integrated learning can solve the problem that the historical information is not effectively used in the example-based method, but most of the existing methods adopt the blind learning method, which cannot effectively deal with the dynamic data phenomenon of sudden change, that is, the so-called concept drift of sudden change

Images