Methods and apparatus for data stream clustering for abnormality monitoring

Abstract

Techniques for monitoring abnormalities in a data stream are provided. A plurality of objects are received from the data stream and one or more clusters are created from these objects. At least a portion of the one or more clusters have statistical data of the respective cluster. It is determined from the statistical data whether one or more abnormalities exist in the data stream.

Description

FIELD OF THE INVENTION [0001] The present invention is related to techniques for clustering a data stream and, more particularly, techniques for monitoring data abnormalities in the stream through the clustering of the data stream. BACKGROUND OF THE INVENTION [0002] In general, large volumes of continuously evolving data, which may be stored, is referred to as a data stream. Data streams have received increased attention in recent years due to technological innovations, which have facilitated the creation, maintenance and storage of such data. A number of data mining studies have been conducted in the data stream context in recent years, see, e.g., C. C. Aggarwal, “A Framework for Diagnosing Changes in Evolving Data Streams,” ACM SIGMOD Conference, 2003; B. Babcock et al., “Models and Issues in Data Stream Systems,” ACM PODS Conference, 2002; P. Domingos et al., “Mining High-Speed Data Streams,” ACM SIGKDD Conference, 1998; S. Guha et al., “ROCK: A Robust Clustering Algorithm for Ca...

AI Technical Summary

Benefits of technology

[0009] Thus, a framework may be provided in which select statistical data may be stored at regular intervals. This results in a technique which is able to analyze different characteristics of the clusters in an effective manner. Advantageously, the inventive techniques may be useful for clustering different kinds of categorical data sets, and adapting to the rapidly evolving nature of a data stream.

[0010] Additional advantages of the inventive techniques of the present invention include the ability to explore the clusters in an online fashion, and store statistical data which may be utilized for a better understanding and analysis of the data stream. In applications in which the data stream evolves considerably, different kinds of clusters may assist in understanding the behavior of the data stream over different periods in time. This is advantageous since a fast data stream cannot be repeatedly processed in order to resolve different kinds of queries.

Problems solved by technology

However, these techniques cannot be utilized for clustering data streams, since they do not naturally scale well with increasing data size.

Clustering and outlier monitoring present a number of unique challenges in an evolving data stream environment.

In the data stream environment, outlier and abnormality monitoring is especially problematic, since the temporal component of the data stream influences whether an outlier is defined as an abnormality.

On the other hand, in many other cases, data points may not join the outlier or newly created cluster over time, thereby defining an abnormality.

Images