Model selection for cluster data analysis

Abstract

A model selection method is provided for choosing the number of clusters, or more generally the parameters of a clustering algorithm. The algorithm is based on comparing the similarity between pairs of clustering runs on sub-samples or other perturbations of the data. High pairwise similarities show that the clustering represents a stable pattern in the data. The method is applicable to any clustering algorithm, and can also detect lack of structure. We show results on artificial and real data using a hierarchical clustering algorithm.

Description

FIELD OF THE INVENTION The present invention relates to the use of learning machines to identify relevant patterns in datasets containing large quantities of diverse data, and more particularly to a method and system for unsupervised learning for determining an optimal number of data clusters into which data can be divided to best enable identification of relevant patterns. BACKGROUND OF THE INVENTION Knowledge discovery is the most desirable end product of data collection. Recent advancements in database technology have lead to an explosive growth in systems and methods for generating, collecting and storing vast amounts of data. While database technology enables efficient collection and storage of large data sets, the challenge of facilitating human comprehension of the information in this data is growing ever more difficult With many existing techniques the problem has become unapproachable. Thus, there remains a need for a new generation of automated knowledge discovery tools....

AI Technical Summary

Benefits of technology

In one embodiment, the inventive method provides means for extraction of information from gene expression profiles, in particular, extraction of the most relevant clusters of temporal expression profiles from tens of thousands of measured profiles. In one variation of the present embodiment, the clustering algorithm is the k-means algorithm. Other embodiments include all pairwise clustering methods (e.g., hierarchical clustering) and support vector clustering (i.e., a support vector machine (SVM)). In one variation, the fit involves affine transformations such as translation, rotation, and scale. Other transformations could be included, such as elastic transformations. In the case of k-means, the algorithm is applied in a hierarchical manner by sub-sampling the data and clustering the sub-samples. The resulting cluster centers for all the runs are then clustered again. The resulting cluster centers are considered to be the most significant profiles. To facilitate the clustering task, a ranking of the genes is first performed according to a given quality criterion combining saliency (significant difference in expression in the profile), smoothness, and reliability (low noise level). Other criteria for ranking include the local density of examples.

Problems solved by technology

Recent advancements in database technology have lead to an explosive growth in systems and methods for generating, collecting and storing vast amounts of data.

While database technology enables efficient collection and storage of large data sets, the challenge of facilitating human comprehension of the information in this data is growing ever more difficult With many existing techniques the problem has become unapproachable.

Currently, there are no methods, systems or devices for adequately analyzing the data generated by such biological investigations into the genome and proteome.

Clustering analysis is unsupervised learning—it is done without suggestion from an external supervisor; classes and training examples are not given a priori.

However, most clustering algorithms do not address this problem.

The question remains, however, of how to set the input parameter, or how to determine which level of the tree representation of the data to look at: Clustering algorithms are unsophisticated in that they provide no insight into the level of granularity at which the “meaningful” clusters might be found.

This is seen as the problem of finding the optimal number of clusters in the data, relative to some clustering algorithm.

Other model selection techniques have difficulty detecting the absence of structure in the data, i.e., that there is a single cluster.

Further, many of algorithms make assumptions as to cluster shape, and do not perform well on real data, where the cluster shape is generally not known.

Images