Hierarchical clustering method based on Hadoop and HBase

Abstract

The invention discloses a hierarchical clustering method based on Hadoop and HBase. The method comprises the following steps: computing a distance matrix through Hadoop, and introducing a method for converting a result into Bulk Load for HFile file into the HBase, wherein the HBase is used for storing the distance matrix and mainly divided into two tables, one table is ordered according to cluster ID pair, and the other table is ordered according to the distance between the clusters, thereby conveniently taking out two nearest clusters to combine in each iteration; and finally realizing one multi-thread algorithm and combining with the cache technology to process the distance matrix in the HBase in an unifned manner to realize the hierarchical clustering algorithm and reserving multiple adjustable parameters. The algorithm simultaneously supports three clustering methods of single-linkage, complete-linkage and average-linkage. The scheme provided by the invention uses the parallel computation capacity of the Hadoop and the mass data storage capacity of the HBase, thereby improving the big data processing capacity and the expansibility of the hierarchical clustering algorithm.

Description

technical field [0001] The invention relates to the technical field of hierarchical clustering algorithm, Hadoop and HBase, in particular to the design and realization of a hierarchical clustering method based on Hadoop and HBase. Background technique [0002] As a simple and widely accepted clustering algorithm, hierarchical clustering algorithm has been applied in many fields, such as information retrieval and bioinformatics. The advantage of the hierarchical clustering algorithm is that it expresses the clustering results in a more detailed way. It organizes the clustering relationship between clusters into a dendrogram, and users can clearly know how each cluster is clustered together. Many other clustering algorithms do not give such results. Moreover, compared with other clustering algorithms such as k-means, the hierarchical clustering algorithm does not require the user to specify the number of clusters in advance. Although the hierarchical clustering algorithm has...

AI Technical Summary

Problems solved by technology

But currently there is no hierarchical clustering algorithm that can support single-linkage, complete-linkage and average-linkage multiple clustering methods on Hadoop and HBase platforms

Images