Balance clustering compression method based on data similarity

Abstract

The invention discloses a cluster compression method based on data similarity. By analyzing file data, structural characteristic vector of characteristic fingerprint is extracted from files to calculate the data similarity; files are input in cluster by utilizing a graph partitioning method with a restricted condition, so that a plurality of categories in even sizes are formed; and finally, compression is respectively performed on each category by utilizing compression methods, such as BMCOM, so as to remove the redundant data in category interior. The invention adopts a clustering method based on data sampling; and key data with a high condensability serves as sample data. Firstly, clustering is performed on the sample data; then, the remaining data is classified through a marriage stabilizing method, thereby improving clustering efficiency under a condition that the compressing effect is not reduced. As a compressing and filing method, the invention can be applied to a distributed storage system, so that the problem of uneven data dependence and load in the prior method can be solved.

Description

Technical field [0001] The invention relates to the fields of data compression, distributed storage and archiving and data mining, in particular to a balanced clustering compression method based on data similarity. Background technique [0002] With the explosive growth of the total amount of information, massive distributed storage systems have become the core facilities of current Internet applications. The performance of distributed storage systems directly determines the performance of the entire information system. In a distributed storage system, except for a small part of hot data, a large part of the data is rarely accessed at all, but it takes up a lot of storage space and system resources. Therefore, compressing and archiving such data can not degrade the user experience Under the premise of reducing the occupation of system resources and saving costs. [0003] The general compression method only compresses a single file, and cannot make use of the data redundancy betwe...

AI Technical Summary

Problems solved by technology

[0003] The general compression method only compresses a single file and cannot take advantage of the data redundancy between files, so the compression ratio is very limited

In addition, although various methods proposed by the academic community can utilize data redundancy between files, the amount of calculation is too large and the execution efficiency is insufficient. Moreover, these methods rarely consider the storage of compressed data, and have not been developed for massive distributed storage systems. Optimization, it is easy to cause dependencies between data and uneven system load

Images