A cold and hot judgment method for mass data in a distributed storage system

Abstract

The invention discloses a cold and hot judgment method for mass data in a distributed storage system, which comprises the following steps of: counting the access frequency of data by adopting a multi-version hash table, and determining the cold and hot conditions of the data according to the access frequency. The method mainly comprises the steps of selecting a proper number of Hash functions, calculating corresponding Hash results of accessed indexes through the Hash functions in each time of data access, and then increasing values of the Hash results at corresponding positions in a Hash table of a current version; After multiple times of access, carrying out attenuation by switching the hash table of the current version and removing the first bits of all the hash tables of the version, so that the influence of old information is reduced; And finally, determining the cold and hot of the data by counting the data access information in all versions of hash tables. Compared with other methods in the field, the method has the advantages that higher cold and hot data judgment accuracy can be provided when the memory space with the same size is used, the cold and hot data are favorablyand respectively processed, and the system performance is improved.

Description

technical field [0001] The invention relates to the technical field of data storage, in particular to a method for judging the hotness and coldness of massive data in a distributed storage system. Background technique [0002] With the extensive development and application of technologies such as the Internet, cloud computing, and the Internet of Things, data has shown explosive growth, and massive data needs to be processed and stored all the time, which poses a huge challenge to the performance and reliability of the storage system. . Distributed storage solutions are commonly used for mass data storage. The more famous distributed storage systems include Google's GFS (Google File System), the open source version of GFS HDFS (Hadoop Distributed FileSystem), Microsoft's WAS (Windows Azure Storage) and Ceph et al. However, statistics show that most of the data in the distributed storage system is cold data, which is rarely accessed, and the access is mainly concentrated on...

AI Technical Summary

Problems solved by technology

However, with the increase of data size, it will be impossible to record massive data access information in memory

This method will occupy a large amount of computing resources and bring a large amount of storage overhead when processing massive data access. When the memory size is exceeded, it will cause excessive disk IO and seriously affect system performance.

Images