A low memory overhead hot spot data identification method for hybrid storage systems

Abstract

The invention discloses a low-memory-overhead hotspot data identification method oriented to a hybrid storage system. The specific steps are: 1) defining a Bloom filter UBF with a super large counting capacity to record the number of page visits; 2) when page x is accessed, query The popularity of page x; 3) If x has not been recorded in UBF, record x by flipping several bits corresponding to x in UBF from 0 to 1; if x has been recorded in UBF, record x by flipping a corresponding 0 The bit is flipped to 1 with a certain probability, thereby increasing the popularity of x; 4) Compare the popularity of page x with the specified popularity threshold, if the popularity of page x is greater than the popularity threshold, the data corresponding to page x is identified as hot data; jump Execute step 2). The invention has the advantages of low memory overhead, capable of realizing identification of hotspot data with a large amount of data, and high identification accuracy.

Description

technical field [0001] The invention relates to the technical field of large-scale hybrid storage, in particular to a method for identifying hotspot data with low memory overhead for a hybrid storage system. Background technique [0002] The current large-scale storage systems have high requirements on performance and cost, and the hybrid storage architecture is a solution that can meet these requirements at the same time. In a hybrid storage system, high-end devices with small capacity and high performance are used to ensure performance, while low-end devices with low performance and low prices are used to reduce costs. Therefore, in such a heterogeneous storage system, high-end devices are generally used to save frequent Accessed hot data, low-end devices are used to save infrequently accessed cold data, but to achieve this optimization measure depends to a large extent on the accurate identification of hot data. [0003] Hot data is generally identified through statistic...

AI Technical Summary

Problems solved by technology

[0005] 1) Bloom Filter cannot record repeated visits to data; no matter how many times page x is accessed, BloomFilter only records k bits h corresponding to x 1 (x), h 2 (x),...,h k (x) is set to 1, but no action is taken to record the number of times x is accessed

Bloom Filter cannot identify hotspot data from the visit history because there is no visit count information

[0006] 2) Bloom Filter can only continuously record new pages, but cannot delete any pages already recorded in Bloom Filter, resulting in longer and longer recorded access history; in fact, early access history has no effect on hotspot data identification Using value, recording too long access history will increase memory overhead

[0008] Although the MBFs method can use multiple Bloom Filters to record the number of page visits, and delete the early access history by periodically clearing a Bloom Filter, it can efficiently identify hot data from the access history; however, the MBFs method also has two problems: Defects in aspects: on the one hand, due to the use of multiple Bloom Filter data structures, the memory overhead of the MBFs method is still high; on the other hand, the range of page access times that MBFs can record is limited; assuming that the MBFs method maintains n Bloom Filters, for For a specific page, the maximum number of visits that this method can record for it is n; if the number of visits to this page exceeds n, the excess visit information cannot be recorded in the Bloom Filter, resulting in the loss of some hotspot information

The shortcomings of the MBFs method in the above two aspects cannot be compensated at the same time, that is, increasing the counting range (n) of the number of accesses will inevitably lead to an increase in memory overhead, so the MBFs method is also difficult to apply to large-scale hybrid storage systems

Images