Distributed duplication elimination system-oriented data routing method

Abstract

The invention provides a distributed duplication elimination system-oriented data routing method, which comprises the following steps that a server categorizes fingerprints of all data blocks forming data, and transmits the fingerprints of different categories to different abstract storage nodes for storing data abstracts of the fingerprints of the corresponding categories respectively; the received fingerprints in the abstract storage nodes are queried to obtain hit scores of the fingerprints in each duplication elimination node; the hit scores are returned to the server; the server obtains a summery score of each duplication elimination node according to the hit scores of the fingerprints in each duplication elimination node, and determines a target duplication elimination node by combining the summery scores and the storage conditions of the duplication elimination nodes. According to the method, duplication elimination effect and the storage utilization rate are balanced, the communication and calculation overhead in a fingerprint query process is effectively inhibited, and the data routing expandability of a distributed duplication elimination system is improved.

Description

technical field [0001] The present invention generally relates to de-duplication technology, and specifically relates to a data routing method for a distributed de-duplication system. Background technique [0002] Since mankind entered the digital information age, a large amount of information has been recorded as data. From the basic living needs of food, clothing, housing and transportation to education, medical care and business fields, from the traditional Internet to the mobile Internet developed by smart phones, more and more people and devices are involved in the creation of data, and the total amount of data generated each year shows Explosive growth. At the same time, due to the potential commercial and scientific value in the data, more and more data are recorded and preserved. According to the research report of International Data Corporation (IDC), the data created and copied globally in 2011 was as high as 1.8ZB, and according to the trend, this number will be...

AI Technical Summary

Problems solved by technology

However, this method has two shortcomings: first, because there is no reference to the stored data, the deduplication rate of the data at the target deduplication node cannot be guaranteed; secondly, because the existing storage utilization of the deduplication node is not considered, and the data The deduplication effect of different deduplication server nodes is different, so the problem of data islands will arise, that is, the data stored by a deduplication server node is much higher than that of other deduplication server nodes

The disadvantage is that this method requires additional summary storage nodes for query, and the memory overhead of data summary is very large, so this method is difficult to obtain good scalability

[0006] It can be seen that how to improve the scalability of data routing in the distributed deduplication system while achieving a balance between the deduplication effect and storage utilization, and how to suppress the increase in communication and computing overhead during the fingerprint query process is currently an unsolved problem

Images