Repeated data deleting method and device

Abstract

The invention discloses a repeated data deleting method and device, and improves system reliability of a mobile intelligent terminal. The method comprises the steps of: carrying out matching on a data fingerprint of a current data block and data fingerprints in a hotspot hash table, and if a matching result is that the data fingerprint of the current data block is consistent with one data fingerprint in the hotspot hash table, determining the current data block as repeated data; and if a matching result is that the data fingerprint of the current data block is inconsistent with the data fingerprints in the hotspot hash table, carrying out matching on the data fingerprint of the current data block and data fingerprints in a hash fingerprint table, and when a matching result is that the data fingerprint of the current data block is consistent with one data fingerprint in the hash fingerprint table, determining the current data block as the repeated data, wherein the data fingerprint of each harsh table item in the hotspot hash table is a data fingerprint of which repeated times in at least one file reach a set threshold value, and the data fingerprints of the hash fingerprint table are stored data fingerprints of all the data blocks.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and device for deduplication of data. Background technique [0002] With the development of computers, smart mobile terminals have profoundly changed people's lives. In recent years, the computing and storage capabilities of mobile smart terminals have developed rapidly. [0003] At present, the storage system of mobile smart terminals has the following specific characteristics: [0004] First, there is data duplication in the storage system. For example, the redundancy of duplicate data generated by application installation and update is about 45%. [0005] Second, the smart terminal adopts a permanent storage medium of flash memory (flash), which has poor write operation performance and a limited number of erase operations. [0006] Due to the existence of duplicate data in the storage system, the reliability of the system is greatly affected, so it is urgen...

AI Technical Summary

Problems solved by technology

[0004] First, there is data duplication in the storage system. For example, the redundancy of duplicate data generated by application installation and update is about 45%.

[0005] Second, smart terminals use flash memory (flash) as a permanent storage medium, which has poor write performance and limited number of erase operations

In addition, the erasing times of the flash storage medium are limited. If there are many duplicate data, it will cause a large number of write operations, which will reduce the service life of the flash and affect the system reliability of the mobile smart terminal.

Since the duplicate data fingerprint query operation provided by the existing technology has high requirements on computing resources and storage resources, the application of the existing data deduplication technology to the smart terminal will seriously affect the system reliability of the mobile smart terminal

Images