Supercharge Your Innovation With Domain-Expert AI Agents!

A Distributed Data Deduplication Method and Device Based on Self-Matching Feature

A technology of de-duplication and self-matching, applied in the direction of digital data processing, input/output process of data processing, instruments, etc., can solve the problems of data redundancy in distributed storage systems, save network bandwidth and save storage space Effect

Active Publication Date: 2020-08-28
SHAOGUAN COLLEGE +2
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0015] The invention provides a method for deduplicating data based on self-matching features to solve the problem of data redundancy in distributed storage systems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A Distributed Data Deduplication Method and Device Based on Self-Matching Feature
  • A Distributed Data Deduplication Method and Device Based on Self-Matching Feature
  • A Distributed Data Deduplication Method and Device Based on Self-Matching Feature

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The invention provides a distributed duplicate data deletion method based on a self-matching feature, which is used to solve the data redundancy problem in a distributed storage system. In view of the low detection rate of duplicate data in the fixed block method and the content-based block method in the prior art, and the problem that the sliding block method cannot be applied in a distributed storage environment due to excessive network traffic, the embodiment of the present invention simultaneously Calculate the digital fingerprint and checksum of the data in the sliding window, and preliminarily screen out possible duplicate data blocks by querying the self-matching table; Data blocks with the same hash value are discarded, otherwise they need to be uploaded and saved; before uploading files, customers download the self-matching table from the name node to obtain the initial feature information of self-matching, so that the entire system can achieve the effect of dis...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention discloses a distributed data deduplocation method based on self-matching characteristics. The method comprises creating a self-matching table in a name node of a distributed storage system, and storing initial characteristic information of stored data blocks, wherein a client downloads the self-matching table from the name node before uploading a file, so that distributed data deduplocation is achieved; applying a sliding window technology at the client, calculating digital fingerprint and checksum of data in a sliding window, and preliminarily filtering suspected duplicate data blocks through querying the self-matching table; and calculating Hash values of the suspected duplicate blocks by using a public SHA-1 algorithm, uploading the Hash values to the name node for comparison, and confirming the data blocks passing the Hash comparison as duplicate data. The problems that in distributed storage environment, duplicate data detection rates based on a fixed-sized block method and a content defined chunking method are low, and a sliding block method cannot be applied due to high network traffic are solved. The present invention also discloses a distributed data deduplocation apparatus based on self-matching characteristics.

Description

technical field [0001] The invention relates to the deduplication technology in a storage system, in particular to a distributed deduplication method based on self-matching features. Background technique [0002] In recent years, with the explosive growth of data information, people's demand for storage has become larger and larger, which has risen from the previous TB level to the PB level, or even the EB level. Faced with the rapid expansion of data, in addition to continuously purchasing disks to increase storage physical space, people also study data reduction technologies. Duplicate data detection is a key technology to achieve data reduction. It can greatly reduce the storage capacity of data centers, reduce construction and operation and maintenance costs, save network bandwidth, and improve data backup and recovery performance. Different from traditional data compression technology, duplicate data detection technology can not only eliminate data redundancy within a ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
CPCG06F3/0641G06F3/067
Inventor 谢垂益曾建国
Owner SHAOGUAN COLLEGE
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com