Method of reducing redundancy between two or more datasets

Abstract

A method for reducing redundancy between two or more datasets of potentially very large size. The method improves upon current technology by oversubscribing the data structure that represents a digest of data blocks and using positional information about matching data so that very large datasets can be analyzed and the redundancies removed by, having found a match on digest, expands the match in both directions in order to detect and eliminate large runs of data by replace duplicate runs with references to common data. The method is particularly useful for capturing the states of images of a hard disk. The method permits several files to have their redundancy removed and the files to later be reconstituted. The method is appropriate for use on a WORM device. The method can also make use of L2 cache to improve performance.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application is a continuation of U.S. patent application Ser. No. 12 / 455,864, filed Jun. 8, 2009, which claims the benefit of U.S. Provisional Patent Application No. 61 / 059,276, each of which is hereby incorporated herein by reference in its entirety.FIELD OF THE INVENTION[0002]This invention relates generally to a system and method for storing data. More particularly, this invention relates to a form of data size reduction which involves finding redundancies in and between large data sets (files) and eliminating these redundancies in order to conserve repository memory (generally, disk space).BACKGROUND OF THE INVENTION[0003]This invention relates generally to a system and method for storing data. More particularly, this invention relates to storing data efficiently in both the time and space domains by removing redundant data between two or more data sets.[0004]The inventors of this invention noticed that there are many times when ...

AI Technical Summary

Benefits of technology

The invention is a method that reduces the amount of memory needed to remove wholesale duplicate data from large sets of files. This method is much more efficient than existing techniques, allowing for high deduplication ratios with small data blocks. This results in a much more efficient use of memory, making it faster and easier to de-duplicate large amounts of data.

Problems solved by technology

Maintaining multiple versions of such datasets consumes a great deal of disk space, so that only a relatively few, if any, versions are, generally, kept.

Users of this Method will find this method particularly efficacious when used with Repositories with the hardware property of “read many, write many” or “read many, write few” or “read many, write once”.

Modern disk drives also have the undesirable property that random seeks take milliseconds of time.

Modern solid-state memory has the undesirable properties of being far more expensive per byte than disk-based memory as well as (with some types of solid-state memory) limiting the number of rewrites before the device fails to be rewriteable.

Because users of computers accidentally delete files or their computers become infected by computer viruses, users often wish to retain multiple backups over time.

Thus, an unsophisticated backup scheme of uncompressed 250 gigabytes (125 gigabytes compressed) would cost the user about $12 for each backup, quickly limiting the number of backups the user maintains.

Images