Method of reducing redundancy between two or more datasets

Abstract

A method for reducing redundancy between two or more datasets of potentially very large size. The method improves upon current technology by oversubscribing the data structure that represents a digest of data blocks and using positional information about matching data so that very large datasets can be analyzed and the redundancies removed by, having found a match on digest, expands the match in both directions in order to detect and eliminate large runs of data by replace duplicate runs with references to common data. The method is particularly useful for capturing the states of images of a hard disk. The method permits several files to have their redundancy removed and the files to later be reconstituted. The method is appropriate for use on a WORM device. The method can also make use of L2 cache to improve performance.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application is a continuation of U.S. patent application Ser. No. 12 / 455,864, filed Jun. 8, 2009, which claims the benefit of U.S. Provisional Patent Application No. 61 / 059,276, each of which is hereby incorporated herein by reference in its entirety.FIELD OF THE INVENTION[0002]This invention relates generally to a system and method for storing data. More particularly, this invention relates to a form of data size reduction which involves finding redundancies in and between large data sets (files) and eliminating these redundancies in order to conserve repository memory (generally, disk space).BACKGROUND OF THE INVENTION[0003]This invention relates generally to a system and method for storing data. More particularly, this invention relates to storing data efficiently in both the time and space domains by removing redundant data between two or more data sets.[0004]The inventors of this invention noticed that there are many times when ...

AI Technical Summary

Benefits of technology

[0032]As currently implemented prior to this invention, the random access memory (RAM) requirements to remove redundancies in, say, a pair of multi-terabyte files, exceed the capacity of most modern home computers and de-duplicating appliances. This Method vastly reduces the RAM requirements to remove much of the duplicate data between these two files as well as removing duplicate data within the files themselves.V.(2) Definition of Local Information

[0090]The Method we teach produces high deduplication efficiency (often up to 20-1; higher deduplication performance will be achieved with highly similar data in the Current and Reference Files) with oversubscription ratios of 30 to 1 or more, allowing us to use small (e.g., 512 byte) blocks even for 200+ GB files while consuming only 128 MB of RAM.

Problems solved by technology

Maintaining multiple versions of such datasets consumes a great deal of disk space, so that only a relatively few, if any, versions are, generally, kept.

Users of this Method will find this method particularly efficacious when used with Repositories with the hardware property of “read many, write many” or “read many, write few” or “read many, write once”.

Modern disk drives also have the undesirable property that random seeks take milliseconds of time.

Modern solid-state memory has the undesirable properties of being far more expensive per byte than disk-based memory as well as (with some types of solid-state memory) limiting the number of rewrites before the device fails to be rewriteable.

Because users of computers accidentally delete files or their computers become infected by computer viruses, users often wish to retain multiple backups over time.

Thus, an unsophisticated backup scheme of uncompressed 250 gigabytes (125 gigabytes compressed) would cost the user about $12 for each backup, quickly limiting the number of backups the user maintains.

Images