Exploiting Locality of Primary Data for Efficient Retrieval of Lossless Reduced Data Using Primary Data Sieves

Abstract

The amount of memory required to hold basic data elements during reconstruction can be determined by examining the creation and use of basic data elements and their spatial and temporal characteristics during data distillation.

Description

[Technical Field]

[0001] background This disclosure relates to the storage, retrieval, and communication of data. More particularly, this disclosure relates to the efficient retrieval and reconstruction of data that has been losslessly reduced using elementary data sieves. [Background technology]

[0002] Related technologies The modern information age is characterized by the creation, capture, and analysis of enormous amounts of data. New data is generated from a variety of sources, including purchase transaction records, corporate and government records and communications, emails, social media posts, digital photographs and videos, machine logs, signals from embedded devices, digital sensors, mobile phones, global positioning satellites, space satellites, scientific computing, and grand challenge science. Data is generated in a variety of formats, most of which is unstructured and unsuitable for entry into traditional databases. Businesses, governments, and individuals are generating data at an unprecedented rate and struggling to store, analyze, and communicate this data. Tens of billions of dollars are spent annually on purchasing storage systems to hold the accumulating data. Similar large amounts are spent on computer systems to process the data.

[0003] In modern computer and storage systems, data is contained and deployed across multiple storage tiers, organized as a storage hierarchy. Data that needs to be accessed frequently and quickly is placed in the most expensive but fastest tier, while the majority of the data (including backup copies) is preferably stored on the densest and cheapest storage medium. The fastest and most expensive data storage tier is the computer system's volatile random access memory or RAM, which resides in close proximity to the microprocessor core and provides the lowest latency and highest bandwidth for random access of data. Progressively denser and cheaper but slower tiers (with progressively higher latency and lower bandwidth for random access) include non-volatile solid-state memory or flash storage, hard disk drives (HDDs), and finally tape drives.

[0004] To more effectively store and process increasing amounts of data, the computer industry continues to improve the density and speed of data storage media and the processing power of computers. However, the increase in data volume has far outpaced the improvements in the capacity and density of computing and data storage systems. Statistics from the data storage industry in 2014 revealed that new data created and captured in the past few years comprises the majority of data ever captured worldwide. It is estimated that the amount of data created worldwide to date exceeds several zettabytes (a zettabyte is 1021 bytes). The massive increase in data places great demands on data storage, computing, and communication systems that must reliably store, process, and communicate this data. This has motivated the increasing use of lossless data reduction, or compression, techniques, which compress data so that it can be stored at low cost and similarly processed and communicated efficiently.

[0005] A variety of lossless data reduction or compression techniques have emerged and evolved over the years. These techniques examine data for some form of redundancy within the data and exploit that redundancy to achieve a reduction in the data footprint without losing information. For a given technique that attempts to exploit a particular form of redundancy within the data, the degree of data reduction achieved depends on how frequently that particular form of redundancy is found within the data. It is desirable for a data reduction technique to be flexible in discovering and exploiting any available redundancy within the data. Because data originates in a variety of formats from a wide variety of sources and environments, there is great interest in developing and adopting general-purpose lossless data reduction techniques that can handle this variety of data. A general-purpose data reduction technique is one that does not require any prior knowledge of the input data other than the alphabet. It is therefore generally applicable to any data without requiring prior knowledge of the data's structure and statistical distribution characteristics.

[0006] Goodness-of-fit metrics that can be used to compare various implementations of data compression techniques include the degree of data reduction achieved on the target data set, the efficiency with which the compression or reduction is achieved, and the efficiency with which the data is restored and retrieved for future use. Efficiency metrics evaluate the performance and cost-effectiveness of a solution. Performance metrics include the throughput or ingestion rate at which new data can be consumed and reduced, the latency or time required to reduce the input data, the throughput or rate at which the data can be restored and retrieved, and the latency or time required to restore and retrieve the data. Cost metrics include the cost of any dedicated hardware components required, such as microprocessor cores or microprocessor utilization (central processing unit utilization), the amount of dedicated scratch memory and memory bandwidth, and the number of accesses and bandwidth required from the various storage tiers that hold the data. Note that reducing the data footprint while simultaneously providing efficient and fast compression and restoration and retrieval has the advantage of not only reducing the overall cost of storing and communicating data, but also enabling efficient subsequent processing of the data.

[0007] Many of the general-purpose data compression techniques currently used in industry originate from the Lempel-Ziv compression method, developed by Abraham Lempel and Jacob Ziv in 1977. See, for example, Jacob Zivand Abraham Lempel, "A Universal Algorithm for Sequential Data Compression," IEEE Transactions on Information Theory, Vol. IT-23, No. 3, May 1997. This method became the foundation for enabling efficient data transmission over the Internet. Lempel-Ziv methods (named LZ77, LZ78, and their variants) reduce data footprint by replacing recurring strings with references to previous occurrences found within a sliding window of a sequentially presented input data stream. Upon consuming a new string from a given data block from the input data stream, these techniques search for all strings previously seen within the current and previous blocks, up to the length of the window. If the new string is a duplicate, it is replaced with a backward reference to the original string. If the number of bytes removed by the duplicate string is greater than the number of bytes required for the backward reference, data reduction has been achieved. To search all strings visible within the window and provide maximal string matches, implementations of these techniques use various schemes that involve iterative scanning and building a temporary ledger structure that contains a dictionary of all strings visible within the window. Upon consuming new input bytes to assemble a new string, these techniques either scan all bytes within the existing window or create a dictionary lookup of the string (and then perform some calculations), determine if a duplicate is found, and replace the duplicate with a backward reference (or determine if an addition to the dictionary is needed).

[0008] Another notable prior art method achieves compression by recoding data within a block or message based on its entropy. In this method, source symbols are dynamically recoded based on the frequency or probability of their occurrence within the data block being compressed, often using a variable-width coding scheme so that shorter length codes are used for more frequent symbols, thereby reducing data. For an example of such an entropy-based recoding method, see David A. Huffman, "A Method for the Construction of Minimum-Redundancy Codes," Proceedings of the IRE-Institute of Radio Engineers, September 1952, pp. 1098-1101. This technique, known as Huffman recoding, typically requires a first pass through the data to calculate the frequency and a second pass to actually encode the data. Several variations on this theme are also in use.

[0009] One example of using these techniques is a scheme known as "Deflate," which combines the Lempel-Ziv LZ77 compression method with Huffman recoding. Deflate provides a compressed stream data format specification that specifies how to represent a sequence of bytes as a (usually shorter) sequence of bits and how to pack the latter bit sequences into bytes. The Deflate scheme was originally designed by Phillip W. Katz of PKWARE, Inc., for the PKZIP archive utility. See, for example, "String searcher, and compressor using same," Phillip W. Katz, U.S. Pat. No. 5,051,745 (September 24, 1991). U.S. Pat. No. 5,051,745 describes a method for searching a vector of symbols (a window) for a predetermined target string (the input string). This solution uses a pointer array with a pointer for each symbol in the window and uses a hashing method to filter possible locations within the window where an identical copy of the input string needs to be searched. Scanning and string matching are then performed at those locations.

[0010] The Deflate scheme is implemented in the zlib library for data compression. zlib is a software library that is a major component of several software platforms, such as Linux, Mac OS X, and iOS, as well as various gaming consoles. The zlib library provides the Deflate compression and decompression code used by zip (file archiving), gzip (single-file compression), png (Portable Network Graphics Format for losslessly compressed images), and many other applications. zlib is now widely used for data transmission and storage. Most HTTP transactions by servers and browsers use zlib to compress and decompress data. Similar implementations are increasingly being used by data storage systems.

[0011] A paper published by Intel in April 2014, titled "High Performance ZLIB Compression on Intel® Architecture Processors," characterized the compression and performance of an optimized version of the zlib library running on a current Intel processor (Core i7 4770 processor, 3.4 GHz, 8 MB cache) and performed against the Calgary corpus of data. The Deflate format used by zlib sets the minimum string length for a match to 3 characters, the maximum match length to 256 characters, and the window size to 32 kilobytes. This implementation offers nine levels of optimization control, with level 9 providing the highest compression but using the majority of the computation to perform the most exhaustive string match, and level 1, the fastest level, using greedy string matching. The paper reported a 51% compression ratio using zlib level 1 (the fastest level) with a single-threaded processor, consuming an average of 17.66 clocks per byte of input data. At a clock frequency of 3.4 GHz, this translates to an ingest rate of 192 MB / s while exhausting a single microprocessor core. The report further states that performance drops sharply to an ingest rate of 38MB / s (average 88.1 clocks / bytes) using optimization level 6 for a moderate compression gain, and to 16MB / s (average 209.5 clocks / bytes) using optimization level 9.

[0012] Existing data compression solutions typically operate on current microprocessors with ingest rates ranging from 10MB / s to 200MB / s using a single processor core. To further increase ingest speeds, multiple cores are used or the window size is reduced. Further increases in ingest speed can be achieved using custom hardware accelerators, albeit at increased cost.

[0013] The existing data compression methods described above are effective at exploiting fine redundancies at the level of short strings and symbols within a local window, typically the size of one message or file, or possibly several files. These methods have significant limitations and drawbacks when used in applications that operate on large or very large data sets and require high speed data ingestion and retrieval.

[0014] One important limitation is that practical implementations of these methods can only efficiently exploit redundancies within a local window. While these implementations can accept arbitrarily long input streams of data, efficiency imposes limits on the size of the window within which fine redundancies must be discovered. These methods are highly computationally intensive, requiring frequent and rapid access to all data within the window. String matches and lookups of various ledger structures are triggered upon consuming a new byte (or a few bytes) of input data, creating a new input string. To achieve the desired retrieval rate, the window for string matching and associated machinery must reside mostly within the processor cache subsystem, which imposes practical constraints on the window size.

[0015] For example, to achieve an ingest rate of 200 MB / s on a single processor core, the average available time budget per ingested byte (including all data accesses and computation) is 5 ns, or 17 clock cycles using current processors operating at 3.4 GHz. This budget accommodates accesses to the on-chip cache (which take up a few cycles), followed by any string matching. Current processors have on-chip caches of several megabytes in capacity. Because accesses to main memory take up 200 cycles (~70 ns), a large window that resides mostly in memory further slows the ingest rate. Also, as the window size increases and as the distance to the duplicate string increases, the cost of specifying the length of the backward reference also increases, encouraging only longer strings to be searched for duplicates over a wider range.

[0016] In most current data storage systems, the footprint of data stored across various tiers of the storage hierarchy is several orders of magnitude larger than the memory capacity within the system. For example, while a system may offer hundreds of gigabytes of memory, the data footprint of active data residing in flash storage may be tens of terabytes, and the total data within the storage system may range from hundreds of terabytes to several petabytes. Additionally, the achievable throughput of data access to subsequent storage tiers decreases by an order of magnitude or more with each subsequent tier. Once the sliding window becomes too large to fit into memory, these technologies are hampered by fairly low bandwidth and high latency for random IO (input or output operation) access to the next level of data storage.

[0017] For example, consider a 4 kilobyte file or page of incoming data that can be assembled from existing data by referencing 100 strings, say, with an average length of 40 bytes, that already exist within the data and are distributed across a 256 terabyte footprint. Each reference requires 6 bytes to specify its address and 1 byte for the string length, while committing to storing 40 bytes. While this example page is compressible by more than five times, the ingestion rate for this page would be limited by the 100+ IO accesses to the storage system required to fetch and examine the 100 duplicate strings (even if we could perfectly and inexpensively predict where these strings would reside). A storage system offering 250,000 random IO accesses / second (representing a 1 GB / second random access bandwidth to 4 KB pages) would be unusable because it could only compress 2,500 such 4 KB-sized pages per second for an ingestion rate of only 10 MB / second, using up the entire bandwidth of the storage system.

[0018] Implementations of conventional compression methods that use large window sizes on the order of terabytes or petabytes are starved by the reduced bandwidth of data access to the storage system and become unacceptably slow. Thus, practical implementations of these techniques efficiently discover and exploit redundancies only when they are local, for window sizes that fit into processor caches or system memory. When the redundant data is separated in space or time by several terabytes, petabytes, or exabytes from the incoming data, these implementations are unable to discover redundancies at an acceptable rate and are limited by storage access bandwidth.

[0019] Another limitation of conventional methods is that they are not suitable for random access of data. A block of data spanning an entire compressed window must be decompressed before any chunk within any block can be accessed. This imposes a practical limit on the size of the window. Also, operations traditionally performed on uncompressed data (e.g., search operations) cannot be performed efficiently on compressed data.

[0020] Yet another limitation of traditional methods (and especially those based on Lempel-Ziv) is that they search for redundancies along only one dimension, replacing identical strings with backward references. A limitation of the Huffman recoding scheme is that it requires two passes through the data to calculate the frequencies and then recode them, which becomes slow as the blocks get larger.

[0021] Data compression methods that detect long, duplicate strings throughout a global store of data often use a combination of digital fingerprinting and hashing schemes. This compression process is called data deduplication. The most basic data deduplication technique divides a file into fixed-size blocks and searches for duplicate blocks throughout a data repository. When a copy of a file is created, each block in the first file will have a duplicate in the second file, and the duplicate can be replaced with a reference to the original block. To speed up matching potentially duplicate blocks, hashing methods are used. A hash function converts a string into a number called its hash value. If two strings are equal, their hash values ​​are also equal. A hash function maps multiple strings to a given hash value, which can reduce long strings to hash values ​​of much shorter length. Matching hash values ​​is much faster than matching two long strings. Therefore, hash value matching is performed first to filter out potentially duplicate strings. If the hash value of an input string or block matches the hash value of a string or block that exists in the repository, the input string may then be compared with each string in the repository that has the same hash value to check for duplicates.

[0022] Dividing files into fixed-size blocks is simple and convenient, and fixed-size blocks are highly desirable in high-performance storage systems. However, this technique is limited in the amount of redundancy it can find, meaning that these techniques provide a low level of compression. For example, if a copy of a first file is made to create a second file, inserting even a single byte of data into the second file will change the alignment of all downstream blocks, requiring a new hash value to be calculated for each new block, preventing data deduplication methods from finding all duplicates.

[0023] To address this limitation in data deduplication methods, the industry has adopted the use of fingerprinting to synchronize and align data streams at locations of matching content. This latter scheme results in variable-sized blocks based on the fingerprint. Michael Rabin has shown that bit strings can be fingerprinted using randomly chosen irreducible polynomials. See, for example, Michael O. Rabin, "Fingerprinting by Random Polynomials," Center for Research in Computing Technology, Harvard University, TR-15-81, 1981. In this scheme, a long string is fingerprinted by computing the remainder of the string taken as a large integer modulo p using a randomly chosen prime number p. This scheme requires integer arithmetic on k-bit integers, where k = log2(p). Alternatively, a random irreducible prime polynomial of order k can be used, in which case the fingerprint is a polynomial representation of the data modulo the prime polynomial.

[0024] This fingerprinting method can be used in a data deduplication system to identify suitable locations for establishing chunk boundaries, allowing the system to search for duplicates of these chunks in a global repository. Chunk boundaries can be set when a fingerprint of a particular value is found. As an example of such use, a fingerprint can be calculated for every 48-byte string in the input data (starting with the first byte of the input, then for every consecutive byte thereafter) using a polynomial of order 32 or less. The lower 13 bits of the 32-bit fingerprint can then be examined to set a breakpoint whenever the value of those 13 bits is a pre-specified value (e.g., the value 1). For random data, there is a 1 in 2 chance that the 13 bits have that particular value, so such a breakpoint is likely to be encountered approximately every 8 KB, resulting in variable-sized chunks with an average size of 8 KB. The breakpoint or chunk boundary will effectively align with the fingerprint depending on the content of the data. If a fingerprint is not found for a long time, a breakpoint can be enforced at some pre-specified threshold, so that the system ensures that chunks shorter than a pre-specified size are created for the repository. See, e.g., Athicha Muthitacharoen, Benjie Chen and David Mazieres, "A Low-bandwidth Network File System," SOSP '01, Proceedings of the eighteenth ACM symposium on Operating Systems Principles, 10 / 21 / 2001, pp. 174-187.

[0025] The Rabin-Karp string matching technique, developed by Michael Rabin and Richard Karp, further improved the efficiency of fingerprinting and string matching (see, for example, Michael O. Rabin and R. Karp, "Efficient Randomized Pattern-Matching Algorithms," IBM Jour. of Res. and Dev., Vol. 31, 1987, pp. 249-260). Note that a fingerprinting method that examines an m-byte substring for its fingerprint can evaluate a fingerprinting polynomial function in O(m) time. Because this method must be applied to a substring starting at each byte of, for example, an n-byte input stream, the total effort required to fingerprint the entire data stream is O(n × m). Rabin-Karp identified a hash function called a rolling hash, for which the hash value of a next substring can be calculated from the previous substring by performing only a constant number of operations, regardless of the substring's length. Therefore, after shifting one byte to the right, the fingerprint calculation can be done incrementally on the new m-byte string. This reduces the effort to calculate the fingerprint to O(1) and the total effort to fingerprint the entire data stream to O(n), linearly with the size of the data. This greatly speeds up fingerprint calculation and identification.

[0026] Typical data access and computation requirements for the above data deduplication methods can be described as follows: For a given input, once fingerprinting is complete and chunks are created, their hash values ​​are calculated, and then these methods first require a set of accesses to memory and subsequent storage tiers to search and look up a global hash table that maintains the hash values ​​of all chunks in the repository. This typically requires one IO access to storage. If there is a match in the hash table, then a second set of storage IOs (typically one, but could be more than one depending on how many chunks with the same hash value exist in the repository) is performed to fetch the actual data chunk with the same hash value. Finally, a byte-by-byte match is performed, comparing the input chunk with the fetched potentially matching chunk to confirm and identify duplicates. This is followed by a third storage IO access (to the metadata space) to replace the new duplicate block with the original reference. If there is no match in the global hash table (or no duplicate is found), the system requires one IO to enter the new block into the repository, and another IO to update the global hash table and enter the new hash value. Thus, for large datasets (where the metadata and global hash table do not fit in memory and therefore require storage IO to access them), such a system may require an average of three IOs per input chunk. Further improvement is possible by using various filters to frequently detect misses in the global hash table without requiring a first storage IO to access the global hash table, reducing the number of IOs required to process some of the chunks to as little as two.

[0027] A storage system offering 250,000 random IO accesses per second (representing a bandwidth of 1 GB / s random access to 4 KB pages) can achieve an ingest rate of 333 MB / s by ingesting and deduplicating approximately 83,333 input chunks of average size 4 KB per second (250,000 divided by three IOs per input chunk), while exhausting the storage system's full bandwidth. Even if only half of the storage system's bandwidth is utilized (thus leaving the other half available for accessing stored data), such a deduplication system can still provide an ingest rate of 166 MB / s. These ingest rates (limited by I / O bandwidth) are achievable, provided sufficient processing power is available in the system. Thus, given sufficient processing power, a data deduplication system can find significant duplicates of data across a global scope of data with lean IO, providing data reduction at ingest rates of hundreds of megabytes per second for current storage systems.

[0028] Based on the above discussion, it should be clear that while these deduplication methods are effective at finding long string duplicates across a global scope, they are primarily effective at finding large duplicates. When there are variations or changes in the data at a finer level, available redundancies cannot be found using this method. This significantly reduces the breadth of the data set for which these methods are useful. These methods are used in certain data storage systems and applications, such as periodic data backups, where the new data in the backup contains only a small number of modified files, with the rest being duplicates of files stored in previous backups. Similarly, systems based on data deduplication are often deployed in environments where multiple exact copies of data or code are created, such as virtual environments in data centers. However, as data evolves and changes more commonly or at a finer level, data deduplication-based techniques are losing their effectiveness.

[0029] Some approaches (typically used in data backup applications) do not perform an actual byte-by-byte comparison between the input data and a string whose hash value matches the input's hash value. Such solutions rely on a low probability of collision using a strong hash function such as SHA-1. However, due to the finite, non-zero probability of a collision (where multiple different strings may map to the same hash value), such methods cannot be considered to provide lossless data reduction and therefore do not meet the high data integrity requirements of primary storage and communication.

[0030] Some approaches combine multiple existing data compression techniques. Typically, in such a setup, a global data deduplication method is first applied to the data. Then, Lempel-Ziv string compression combined with Huffman recoding using a small window is applied to the deduplicated data set to achieve further data reduction.

[0031] However, despite the use of all known technologies, an order-of-magnitude gap continues to exist between the need for increasingly accumulated data and what the global economy can conveniently accommodate using the best available modern storage systems. Given the extraordinary storage capacity requirements that growing data poses, improved methods for further reducing the data footprint remain necessary. There remains a need to develop methods that address the limitations of existing technologies or exploit available redundancies in the data along dimensions not addressed by existing technologies. At the same time, it remains important to be able to efficiently access and retrieve data at an acceptable speed and with acceptable processing costs. There also remains a need to be able to efficiently perform search operations directly on the reduced data.

[0032] In summary, there is a long felt need for a lossless data reduction solution that can exploit redundancy across large and very large data sets and provide high speed data ingestion, data search, and data retrieval. Summary of the Invention [Means for solving the problem]

[0033] overview The embodiments described herein feature techniques and systems that can perform lossless data reduction on large and very large data sets while providing high speed data ingestion and retrieval, and that do not suffer from the drawbacks and limitations of existing data compression systems. [Brief explanation of the drawings]

[0034] [Figure 1A] FIG. 1 illustrates a method and apparatus for data reduction that factors input data into elements and derives these from basic data elements present in a basic data sieve, according to some embodiments described herein. [Figure 1B] 1B illustrates a variation of the method and apparatus shown in FIG. 1A, according to some embodiments described herein. [Figure 1C] 1B illustrates a variation of the method and apparatus shown in FIG. 1A, according to some embodiments described herein. [Figure 1D] 1B illustrates a variation of the method and apparatus shown in FIG. 1A, according to some embodiments described herein. [Figure 1E] 1B illustrates a variation of the method and apparatus shown in FIG. 1A, according to some embodiments described herein. [Figure 1F] 1B illustrates a variation of the method and apparatus shown in FIG. 1A, according to some embodiments described herein. [Figure 1G] 1B illustrates a variation of the method and apparatus shown in FIG. 1A, according to some embodiments described herein. [Figure 1H]1A-1C provide examples of formats and specifications describing the structure of distillation data according to some embodiments described herein. [Figure 1I] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1J] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1K] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1L] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1M] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1N] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1O] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 1P] 1A-1G illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in FIGS. 1A-1G. [Figure 2] FIG. 1 illustrates a process for data reduction by factoring input data into elements and deriving these elements from basic data elements present in a basic data sieve, according to some embodiments described herein. [Figure 3A]FIG. 1 illustrates a data organization system that may be used to organize basic data elements based on their names, according to some embodiments described herein. [Figure 3B] FIG. 1 illustrates a data organization system that may be used to organize basic data elements based on their names, according to some embodiments described herein. [Figure 3C] FIG. 1 illustrates a data organization system that may be used to organize basic data elements based on their names, according to some embodiments described herein. [Figure 3D] FIG. 1 illustrates a data organization system that may be used to organize basic data elements based on their names, according to some embodiments described herein. [Figure 3E] FIG. 1 illustrates a data organization system that may be used to organize basic data elements based on their names, according to some embodiments described herein. [Figure 3F] FIG. 1 presents a self-describing tree node data structure according to some embodiments described herein. [Figure 3G] FIG. 1 presents a self-describing leaf node data structure according to certain embodiments described herein. [Figure 3H] 1 presents a self-describing leaf node data structure including a navigation lookahead field according to certain embodiments described herein. [Figure 4] FIG. 1 illustrates an example of how 256TB of basic data may be organized into a tree, and presents how the tree may be laid out in memory and storage, according to some embodiments described herein. [Figure 5A] FIG. 1 shows a practical example of how data can be organized using embodiments described herein. [Figure 5B] FIG. 1 shows a practical example of how data can be organized using embodiments described herein. [Figure 5C] FIG. 1 shows a practical example of how data can be organized using embodiments described herein. [Figure 6A] 1B is a diagram illustrating how a tree data structure may be used for the content association mapper described with reference to FIG. 1A, according to some embodiments described herein. [Figure 6B] 1C is a diagram illustrating how a tree data structure may be used for the content association mapper described with reference to FIG. 1B, according to some embodiments described herein. [Figure 6C] 1D is a diagram illustrating how a tree data structure may be used for the content association mapper described with reference to FIG. 1C, according to some embodiments described herein. [Figure 7A] 10A-10C provide examples of transformations that may be specified in a reconstruction program, according to some embodiments described herein. [Figure 7B] 10A-10C illustrate example results of candidate elements being derived from base data elements according to some embodiments described herein. [Figure 8A] FIG. 3B illustrates how data reduction can be performed by factoring input data into fixed-size elements and organizing the elements into a tree data structure as described with reference to FIGS. 3D and 3E, in accordance with some embodiments described herein. [Figure 8B] FIG. 3B illustrates how data reduction can be performed by factoring input data into fixed-size elements and organizing the elements into a tree data structure as described with reference to FIGS. 3D and 3E, in accordance with some embodiments described herein. [Figure 8C] FIG. 3B illustrates how data reduction can be performed by factoring input data into fixed-size elements and organizing the elements into a tree data structure as described with reference to FIGS. 3D and 3E, in accordance with some embodiments described herein. [Figure 8D]FIG. 3B illustrates how data reduction can be performed by factoring input data into fixed-size elements and organizing the elements into a tree data structure as described with reference to FIGS. 3D and 3E, in accordance with some embodiments described herein. [Figure 8E] FIG. 3B illustrates how data reduction can be performed by factoring input data into fixed-size elements and organizing the elements into a tree data structure as described with reference to FIGS. 3D and 3E, in accordance with some embodiments described herein. [Figure 9A] FIG. 1D illustrates an example of a Data Distillation™ scheme based on the system shown in FIG. 1C, according to some embodiments described herein. [Figure 9B] FIG. 1D illustrates an example of a Data Distillation™ scheme based on the system shown in FIG. 1C, according to some embodiments described herein. [Figure 9C] FIG. 1D illustrates an example of a Data Distillation™ scheme based on the system shown in FIG. 1C, according to some embodiments described herein. [Figure 10A] FIG. 10 provides an example of how transformations specified in a reconstruction program are applied to base data elements to result in derived elements, according to some embodiments described herein. [Figure 10B] FIG. 1 illustrates a data retrieval process according to certain embodiments described herein. [Figure 10C] FIG. 1 illustrates a data retrieval process according to certain embodiments described herein. [Figures 11A-11F] FIG. 1 illustrates a system including a Data Distillation™ mechanism (which may be implemented using software, hardware, or a combination thereof) according to some embodiments described herein. [Figure 11G]FIG. 1 illustrates a system including a Data Distillation™ mechanism (which may be implemented using software, hardware, or a combination thereof) according to some embodiments described herein. [Figure 11H] FIG. 1 illustrates how a Data Distillation™ device may interface with a sample general-purpose computing platform, according to some embodiments described herein. [Figure 11I] FIG. 1 illustrates how a Data Distillation™ device can be used for data reduction in a block-processing storage system. [Figure 12A] FIG. 1 illustrates the use of a Data Distillation™ device for communication of data across a bandwidth-constrained communication medium, in accordance with some embodiments described herein. [Figure 12B] FIG. 1 illustrates the use of a Data Distillation™ device for communication of data across a bandwidth-constrained communication medium, in accordance with some embodiments described herein. [Figure 12C] 3A-3C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12D] 3A-3C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12E] 3A-3C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12F]1A-1C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12G] 1A-1C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12H] 1A-1C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12I] 1A-1C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12J] 1A-1C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12K] 1A-1C illustrate various components of reduced data generated by a Data Distillation™ device for various usage models, according to some embodiments described herein. [Figure 12L] FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12M] FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12N]FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12O] FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12P] FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12Q] FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12R] FIG. 1 illustrates how the distillation process, according to some embodiments described herein, can be deployed and executed on a distributed system to accommodate very large datasets with ultra-fast ingestion rates. [Figure 12S] FIG. 1 illustrates modifications to a distillation apparatus to improve the efficiency of the reconstitution process through the use of metadata collected during the distillation process. [Figure 12T] FIG. 1 illustrates modifications to a distillation apparatus to improve the efficiency of the reconstitution process through the use of metadata collected during the distillation process. [Figure 12U] FIG. 1 illustrates modifications to a distillation apparatus to improve the efficiency of the reconstitution process through the use of metadata collected during the distillation process. [Figure 12V] FIG. 1 illustrates modifications to a distillation apparatus to improve the efficiency of the reconstitution process through the use of metadata collected during the distillation process. [Figure 13] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 14A] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 14B] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 14C] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 14D] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 15] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 16A] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 16B] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 16C] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 16D] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 16E] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 17] FIG. 1 illustrates how multi-dimensional search and data retrieval can be performed on reduced data according to some embodiments described herein. [Figure 18A] 1 shows a block diagram of an encoder and decoder for compressing and decoding audio data according to the MPEG1, Layer 3 standard (also referred to as MP3). [Figure 18B] 1 shows a block diagram of an encoder and decoder for compressing and decoding audio data according to the MPEG1, Layer 3 standard (also referred to as MP3). [Figure 18C] FIG. 1B illustrates how the data distiller first shown in FIG. 1A can be improved to perform data reduction on MP3 data. [Figure 19] FIG. 1B illustrates how the data distiller first shown in FIG. 1A can be improved to perform data reduction on video data. DETAILED DESCRIPTION OF THE INVENTION

[0035] Detailed Description The following description is presented to enable any person skilled in the art to make and use the invention, and is provided in the context of a particular application and its requirements. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the invention. Thus, the present invention is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features disclosed herein. In this disclosure, when a phrase uses the word "and / or" in conjunction with a set of entities, the phrase includes all possible combinations of the set of entities unless otherwise stated. For example, the phrase "X, Y, and / or Z" includes "X only," "Y only," "Z only," "X and Y without Z," "X and Z without Y," "Y and Z without X," and the seven combinations of "X, Y, and Z."

[0036] Efficient lossless reduction of data using basic data sieves In some embodiments described herein, data is organized and stored to efficiently discover and exploit redundancies globally across the entire data set. The input data stream is divided into constituent pieces or chunks called elements, and redundancies between elements are detected and exploited at a granularity smaller than the elements themselves, thereby reducing the overall footprint of stored data. A set of elements called base data elements is identified and used as common and shared building blocks for the data set and stored in a structure called a base data store or base data sieve. A base data element is simply a sequence of bits, bytes, or digits of a certain size. The base data element may be fixed-size or variable-size, depending on the implementation. Other components of the input data are derived from the base data elements and are called derived elements. Thus, the input data is factorized into base data elements and derived elements.

[0037] The base data sieve orders and organizes base data elements so that the base data sieve can be searched and accessed content-associatively. Given some input content, the base data sieve can be queried, with some restrictions, to retrieve the base data element containing that content. Given an input element, the base data sieve can be searched using the element's value, or the values ​​of certain fields within the element, to quickly provide one or a small set of base data elements from which the input element can be derived with minimal storage required to specify the derivation. In some embodiments, elements in the base data sieve are organized in a tree form. Derived elements are derived from base data elements by performing transformations on them, and such transformations are specified in a reconstruction program that describes how to generate the derived element from one or more base data elements. A distance threshold specifies a limit on the size of the storage footprint of the derived element. This threshold specifies the maximum allowable distance of a derived element from a base data element and also imposes a limit on the size of the reconstruction program that can be used to generate the derived element.

[0038] Retrieval of the derived data is accomplished by running a reconstruction program against one or more of the base data elements specified by the derivation.

[0039] In this disclosure, the above general-purpose lossless data reduction technique may be referred to as a Data Distillation™ process, which functions similar to chemical distillation, separating a mixture into its components. The basic data sieve is also referred to as a sieve, a Data Distillation™ sieve, or a basic data store.

[0040] In this scheme, the input data stream is factored into a sequence of elements, each of which is a base data element or a derived element derived from one or more base data elements. Each element is converted into a lossless reduced representation, which, in the case of a base data element, includes a reference to the base data element, and, in the case of a derived element, includes a reference to one or more base data elements involved in the derivation and a description of the reconstruction program. Thus, the input data stream is factored into a sequence of elements that are in the lossless reduced representation. This sequence of elements (as they appear in the lossless reduced representation) is referred to as the distilled data stream or distilled data. The sequence of elements in the distilled data has a one-to-one correspondence with the sequence of elements in the input data, i.e., the nth element in the sequence of elements in the distilled data corresponds to the nth element in the sequence of elements in the input data.

[0041] The generalized lossless data reduction techniques described in this disclosure receive an input data stream and convert it into a combination of a distilled data stream and an elementary data sieve such that the sum of the footprints of the distilled data stream and the elementary data sieve is typically smaller than the footprint of the input data stream. In this disclosure, the distilled data stream and the elementary data sieve are collectively referred to as lossless reduced data, and are also referred to interchangeably as "reduced data stream" or "reduced data." Similarly, the terms "reduced output data stream," "reduced output data," "distilled data stream," and "distilled data" are used interchangeably to refer to the sequence of elements produced by the lossless data reduction techniques described in this disclosure and appearing in a lossless reduced format.

[0042] 1A illustrates a method and apparatus for data reduction that factors input data into elements and derives these from basic data elements present in a basic data sieve, according to some embodiments described herein. The figure shows an overall block diagram of a data reduction or Data Distillation™ method and apparatus, providing an overview of its functional components, structure, and operations. The components and / or operations illustrated in FIG. 1A may be implemented using software, hardware, or a combination thereof.

[0043] A sequence of bytes is received from an input data stream and presented as input data 102 to a data reduction unit 103, also referred to as a Data Distillation™ unit. A parser and factorizer 104 parses the received data and divides it into chunks or candidate elements. The factorizer determines where to insert breaks in the input stream to slice up the stream into candidate elements. When two consecutive breaks in the data are identified, a candidate element 105 is created by the parser and factorizer and presented to a basic data sieve 106, also referred to as a Data Distillation™ sieve.

[0044] The Data Distillation™ sieve or base data sieve 106 contains all the base data elements (labeled PDE in FIG. 1A ) and orders and organizes them based on their value or content. The sieve provides support for two types of access. First, each base data element can be accessed directly by reference to where it resides in the sieve. Second, elements can be accessed content-associatively by using a content-associative mapper 121, which can be implemented in software, hardware, or a combination thereof. This second form of access to the sieve is an important feature used by the disclosed embodiments to identify base data elements that are exact matches for candidate elements 105 or from which candidate elements can be derived. Specifically, given a candidate element, such as candidate element 105, the base data sieve 106 can be searched (based on the value of candidate element 105 or based on the values ​​of certain fields within candidate element 105) to quickly provide one or a small set of base data elements 107 from which the candidate element can be derived with minimal storage required to specify the derivation.

[0045] A sieve or basic data sieve 106 may be initialized with a set of basic data elements whose values ​​are distributed across the data space. Alternatively, the sieve may start empty and basic data elements may be dynamically added to the sieve as data is ingested according to the Data Distillation™ process described herein with reference to Figures 1A-1C and 2.

[0046] The derivation unit 110 receives candidate elements 105 and retrieved base data elements 107 suitable for derivation (content associatively retrieved from the base data sieve 106), determines whether the candidate element 105 is derivable from one or more of these base data elements, generates a reduced data component 115 (consisting of references to the associated base data elements and a reconstruction program), and provides updates 114 to the base data sieve. If the candidate element is a duplicate of a retrieved base data element, the derivation unit places a reference (or pointer) to the base data element in the base data sieve and an indicator that this is a base data element in the distilled data 108. If no duplicates are found, the derivation unit represents the candidate element as the result of one or more transformations performed on one or more retrieved base data elements; this set of transformations is collectively referred to as a reconstruction program, such as reconstruction program 119A. Each derivation may require its own program to be constructed by the derivation unit. The reconstruction program specifies transformations such as insertion, deletion, substitution, concatenation, arithmetic, and logical operations that can be applied to the base data elements. Provided that the footprint of the derived element (calculated as the size of the reconstruction program plus the size of the necessary base data element references) is within a certain specified distance threshold with respect to the candidate element (to enable data reduction), the candidate element is reformulated as a derived element and replaced with the combination of the reconstruction program and references to one or more associated base data elements, which then form the reduced data component 115. If the threshold is exceeded, or if no suitable base data element is retrieved from the base data sieve, the base data sieve can be instructed to install the candidate as a new base data element. In this case, the derivation unit places a reference to the newly added base data element in the distilled data, along with an indicator that this is a base data element.

[0047] A data retrieval request (e.g., retrieval request 109) may be in the form of a reference to a location in a basic data sieve that contains a basic data element, or, in the case of a derivative, a combination of such a reference to a basic data element and an associated reconstruction program (or, in the case of a derivative based on multiple basic data elements, a combination of references to multiple basic data elements and associated reconstruction programs). Using one or more references to basic data elements in the basic data sieve, the retrieval unit 111 can access the basic data sieve to fetch one or more basic data elements and provide the one or more basic data elements and the reconstruction program to the reconstruction unit 112, which performs transformations (specified in the reconstruction program) on the one or more basic data elements to generate reconstructed data 116 (the requested data) and provides it to the retrieved data output 113 in response to the data retrieval request.

[0048] In a variation of this embodiment, the base data elements may be stored in a sieve in compressed form (using techniques known in the prior art, including Huffman coding and Lempel-Ziv coding), and restored as needed. This has the advantage of reducing the overall footprint of the base data sieve. The only constraint is that the content associative mapper 121 must continue to provide content associative access to the base data elements as before.

[0049] 1B and 1C illustrate variations of the method and apparatus shown in FIG. 1A , according to some embodiments described herein. In FIG. 1B , the reconstructed program may be stored in a basic data sieve and treated similarly to a basic data element. Instead of providing the reconstructed program 119A itself, a reference or pointer 119B to the reconstructed program is provided within the distilled data 108. Further data reduction is achieved when the reconstructed program is shared by other derivatives, and when the reference or pointer to the reconstructed program (plus any metadata necessary to distinguish between the reconstructed program and the reference to the reconstructed program) requires less storage space than the reconstructed program itself.

[0050] 1B, reconstructed programs are treated and accessed similarly to basic data elements and may be stored as basic data elements in the basic data sieve, thereby enabling content-associative search and retrieval of the reconstructed programs from the basic data sieve. During the derivation process to create derived elements, once the derivation unit 110 determines the reconstructed program needed for derivation, the derivation unit 110 may then determine whether the candidate reconstructed program already exists in the basic data sieve or whether the candidate reconstructed program is derivable from another entry already existing in the basic data sieve. If the candidate reconstructed program already exists in the basic data sieve, the derivation unit 110 may determine a reference to the existing entry and include the reference in the distilled data 108. If the candidate reconstructed program is derivable from an existing entry already existing in the basic data sieve, the derivation unit may provide a derivation or reformulation of the candidate reconstructed program to the distilled data; i.e., the derivation unit places a reference to the entry pre-existing in the basic data sieve in the distilled data, along with an incremental reconstruction program that derives the candidate reconstructed program from the pre-existing entry. If the candidate reconstructed program is not present in the basic data sieve and cannot be derived from an entry in the basic data sieve, the derivation unit 110 may add the reconstructed program to the basic data sieve (the operation of adding the reconstructed program to the sieve may return a reference to the newly added entry) and include a reference to the reconstructed program in the distilled data 108.

[0051] FIG. 1C presents a variation of the method and apparatus shown in FIG. 1B in accordance with certain embodiments described herein. Specifically, the mechanism in FIG. 1C used to store and query reconstruction programs is similar to the mechanism used to store and query base data elements, except that the reconstruction programs are maintained in a structure (referred to as a base reconstruction program sieve) separate from the structure containing the base data elements. An entry in such a structure is referred to as a base reconstruction program (labeled PRP in FIG. 1C). Recall that base data sieve 106 includes a content association mapper 121 that supports fast content association lookup operations. The embodiment shown in FIG. 1C includes a content association mapper 122 similar to content association mapper 121. In FIG. 1C, content association mapper 122 and content association mapper 121 are shown as being part of the base data sieve or base data store 106. In other embodiments, the content association mapper 122 and the reconstruction program may be stored in a structure called a base reconstruction program sieve separate from the base data sieve or base data store 106 .

[0052] In a variation of this embodiment, the base data elements may be stored in the sieves in compressed form (using techniques known in the prior art, including Huffman coding and Lempel-Ziv coding), and restored as needed. Similarly, the base reconstructed programs may be stored in the base reconstructed program sieves in compressed form (using techniques known in the prior art, including Huffman coding and Lempel-Ziv coding), and restored as needed. This has the advantage of reducing the overall footprint of the base data sieves and the base reconstructed program sieves. The only constraint is that the content associative mappers 121 and 122 must continue to provide content associative access to the base data elements and base reconstructed programs as before.

[0053] FIG. 1D presents a variation of the method and apparatus shown in FIG. 1A in accordance with some embodiments described herein. Specifically, in the embodiment described in FIG. 1D, the basic data elements are stored inline within the distilled data. The basic data sieve or basic data store 106 continues to provide content-associative access to the basic data elements and continues to logically contain them. It maintains references or links to the basic data elements located inline within the distilled data. For example, in FIG. 1D, the basic data element 130 is located inline within the distilled data 108. The basic data sieve or basic data store 106 maintains a reference 131 to the basic data element 130. Again, in this setup, the lossless reduced representation of the derived elements includes references to the required basic data elements. During data retrieval, the retrieval unit 111 fetches the required basic data elements from their locations.

[0054] FIG. 1E presents a variation of the method and apparatus shown in FIG. 1D , in accordance with some embodiments described herein. Specifically, in the embodiment described in FIG. 1E , similar to the setup shown in FIG. 1B , reconstruction programs may be derived from other base reconstruction programs and specified as incremental reconstruction programs plus a reference to the base reconstruction program. Such base reconstruction programs are treated similarly to base data elements and are logically installed in the base data sieve. Furthermore, in this setup, both the base data elements and the base reconstruction programs are stored inline within the distilled data. The base data sieve or base data store 106 continues to provide content-associative access to the base data elements and base reconstruction programs and continues to logically contain these base data elements and base reconstruction programs while maintaining references or links to where they are located inline within the distilled data. For example, in FIG. 1E , base data element 130 is located inline within distilled data 108. Also in FIG. 1E , base reconstruction program 132 is located inline within the distilled data. The base data sieve or base data store 106 stores the reference 13 of the base data element 130 (PDE_i). 1 (Reference_to_PDE_i), and maintain the reference 133 (Reference_to_PDE_j) of the prime reconstruction program 132 (Prime_Recon_Program_l). Again, in this setup In this case, the lossless reduced representation of the derived element includes references to the required basic data elements and the required basic reconstruction programs. During data retrieval, the retrieval unit 111 fetches the required components from their locations in the corresponding distilled data.

[0055]

[0023] Figure 1F presents a variation of the method and apparatus shown in Figure 1E, in accordance with some embodiments described herein. Specifically, in the embodiment described in Figure 1F, similar to the setup shown in Figure 1C, the base data sieve 108 includes separate mappers: a content association mapper 121 for base data elements, and a content association mapper 122 for base reconstruction programs.

[0056] FIG. 1G presents a more generalized variation of the method and apparatus shown in FIGS. 1A through 1F. Specifically, in the embodiment described in FIG. 1G, basic data elements may be placed within the basic data sieve or in-line within the distillation data. Some basic data elements may be placed within the basic data sieve, and other basic data elements may be placed in-line within the distillation data. Similarly, basic reconstruction programs may be placed within the basic data sieve or in-line within the distillation data. Some basic reconstruction programs may be placed within the basic data sieve, and other basic reconstruction programs may be placed in-line within the distillation data. The basic data sieve logically encompasses all basic data elements and basic reconstruction programs, and if a basic data element or basic reconstruction program is placed in-line within the distillation data, the basic data sieve provides a reference to its location.

[0057] The foregoing description of a method and apparatus for data reduction that factorizes input data into elements and derives these from basic data elements present in a basic data sieve has been presented for purposes of illustration and description only. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Accordingly, many modifications and variations will be apparent to those skilled in the art.

[0058] FIG. 1H provides an example of a format and specification describing the structure of the distilled data 119A of FIGS. 1A-1G of a method and apparatus for a Data Distillation™ process, according to some embodiments described herein. Because the Data Distillation™ process factors input data into base data elements and derived elements, the format for the lossless reduced representation of the data identifies these elements and describes the various components of these elements in the distilled data. The self-describing format identifies each element in the distilled data, indicates whether it is a base data element or a derived element, and describes the various components of the element: a reference to one or more base data elements installed in a sieve; a reference to a reconstruction program installed in a base data sieve (as in 119B of FIG. 1B); or a reference to a reconstruction program stored in a base reconstruction program (PRP) sieve (as in 119C of FIG. 1C); and an in-line reconstruction program (RP). The base reconstruction program (PRP) sieve is also interchangeably referred to as a base reconstruction program (PRP) store. The format of FIG. 1H has provisions for specifying derivation by running a reconstruction program on multiple base data elements, with the size of each derived element and base data element being independently specifiable. The format of FIG. 1H also has provisions for specifying base data elements that are located inline within the distilled data rather than within the base data sieve. This is specified by opcode encoding 7, which specifies that the element type is a base data element that is located inline within the distilled data. Distilled data is stored in a data storage system using this format. Data in this format is consumed by the data retrieval unit 111 so that various components of the data can be fetched and then reconstructed.

[0059] Figures 1I through 1P illustrate conceptual transformations of input data into a lossless reduced form for variations of the methods and apparatus for data reduction shown in Figures 1A through 1G. Figure 1I illustrates how a stream of input data is factored into candidate elements, which are then considered base data elements or derived elements. Finally, the data is transformed into a lossless reduced form. Figures 1I through 1N illustrate variations of the lossless reduced form for various embodiments.

[0060] Figures 1I and 1J show examples of lossless reduced forms of data produced by the method and apparatus shown in Figure 1A. The lossless reduced form of Figure 1I includes a content associative mapper, allowing for the continuous incorporation of additional data and the reduction of this data to existing base data elements. Meanwhile, the lossless reduced form of Figure 1J no longer includes a content associative mapper, resulting in data with a smaller footprint. Figures 1K and 1L show examples of lossless reduced forms of data produced by the method and apparatus shown in Figure 1C. The lossless reduced form of Figure 1K includes a content associative mapper, allowing for the continuous incorporation of additional data and the reduction of this data to existing base data elements and base reconstruction programs. Meanwhile, the lossless reduced form of Figure 1L no longer includes a content associative mapper, resulting in data with a smaller footprint.

[0061] Figures 1M and 1N show examples of lossless data reduction forms generated by the method and apparatus shown in Figure 1F, in which the base data elements and base reconstruction programs are placed inline within the distilled data. The lossless reduction form of Figure 1M includes a content associative mapper, allowing for the continuous incorporation of data and the reduction of this data relative to existing base data elements and base reconstruction programs. Meanwhile, the lossless reduction form of Figure 1N no longer retains the content associative mapper, resulting in data with a smaller footprint. Figures 1O and 1P show examples of lossless data reduction forms generated by the method and apparatus shown in Figure 1G, in which the base data elements and base reconstruction programs may be placed inline within the distilled data or within the base data sieve. The lossless reduction form of Figure 1O includes a content associative mapper, allowing for the continuous incorporation of data and the reduction of this data relative to existing base data elements and base reconstruction programs. Meanwhile, the lossless reduction form of Figure 1P no longer retains the content associative mapper, resulting in data with a smaller footprint.

[0062] In variations of the embodiments shown in Figures 1A-1P, various components of the reduced data may be further reduced or compressed using techniques known in the prior art (such as Huffman coding and Lempel-Ziv coding) and stored in this compressed form. These components may then be compressed as they become necessary for use in the data distiller. This has the advantage of further reducing the overall data footprint.

[0063] 2 illustrates a process for data reduction by factoring input data into elements and deriving these elements from base data elements present in a base data sieve, according to some embodiments described herein. When input data arrives, it may be parsed and factored or divided into a set of candidate elements (operation 202). The next candidate element is consumed from the input (operation 204), and a content-associative lookup of the base data sieve is performed based on the content of the candidate element to determine whether there are any suitable elements from which the candidate element can be derived (operation 206). If the base data sieve does not find any such elements (the "No" branch of operation 208), the candidate element is assigned and entered into the sieve as a new base data element, and the entry in the distilled data created for the candidate element becomes a reference to the newly created base data element (operation 216). If the content-associative lookup of the base data sieve yields one or more suitable elements from which candidates may be derived (the "Yes" branch of operation 208), analysis and calculations are performed on the retrieved base data elements to derive candidate elements from them. Note that in some embodiments, only the metadata for the suitable base data elements is first fetched and analysis is performed on this metadata, with the suitable base data elements subsequently fetched only if they are deemed useful (in these embodiments, the metadata for the base data elements provides some information about the content of the base data elements, allowing the system to quickly eliminate matches or evaluate derivability based on the metadata). In other embodiments, the base data sieve retrieves the base data elements directly (i.e., without first retrieving and analyzing the metadata before retrieving the base data elements), so that analysis and calculations are performed on the retrieved base data elements.

[0064] A first check is made to see if the candidate is a duplicate of any of these elements (operation 210). This check can be accelerated using any suitable hashing technique. If the candidate is identical to a base data element retrieved from the base data sieve (the "Yes" branch of operation 210), the entry in the distilled data created for the candidate element is replaced with a reference to this base data element and an indication that this entry is a base data element (operation 220). If no duplicate is found (the "No" branch of operation 210), the entry retrieved from the base data sieve based on the candidate element is considered an entry from which the candidate element may be derived. The following is an important, novel, and non-obvious feature of the base data sieve: if no duplicate is found in the base data sieve, the base data sieve can return a base data element that is not identical to the candidate element but from which the candidate element may be derived by applying one or more transformations to the base data element. The process may then analyze and calculate to derive a candidate element from the optimal base data element or a set of suitable base data elements (operation 212). In some embodiments, the derivation represents the candidate element as the result of a transformation performed on one or more base data elements, where such transformations are collectively referred to as a reconstructed program. Each derivation may require its own program to be constructed. In addition to constructing the reconstructed program, the process may further calculate a distance metric that generally indicates the level of storage and / or computational resources required to store the reformulation of the candidate element and to reconstruct the candidate element from the reformulation. In some embodiments, the footprint of the derived element is used as a measure of the candidate's distance from the base data element; specifically, the distance metric may be defined as the sum of the size of the reconstructed program plus the size of the references to one or more base data elements involved in the derivation. The derivation with the smallest distance may be selected.The distance for this derivation is compared to a distance threshold (operation 214), and if the distance does not exceed the distance threshold, the derivation is accepted (the "Yes" branch of operation 214). To result in data reduction, the distance threshold must always be less than the size of the candidate element. For example, the distance threshold may be set to 50% of the size of the candidate element, such that a derivation is accepted only if its footprint is less than or equal to half the footprint of the candidate element, thereby ensuring a reduction of at least two times for each candidate element for which a suitable derivation exists. The distance threshold may be a predetermined percentage or fraction based on user-specified input or selected by the system. The distance threshold may also be determined by the system based on static or dynamic parameters of the system. Once the derivation is accepted, the candidate element is reformulated and replaced with a combination of a reconstructed program and a reference to one or more base data elements. The entry in the distilled data created for the candidate element is replaced with the derivation, i.e., it is replaced with an indication that this is a derived element, along with the reconstructed program plus a reference to one or more base data elements that accompany the derivation (operation 218). On the other hand, if the distance for the best derivation exceeds the distance threshold (the "No" branch of operation 214), none of the possible derivations are accepted. In that case, the candidate element may be assigned and entered into the sieve as a new base data element, and the entry in the distillation data created for the candidate element will be a reference to the newly created base data element, along with an indication that this is a base data element (operation 216).

[0065] Finally, the process may check whether there are any additional candidate elements (operation 222) and return to operation 204 if there are any additional candidate elements (the "Yes" branch of operation 222), or may end the process if there are no additional candidate elements (the "No" branch of operation 222).

[0066] Various methods can be used to perform operation 202 of FIG. 2 , i.e., to parse the received data and divide it into candidate elements. A factorization algorithm must determine where to insert breaks in the byte stream to slice the stream into candidate elements. Possible techniques include, but are not limited to, dividing the stream into fixed-size blocks (e.g., 4096-byte pages), applying a fingerprinting method (e.g., applying a random prime polynomial to a substring of the input stream) to identify suitable fingerprint locations within the data stream that bound elements (this technique can result in variable-sized elements), or parsing the input to detect headers or some pre-declared structure and delineating elements based on this structure. The input can be parsed to detect a certain structure declared by a schema. The input can be parsed to detect the presence of pre-declared patterns, grammars, or regular expressions in the data. Once two consecutive breaks in the data are identified, candidate elements are created (the candidate element is the data between the two consecutive breaks) and submitted to the underlying data sieve for content-associative lookup. When a variable-sized element is created, the length of the candidate element must be specified and transmitted as metadata with the candidate element.

[0067] One important function of a base data sieve is to provide content-associative lookup based on candidate elements presented to it, and to quickly provide one or a small set of base data elements from which the candidate elements can be derived with minimal storage required to specify the derivation. This is a difficult problem given large data sets. Given terabytes of data, even with kilobyte-sized elements, there are billions of elements to search and select from. This problem becomes more severe as the data set becomes larger. After using a suitable technique to organize and order the elements, it becomes important to be able to detect similarities and derivability of elements within that organization to quickly provide a small set of suitable base data elements.

[0068] Entries in a sieve may be ordered based on the value of each element (i.e., basic data element), so that all entries may be arranged in ascending or descending order by value. Alternatively, entries may be ordered along a major axis based on the value of a certain field within the element, and then along a minor axis using the remaining content of the element. In this context, a field is a set of contiguous bytes from the content of an element. A field may be located by applying a fingerprinting method to the content of the element, such that the location of the fingerprint identifies the location of the field. Alternatively, a field may be located by choosing a certain fixed offset within the content of the element. Other methods may also be used to locate fields, including, but not limited to, parsing the element to find a certain declared structure and then locating the field within that structure.

[0069] In yet another form of organization, certain fields or combinations of fields within an element can be considered dimensions, and the concatenation of these dimensions, followed by the remaining content of each element, may be used to order and organize the data elements. In general, the correspondence or mapping between fields and dimensions can be arbitrarily complex. For example, in some embodiments, only one field may map to only one dimension. In other embodiments, a combination of multiple fields, e.g., F1, F2, and F3, may map to a dimension. Combinations of fields may be achieved by concatenating two fields together or by applying other suitable functions to them. The key requirement is that the arrangement of fields, dimensions, and remaining content of the elements used to organize the elements must be such that all basic data elements can be uniquely identified by their content and ordered within the sieve.

[0070] In yet another embodiment, a suitable function (such as an algebraic or arithmetic transform) may be applied to the elements, with the property that the result of the function uniquely identifies each element. In one such embodiment, each element is divided by a prime polynomial or some selected number or value, and the result of the division (including a quotient and remainder pair) is the function used to organize and order the elements within the basic data sieve. For example, the bits containing the remainder may form the first byte of the function result, followed by the bits containing the quotient. Alternatively, the bits containing the quotient may be used to form the first byte of the function result, followed by the bits containing the remainder. For a given divisor used to divide the input elements, the quotient and remainder pair will uniquely identify the element, and so this pair may be used to form the result of the function used to organize and order the elements within the basic data sieve. By applying this function to each element, the basic data elements may be organized within sieves based on the result of the function. The function will still uniquely identify each base data element and will provide an alternative way of sorting and organizing the base data elements within the base data sieve.

[0071] In yet another embodiment, a suitable function (such as an algebraic or arithmetic transform) may be applied to each field of an element, with the property that the result of the function uniquely identifies that field. For example, a function, such as division by a suitable polynomial or number or value, may be performed on successive fields or successive portions of the contents of each element, and the concatenation of the results of the successive functions may be used to order and organize the elements within the basic data sieve. Note that a different polynomial may be used for the division for each field. Each function will provide a suitably ordered concatenation of the bits from the quotient and remainder produced by the division operation on that portion or field. By using this concatenation of functions applied to the fields of the element, each basic data element may be ordered and organized within the sieve. The concatenation of functions will still uniquely identify each basic data element, providing an alternative method of sorting and organizing the basic data elements within the basic data sieve.

[0072] In some embodiments, the content of an element can be represented as the following expression: element =Head.*sig1.*sig2.*...sigI.*...sigN.*Tail, where "Head" is a sequence of bytes containing the element's leading bytes, "Tail" is a sequence of bytes containing the element's ending bytes, and "sig1," "sig2," "sigI," and "sigN" are various signatures or patterns or regular expressions or sequences of bytes of a certain length within the body of the element's content that characterize the element. The ".*" expression between various signatures is a wildcard expression; that is, it is a regular expression notation that allows any number of intermediate bytes of any value other than the signature following the ".*." In some embodiments, the N-tuple (sig1, sig2,...sigI,...sigN) is referred to as the element's skeletal data structure or skeleton, and can be considered a reduced essential subset or essence of the element. In other embodiments, the (N+2) tuple (Head, sig1, sig2, ... sig1, ... sigN, Tail) is referred to as the skeletal data structure or skeleton of the element. Alternatively, an N+1 tuple containing Head or Tail along with the rest of the signature may be used.

[0073] A fingerprinting method may be applied to the content of an element to determine the location of various components (or signatures) of the skeletal data structure within the content of the element. Alternatively, certain fixed offsets within the content of the element may be selected to locate the components. Other methods may be used to locate the components of the skeletal data structure, including, but not limited to, parsing the element to detect certain declared structures and locating components within that structure. Basic data elements may be ordered into sheaves based on their skeletal data structures. In other words, the various components of an element's skeletal data structure can be considered dimensions, and the concatenation of these dimensions, followed by the remaining content of each element, may be used to order and organize the basic data elements into sheaves.

[0074] In some embodiments, input data is factored into candidate elements, with the size of each candidate element being substantially larger than the size of the references required to access all such elements in the global dataset. One observation about data divided into such data chunks (and accessed content-associatively) is that the actual data is very sparse with respect to all possible values ​​that a data chunk can specify. For example, consider a 1 zettabyte dataset. Addressing every byte in this dataset requires approximately 70 bits. With a chunk size of 128 bytes (1024 bits), there are approximately 2 chunks in a 1 zettabyte dataset, so addressing all of these chunks requires 63 bits (less than 8 bytes). Note that while a 1024-bit element or chunk can have one of 2 possible values, the number of actual values ​​for a given chunk in the dataset is at most 2 (if all chunks are distinct). This indicates that the actual data is very sparse with respect to the number of values ​​that an element's content can reach or name. This enables the use of a tree structure that allows for efficient content-based lookup, allows new elements to be added efficiently to the tree structure, and is suitable for organizing very sparse data in a cost-effective manner in terms of the incremental storage required for the tree structure itself. Because there are only 263 distinct chunks in a 1 zettabyte dataset, only 63 distinguishing bits of information are needed to distinguish them from one another, but the relevant distinguishing bits are spread across the 1024 bits of the element and can occur in different places for each element. Therefore, it is not enough to simply look at a fixed 63 bits from the content to fully distinguish all elements from one another; rather, the entire content of the element needs to be involved in sorting the elements, especially in a solution that provides true content-associative access to all elements in a dataset.In a Data Distillation™ framework, it is desirable to be able to detect derivability within the framework used to order and organize the data. With all of the above in mind, a content-based tree structure (which progressively differentiates the data as more content is examined) is a suitable organization for ordering and differentiating all elements in a factorized dataset. Such a structure provides multiple intermediate levels of subtrees that can be treated as groupings of derivable elements or groupings of elements with similar properties of derivability. Such a structure can be extended hierarchically with metadata characterizing each subtree or each element of the data. Such a structure can effectively communicate the overall composition of the data it contains, including the density, proximity, and distribution of actual values ​​within the data.

[0075] In some embodiments, base data elements are organized within a sheave in a tree configuration. Each base data element has a distinct "name" constructed from the entire content of that base data element. This name is designed to be sufficient to uniquely identify the base data element and distinguish it from all other elements in the tree. There are several ways in which a name can be constructed from the content of a base data element. The name may simply consist of all the bytes of the base data element, with these bytes appearing in the name in the same order as they exist in the base data element. In another embodiment, certain fields or combinations of fields referred to as dimensions (fields and dimensions as defined above) are used to form the first bytes of the name, with the remaining content of the base data element forming the remainder of the name, so that the entire content of the base data element participates in creating the element's complete, unique name. In yet another embodiment, fields of the element's skeletal data structure are selected as dimensions (fields and dimensions as defined above), with these fields used to form the first bytes of the name, with the remaining content of the base data element forming the remainder of the name, so that the entire content of the base data element participates in creating the element's complete, unique name.

[0076] In some embodiments, the names of elements may be calculated by performing an algebraic or arithmetic transformation on the elements while maintaining the property that each name uniquely identifies each element. In one such embodiment, each element is divided by a prime polynomial or some selected number or value, and the result of the division (which is a quotient and remainder pair) is used to form the name of the element. For example, the bits containing the remainder may form the first byte of the name, followed by the bits containing the quotient. Alternatively, the bits containing the quotient may be used to form the first byte of the name, followed by the bits containing the remainder. For a given divisor used to divide the input elements, the quotient and remainder pair will uniquely identify the element, and so this pair may be used to form the name of each element. Using this name formulation, basic data elements may be organized into sieves based on their names. The names will still uniquely identify each basic data element and provide an alternative way of sorting and organizing basic data elements within a basic data sieve.

[0077] In another embodiment, a variation of this name generation method (involving division and extraction of a quotient / remainder pair) may be used, in which division by a suitable polynomial or number or value may be performed on successive fields or successive portions of each element's content to yield successive portions of each element's name (each portion being a suitably ordered concatenation of bits from the quotient and remainder generated by the division operation on that portion or field). Note that a different polynomial may be used for the division for each field. Using this name formulation, basic data elements may be organized into sieves based on their names. The names will still uniquely identify each basic data element and provide an alternative way of sorting and organizing basic data elements within a basic data sieve.

[0078] The name of each base data element is used to order and organize the base data elements into a tree. It is expected that for most practical data sets, and even data sets that are very large in size (e.g., a 1 zettabyte data set consisting of 258 elements of 4KB size), a small subset of the name bytes will often serve to sort and order the majority of the base data elements in the tree.

[0079] 3A, 3B, 3C, 3D, and 3E illustrate various data organization systems that may be used to organize basic data elements based on their names, according to some embodiments described herein.

[0080] Figure 3A shows a trie data structure in which base data elements are organized into progressively smaller groups based on the values ​​of consecutive bytes from each base data element's name. In the example shown in Figure 3A, each base data element has a distinct name constructed from the entire contents of that base data element; this name consists simply of all the bytes of the base data element, which appear in the name in the same order as they appear in the base data element. The root node of the trie represents all base data elements. Other nodes of the trie represent subsets or groups of base data elements. Starting with the root node or first level of the trie (labeled root 302 in Figure 3A), the base data elements are grouped into subtrees based on the value of the most significant byte of their names (labeled N1 in Figure 3A). All base data elements with the same value in the most significant byte of their names are grouped together in a common subtree, and a link indicating that value exists from the root node to the node representing that subtree. For example, in Figure 3A, node 303 represents a subtree or group of basic data elements that each have the same value 2 in their most significant byte N1 of their name. In Figure 3A, this group includes basic data elements 305, 306, and 307.

[0081] At the second level of the trie, each group of basic data elements is further divided into smaller subgroups using the second most significant byte of each basic data element's name. For example, in Figure 3A, the group of basic data elements represented by node 303 is further subdivided into subgroups using the second most significant byte N2. Node 304 represents the subgroup of basic data elements that have a value of 2 in their most significant byte N1 and a value of 1 in their second most significant byte N2 of their name. This subgroup includes basic data elements 305 and 306.

[0082] The subdivision process continues at each level of the trie, creating links from parent nodes to each child node, where the child nodes represent a subset of the base data elements represented by the parent node. This process continues until only distinct base data elements exist at the leaves of the trie. Leaf nodes represent groups of leaves. In FIG. 3A, node 304 is a leaf node. The group of base data elements represented by node 304 includes base data elements 305 and 306. In FIG. 3A, this group is further subdivided into distinct base data elements 305 and 306 using the third most significant byte of their names. A value of N3=3 leads to base data element 305, and a value of N3=5 leads to base data element 306. In this example, only three significant bytes of their full names are sufficient to fully identify base data elements 305 and 306. Similarly, only two significant bytes from the name are sufficient to identify base data element 307.

[0083] This example shows that for a given mix of base data elements, only a subset of the name bytes serves to identify the base data elements in the tree, and the entire name is not required to arrive at a unique base data element. Also, base data elements or groups of base data elements may each require a different number of significant bytes to make them uniquely identifiable. Thus, the depth of the trie from the root node to the base data elements may vary for each base data element. Furthermore, in the trie, each node may have a different number of links descending into lower subtrees.

[0084] In such a trie, each node has a name consisting of a sequence of bytes that specifies how to reach this node. For example, the name for node 304 is "21". Also, the subset of bytes from an element's name that uniquely identifies an element in the current distribution of elements in the tree is the "path" from the root node to this base data element. For example, in Figure 3A, path 301 with value 213 identifies base data element 305.

[0085] The trie structure described herein can create deep trees (ie, trees with many levels) because every distinct byte in the name of an element in the tree adds one level of depth to the trie.

[0086] Note that the tree data structures in Figures 3A-3E are drawn from left to right. Thus, as one moves from the left side of the diagram to the right side of the diagram, one moves from a higher level in the tree to a lower level in the tree. For any child below a given node (i.e., toward the right side of a given node in Figures 3A-3E) selected by a given value in the distinguishing byte from its name, all elements present in the subtree below that child have the same value in that corresponding byte in the element's name.

[0087] Next, a method for content-associative lookup of a trie structure is described, given an input candidate element. The method involves navigating the trie structure using the name of the candidate element, followed by analysis and screening to determine what should be returned as the result of the overall content-associative lookup. In other words, the trie navigation process returns a first result, which is then analyzed and screened to determine the result of the overall content-associative lookup.

[0088] To begin the trie navigation process, the value of the most significant byte from the name of the candidate element is used to select a link (indicated by its value) from the root node to a successor node representing a subtree of base data elements that have that same value in the most significant byte of their name. Proceeding from this node, one level deeper (or lower) into the trie is selected by examining the second byte from the name of the candidate element and selecting the link indicated by that value, selecting a smaller subgroup of base data elements that now share at least two significant bytes from their name with the candidate element. This process continues until a base data element is reached or until no links match the value of the corresponding byte from the name of the candidate element. Under either of these conditions, the tree navigation process terminates. Once a base data element is reached, it can be returned as the result of the trie navigation process. Otherwise, one alternative is to report "missing." Another alternative is to return multiple base data elements in the subtree rooted at the node where navigation terminated.

[0089] Once the trie navigation process is complete, the results of the trie navigation process may be analyzed and screened using other criteria and requirements to determine what should be returned as the result of the content associative lookup. For example, if a base data element or multiple base data elements are returned by the trie navigation process, there may be an additional requirement that they share a certain minimum number of bytes with the name of the candidate element before they are eligible to be returned as the result of the content associative lookup (otherwise, the content associative lookup returns missing). Another example of a screening requirement may be that if the trie navigation process completes without reaching a single base data element, such that multiple base data elements (rooted at the node where the trie navigation ended) are returned as the result of the trie navigation process, these multiple base data elements are eligible to be returned as the result of the overall content associative lookup only if the number of these elements is below a certain specified limit (otherwise, the content associative lookup returns missing). A combination of multiple requirements may be used to determine the result of the content associative lookup. In this way, the lookup process either reports "missing" or returns one base data element, or if not one base data element, returns a set of base data elements that are likely to be a good starting point for deriving candidate elements.

[0090] Figures 3B-3E, described below, relate to variations and modifications of the tree data structure shown in Figure 3A. These variations provide improvements and advantages over the trie data structure shown in Figure 3A, but the process for navigating the data structure is similar to the process described above with reference to Figure 3A. That is, after tree navigation for the tree data structure shown in Figures 3B-3E is completed, subsequent analysis and screening is performed to determine the results of the overall content associative lookup, and the overall process returns either a missing base data element, or a set of base data elements that are likely to be a good starting point for deriving candidate elements.

[0091] Figure 3B illustrates another data organization system that can be used to organize base data elements based on their names. In the example shown in Figure 3B, each base data element has a distinct name constructed from the entire contents of that base data element, consisting simply of all the bytes of that base data element, with these bytes appearing in the name in the same order as they appear in the base data element. Figure 3B illustrates a more compact structure in which a link creates a subdivision or next-level grouping using multiple bytes from the names of base data elements in the subtree below (rather than the single byte used in the trie of Figure 3A). Links from parent nodes to child nodes are now represented by multiple bytes. Furthermore, from any given parent node, each link may use a different number of bytes to distinctly identify the subtree associated with that link. For example, in Figure 3B, the link from the root node to node 308 is distinguished by using four bytes from its name (N1N2N3N4 = 9845), while the link from the root node to node 309 is distinguished by using three bytes from its name (N1N2N3 = 347).

[0092] Note that during tree navigation (using content from a given candidate element), upon arriving at any parent node in the tree, the tree navigation process must examine enough bytes from the candidate element's name to ensure that it can unambiguously determine which link to select. To select a given link, the bytes from the candidate's name must match all the bytes that indicate a transition to that particular link. Again, in such a tree, each node in the tree has a name consisting of a sequence of bytes that specifies how to reach this node. For example, the name of node 309 might be "347" because it represents a group of base data elements (e.g., elements 311 and 312) whose names have the first three bytes "347." When looking up the tree using a candidate element whose name has the first three bytes as 347, this data pattern will cause the tree navigation process to reach node 309, as shown in FIG. 3B. Again, the subset of bytes from the element's name that uniquely identifies an element within the current mix of elements in the tree is the "path" from the root node to this base data element. For example, in FIG. 3B, sequence of bytes 3475 leads to basic data element 312 and uniquely identifies basic data element 312 within the mix of basic data elements shown in the example.

[0093] For diverse and sparse data, the tree structure of Figure 3B proves to be more flexible and compact than the trie structure of Figure 3A.

[0094] Figure 3C illustrates another data organization system that can be used to organize base data elements based on their names. In the example shown in Figure 3C, each base data element has a distinct name constructed from the entire contents of that base data element; the name simply consists of all the bytes of that base data element, with these bytes appearing in the name in the same order as they exist in the base data element. Figure 3C illustrates another variation (to the organization shown in Figure 3B) that further compacts the tree and groups elements into subtrees by using regular expressions (if necessary and / or useful) to specify values ​​from the names of base data elements down to various links. The use of regular expressions allows for efficient grouping of elements that share the same representation on corresponding bytes under the same subtree, followed by more localized disambiguation of distinct base data elements within that subtree. The use of regular expressions also allows for a more compact way of describing the values ​​of the bytes needed to map an element to any subtree below, further reducing the number of bytes needed to specify the tree. For example, regular expression 318 specifies a pattern of 28 consecutive "F"s, and following this link during tree navigation can reach element 314, which contains pattern 320 with 28 consecutive "F"s according to regular expression 318. Similarly, the path to reach element 316 has a link or branch that uses a regular expression that specifies a pattern with 16 consecutive "0"s. For such a tree, the tree navigation process must find and execute such regular expressions to determine which link to select.

[0095] FIG. 3D illustrates another data organization system that can be used to organize basic data elements based on their names. In the example shown in FIG. 3D, each basic data element has a distinct name constructed from the entire contents of the basic data element. A fingerprinting method is applied to each element to identify the location of the field containing the content that evaluates to the selected fingerprint. The field at the location of the first fingerprint found in the element is treated as a dimension, and a certain number of bytes from this field (e.g., x bytes, where x is substantially less than the number of bytes in the element) is extracted and used as the first bytes of the element's name. The remaining bytes of the name are composed of the remaining bytes of the basic data element, appearing in the same cyclic order as they appear in the basic data element. This name is used to organize the basic data elements into a tree. In this example, if no fingerprint is found in an element, a name is formulated simply by using all of the element's bytes in the order they appear in the element. A separate subtree (indicated by an indication that no fingerprint was found) holds and organizes all such elements based on their names.

[0096] For example, as shown in FIG. 3D , a fingerprinting technique may be applied to element 338 (containing t bytes of data, i.e., B1B2B3...Bt) to obtain a fingerprint location "Fingerprint 1" at byte Bi+1 that identifies the field selected as "Dimension 1." "Dimension 1" may then be formed by extracting x bytes from the location identified by "Fingerprint 1," and these x bytes may be used as the first bytes N1N2...Nx of the name of each element in FIG. 3D . The remaining tx bytes from element 338 (starting at Bi+x+1 and later wrapping around to B1B2B3...Bi) are then concatenated and used as the remaining bytes Nx+1Nx+2...Nt of the name. If no fingerprint is found within an element, then the name N1N2...Nt is simply B1B2B3...Bt from element 338. Basic data elements are sorted and organized into a tree using their names. For example, basic data element (PDE) 330 is identified and reached after traversing two levels of the tree using path 13654...06, where bytes 13654...0 are N1N2...Nx bytes from dimension 1. A separate subtree at node 335 reached from the root along link 334 (indicated by the indication that no fingerprint was found) holds and organizes all basic data elements whose contents did not evaluate to the selected fingerprint. Thus, in this organization, some links, such as link 336, may organize elements using names composed of the bytes of the element appearing in the same order as they appear in the element, while other links, such as link 340, may organize elements using names formulated using fingerprints.

[0097] Upon receiving a candidate element, the process applies the same techniques described above to determine the name of the candidate element and uses this name to navigate the tree for content associative lookup. Thus, the same consistent processing is applied to base data elements (as they are installed in the tree) and to candidate elements (as they are received from the parser and factorizer) to create their names. The tree navigation process uses the name of the candidate element to navigate the tree. In this embodiment, if the fingerprint is not found within the candidate element, the tree navigation process navigates down the subtree that organizes and contains base data elements whose content did not evaluate the fingerprint.

[0098] FIG. 3E illustrates another data organization system that can be used to organize basic data elements based on their names. In the example shown in FIG. 3E, each basic data element has a distinct name constructed from the entire content of that basic data element. A fingerprinting method is applied to each element to identify the location of fields whose content evaluates to one of two fingerprints. The field at the first occurrence of the first fingerprint (Fingerprint 1 in FIG. 3E) within the element is treated as the first dimension (Dimension 1), and the field at the first occurrence of the second fingerprint (Fingerprint 2 in FIG. 3E) is treated as the second dimension (Dimension 2). Using fingerprinting to find two distinct fingerprints on an element leads to four possible scenarios: (1) both fingerprints are found within the element; (2) Fingerprint 1 is found but Fingerprint 2 is not; (3) Fingerprint 2 is found but Fingerprint 1 is not; and (4) no fingerprints are found at all. The basic data elements can be grouped into four subtrees corresponding to each of the above scenarios. In Figure 3E, "FP1" indicates the presence of Fingerprint 1, "FP2" indicates the presence of Fingerprint 2, "~FP1" indicates the absence of Fingerprint 1, and "~FP2" indicates the absence of Fingerprint 2.

[0099] For each of the four scenarios, the name of the element is constructed as follows: (1) If both fingerprints are found, x bytes from the location identified by "Fingerprint 1" may be extracted to form "Dimension 1," and y bytes from the location identified by "Fingerprint 2" may be extracted to form "Dimension 2," and these x+y bytes may be used as the leading bytes N1N2...Nx+y of the name of each such element in FIG. 3E. Then, the remaining t-(x+y) bytes from element 348 are extracted cyclically (starting after the bytes from the first dimension), concatenated, and used as the remaining bytes Nx+y+1Nx+y+2...Nt of the name. (2) If Fingerprint 1 is found but Fingerprint 2 is not, x bytes from the location identified by "Fingerprint 1" may be extracted to form the leading dimension, and these x bytes may be used as the leading bytes N1N2...Nx of the name of each such element. Next, the remaining tx bytes from element 348 (starting from Bi+x+1 and later wrapping around to B1B2B3...Bi) are concatenated and used as the remaining bytes Nx+1Nx+2...Nt of the name. (3) If Fingerprint 2 is found but Fingerprint 1 is not, y bytes from the location identified by "Fingerprint 2" can be extracted to form the leading dimension, and these y bytes can be used as the leading bytes N1N2...Ny of the name of each such element. Next, the remaining ty bytes from element 348 (starting from Bj+y+1 and later wrapping around to B1B2B3...Bj) are concatenated and used as the remaining bytes Ny+1Ny+2...Nt of the name. (4) If no fingerprints are found within an element, the name N1N2...Nt is simply B1B2B3...Bt from element 348. Thus, there are separate subtrees for each of these four scenarios. The process for extracting the names (N1N2N3...Nt) for elements 348 can be summarized for four scenarios as follows: (1) Both Fingerprint 1 and Fingerprint 2 are found: N1-Nx ← Bi+1-Bi+x = x bytes from dimension 1 Nx+1-Nx+y ← Bj+1-Bj+y = y bytes from dimension 2 Nx+y+1...Nt = remaining bytes (from candidate element of size t bytes) = Bi+x+1 Bi+x+2Bi+x+3…BjBj+y+1Bj+y+2Bj+y+3…BtB1B2B3…Bi (2) Fingerprint 1 is found, Fingerprint 2 is not found: N1-Nx ← Bi+1-Bi+x = x bytes from dimension 1 Nx+1…Nt = remaining bytes (from candidate element of size t bytes) = Bi+x+1 Bi+x+2 Bi+x+3…Bt B1 B2 B3…Bi (3) Fingerprint 2 is found, Fingerprint 1 is not found: N1-Ny←Bj+1-Bj+y=y bytes from dimension 2 Ny+1…Nt = remaining bytes (from candidate element of size t bytes) = Bj+y+1 Bj+y+2 Bj+y+3…Bt B1 B2 B3…Bj (4) No fingerprint found: N1-Nx←B1-Bt Upon receiving a candidate element, the process applies the same techniques described above to determine the candidate element's name. In this embodiment, the four methods of name construction described above (depending on whether Fingerprint 1 and Fingerprint 2 are found) are applied to the candidate element in the same way as to the base data elements as they are input to the sieve. Thus, the same consistent processing is applied to the base data elements (as they are installed in the tree) and to the candidate elements (as they are received from the parser and factorizer) to create their names. The tree navigation process uses the candidate element's name to navigate the tree for content-associative lookup.

[0100] A successful content-associative lookup results in a base data element that has the same pattern as the candidate element at a particular dimension location. For example, if both fingerprints are found in the candidate element, the tree navigation process starts at the root node and works its way down the tree to link 354. If the candidate element has the pattern "99...3" as "dimension 1" and the pattern "7...5" as "dimension 2," the tree navigation process arrives at node 334. This leads to a subtree containing two base data elements (PDE 352 and PDE 353), which are likely targets for derivation. Additional analysis and screening (first by examining metadata, and if necessary, subsequently by fetching and examining the actual base data elements) is performed to determine which base data elements are best suited for derivation. Thus, the embodiments described herein identify various tree structures that may be used within a sieve. Base data elements may be organized using a combination of such structures or variations thereof. In some embodiments, base data elements are organized in a tree form, with the entire content of the element being used as the element's name. However, the order in which bytes appear in an element's name is not necessarily the order in which those bytes appear in the element. Certain fields of the element are extracted as dimensions and used to form the first byte of the name, and the remaining bytes of the element make up the rest of the name. Using these names, the elements are ordered in a tree within the sieve. The first digits of the name are used to distinguish between higher branches (or links) of the tree, and the remaining digits are used to progressively distinguish all branches (or links) of the tree. Each node in the tree may have a different number of links emanating from it. Also, each link from a node may be distinguished and represented by a different number of bytes, and the description of these bytes can be achieved using regular expressions and other powerful methods to express their specifications. All of these features result in a compact tree structure. At the leaf nodes of the tree reside references to individual basic data elements.

[0101] In one embodiment, a fingerprinting method may be applied to the bytes comprising a basic data element. The number of bytes present at the location identified by the fingerprint may be used to create a component of the element's name. One or more components may be combined to provide a dimension. Multiple fingerprints may be used to identify multiple dimensions. These dimensions are concatenated and used as the first byte of the element's name, with the remaining bytes of the element comprising the remainder of the element's name. Because the dimensions are at the location identified by the fingerprint, this increases the likelihood that the name will be formed from consistent content from each element. Elements with identical values ​​of content in the fields located by the fingerprint are grouped together along the same leg of the tree. In this way, similar elements are grouped together in a tree data structure. Elements within which no fingerprint is found may be grouped together in separate subtrees using alternative formulations of their names.

[0102] In one embodiment, a fingerprinting method may be applied to the content of an element to determine the location of various components (or signatures) of the skeletal data structure (described above) within the content of the element. Alternatively, certain fixed offsets within the content of the element may be selected to locate the components. Other methods may be used to locate the components of an element's skeletal data structure, including, but not limited to, parsing the element to find certain declared structures and locating components within those structures. The various components of an element's skeletal data structure can be considered dimensions, and the concatenation of these dimensions, followed by the remaining content of each element, is used to create the name of each element. The names are used to order and organize the basic data elements into a tree.

[0103] In another embodiment, elements are parsed to discover a certain structure of the element. Certain fields within this structure are identified as dimensions. Multiple such dimensions are concatenated and used as the first byte of the name, with the remaining bytes of the element comprising the remainder of the element's name. Because the dimensions are in positions identified by parsing the element to discover its structure, this increases the likelihood that the name will be formed from consistent content from each element. Elements with identical values ​​of content in the fields located by parsing are grouped together along the same leg of the tree. Thus, again, similar elements are grouped together in a tree data structure.

[0104] It is worth noting that in a basic data sieve, there is a relationship between the distance threshold and the depth of the resulting tree. For example, if all input elements are fixed-size elements of 4096 bytes and the distance threshold is set to 60%, this would mean that reconstructor programs up to 60% of 4 KB, or a maximum of 2458 bytes, are allowed. In such a case, the maximum depth of the basic data sieve in bytes would be approximately 40% of the element's size, or 1638 bytes. The remaining bytes can simply be provided by the candidate element's bytes and placed in the reconstructor program. This means that the maximum number of bytes in the name of an incoming candidate element that will need to be used to traverse the tree data structure to arrive at a base data element that satisfies the derivation is approximately 1638 bytes. Note that in the actual derivation created, it is necessary to accommodate the bytes required to specify various offsets; in this case, strings from the candidate element are appended to portions of the base data element to create the derivation.

[0105] In some embodiments, each node in the tree data structure includes a self-describing specification. A tree node has one or more children. Each child entry includes information about the distinguishing byte on the link to that child and a reference to that child node. A child node can be a tree node or a leaf node. Figure 3F presents a self-describing tree node data structure according to some embodiments described herein. The tree node data structure shown in Figure 3F (A) includes information related to the path from the root node to this tree node, including all or a subset of the following components: the actual sequence of bytes to get to this tree node from the name, the number of name bytes consumed to get to this node from the root node, an indication of whether this number of bytes consumed is greater than some pre-specified threshold, and other metadata that describes the path to this node and is useful for content-associative searches of the tree and for decisions related to building the tree; (B) specifies the number of children the node has; and (C) for each child (each child corresponding to a branch of the tree), specifies: (1) the child ID; (2) the number of distinguishing bytes required from subsequent bytes of the name to move down this link in the tree; (3) a specification of the actual value of the bytes from the name that bring it down this link; and (4) a reference to the child node.

[0106] FIG. 3G presents a self-describing leaf node data structure according to some embodiments described herein. A leaf node has one or more children. Each child is a link to a base data element. Each child entry includes information about the distinguishing byte on the link to the base data element, a reference to the base data element, a count of duplicates and derivations, and other metadata about the base data element. The leaf node data structure shown in FIG. 3G includes: (A) information related to the path from the root node to this leaf node, including all or a subset of the following components: the actual sequence of bytes to get to this leaf node from the name, the number of name bytes consumed to get to this node from the root node, an indication of whether this number of bytes consumed is greater than some pre-specified threshold, and other metadata describing the path to this node and useful for content-associative searches of the tree and for decisions related to building the tree; (B) specifying the number of children the node has; and (C) a list of each child (each child corresponds to one base data element under the leaf node). ), specifies (1) the child ID, (2) the number of distinguishing bytes required from the trailing bytes of the name to move down this link of the tree to the base data element, (3) a specification for the actual value of the byte from the name that moves it down this leg, (4) a reference to the base data element that ends the tree on this path of the tree, (5) a count of how many duplicates and derivatives point to this base data element (this is used to ascertain whether an entry can be removed from a sieve when data in the storage system is deleted), and (6) other metadata about the base data element, including the size of the base data element.

[0107] To increase the efficiency with which new base data elements are installed in a tree, in some embodiments, the leaf node data structure incorporates an additional field for each base data element maintained in the leaf nodes of the tree. Note that when a new element needs to be inserted into a tree, additional bytes of the name or content of each of the base data elements in the target subtree may be needed to determine where in the subtree the new element should be inserted, or to determine whether to trigger further partitioning of the subtree. Because these additional bytes are needed, it may be necessary to fetch some of the target base data elements to extract the relevant distinguishing bytes for each of these elements for the new element. To reduce and optimize (and in most cases completely eliminate) the number of IOs required for this task, the data structure in the leaf node includes a certain number of additional bytes from the name of each base data element under that leaf node. These additional bytes are called navigation lookahead bytes and serve to sort base data elements relative to the new incoming element. The navigation lookahead bytes for a given base data element are installed in the leaf node structure when the base data element is installed in the sieve. The number of bytes to retain for this purpose may be selected statically or dynamically using a variety of criteria, including the depth of the subtree involved and the density of base data elements within that subtree. For example, for base data elements that are being installed at a shallow level in the tree, the solution may add a longer navigation lookahead field than for base data elements that reside in a very deep tree. Also, if a new element is being installed in a sieve and there are already many base data elements in the existing target subtree (high likelihood of impending repartitioning), additional navigation lookahead bytes may be retained for the new base data element while it is being installed in the subtree.

[0108] 3H presents a leaf node data structure for a leaf node that includes a navigation lookahead field. This data structure includes: (A) information related to the path from the root node to this leaf node, including all or a subset of the following components: the actual sequence of bytes to get to this leaf node from the name, the number of name bytes consumed to get to this node from the root node, an indication of whether this number of bytes consumed is greater than some pre-specified threshold, and other metadata that describes the path to this node and is useful for content-associative searches of the tree and for decisions related to building the tree; (B) specifies the number of children the node has; and (C) for each child (each child corresponding to one base data element under the leaf node), (1) the child ID, (2) the number of bytes to traverse this link down the tree to the base data element. (3) a specification for the actual value of the bytes that make it down this leg; (4) a reference to the base data element that ends the tree on this path of the tree; (5) a navigation lookahead field that specifies how many bytes of navigation lookahead are kept for the base data element and the actual value of those bytes; (6) a count of how many duplicates and derivatives point to this base data element (this is used to ascertain whether an entry can be removed from a sieve when data in the storage system is deleted); and (7) other metadata about the base data element, including the size of the base data element.

[0109] In some embodiments, various branches of the tree map various data elements into groups or ranges formed by interpreting the distinction byte along the links to child subtrees as range delimiters. All elements in that child subtree have a value for the corresponding byte in the element that is less than or equal to the value of the distinction byte specified in the link to the particular child subtree. Each subtree thus represents a group of elements whose values ​​fall within a particular range. Within a given subtree, each subsequent level of the tree progressively divides the set of elements into smaller ranges. This embodiment provides a different interpretation for the components of the self-describing tree node structure shown in FIG. 3F. The N children in FIG. 3F are ordered by the value of their distinction byte within the tree node data structure and represent an ordered sequence of non-overlapping ranges. For N nodes, there are N+1 ranges, with the lowest or first range containing values ​​less than or equal to the smallest entry and the N+1st range containing values ​​greater than the Nth entry. The N+1st range is treated as out of range, so N links lead to N lower subtrees or ranges.

[0110] For example, in Figure 3F, Child 1 defines the lowest range, using six bytes (with abef12d6743a value) to delimit the range; Child 1's range is 00000000 to abef12d6743a. If the corresponding six bytes of a candidate element fall within this range, inclusive of the end value, the link for this child is selected. If the corresponding first six bytes of a candidate element are greater than the range delimiter abef12d6743a, Child 1 is not selected. To determine whether a candidate falls within Child 2's range, two conditions must be met: first, the candidate must be outside the range of the immediately preceding child (Child 1 in this example); and second, the corresponding bytes in its name must be less than or equal to Child 2's range delimiter. In this example, Child 2's range delimiter is described by two bytes with a value of dcfa. Therefore, the corresponding two bytes for a candidate element must be less than or equal to dcfa. Using this method, the candidate element and all children in the tree node can be examined to see which of the N+1 ranges the candidate element falls into. In the example shown in Figure 3F, a missing condition is detected if the corresponding 4 bytes of the candidate element's name are greater than the value of the distinction byte for the link to child N, which is f3231929.

[0111] The tree navigation process may be modified to accommodate this new range node. Upon arriving at a range node, to select a given link emanating from that node, a byte from the candidate's name must fall within a range defined for that particular link. If the value of a byte from the candidate's name is greater than the value of the corresponding byte in all links and the candidate element is outside all ranges spanned by the subtree below—in this case (referred to as an "out-of-range condition"), a missing condition is detected and the tree navigation process terminates. If the leading byte of the candidate element's name falls within a range determined by the corresponding distinction byte along the links leading to the child subtree, tree navigation continues down that subtree. Unless terminated due to an "out-of-range condition," tree navigation may continue progressively deeper down the tree until a leaf node data structure is reached.

[0112] This type of range node may be used in a tree structure in conjunction with the trie nodes described in Figures 3A-3E. In some embodiments, the upper nodes a certain number of levels in the tree structure may be trie nodes, with traversal of the tree being based on an exact match of the first byte of the candidate element's name with the corresponding byte along the tree links. Subsequent nodes may be range nodes, with traversal of the tree being determined by the range within which the corresponding byte of the candidate element's name falls. Once the tree navigation process is complete, various criteria may be used to determine what should be returned as the result of the overall content associative lookup, as described above in this document.

[0113] The foregoing descriptions of methods and apparatus for representing and using tree nodes and leaf nodes have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Accordingly, many modifications and variations will be apparent to those skilled in the art.

[0114] When a candidate element is presented as input, the above tree node and leaf node structure can be traversed, and a content-associative lookup of the tree can be performed based on the content of the candidate element. The name of the candidate element is constructed from the bytes of the candidate element, similar to how the name of a base data element was constructed from the content of the base data element when it was installed in a sieve. Given an input candidate element, a method for content-associative lookup of the tree involves navigating the tree structure using the name of the candidate element, followed by analysis and screening to determine what should be returned as the result of the overall content-associative lookup. In other words, the tree navigation process returns a first result, which is then analyzed and screened to determine the result of the overall content-associative lookup.

[0115] If there is a base data element with the same leading bytes of the name as the candidate (or such bytes that they fall within the same range), the tree identifies that subset of base data elements in the form of a subtree of the element indicated by the link. In general, each tree node or leaf node may store information that enables a tree navigation process to determine which outgoing link, if any, to select to navigate to the next lower level in the tree based on the corresponding bytes of the incoming element's name and the identity of the node reached when the tree is navigated along the selected link. If each node contains this information, the tree navigation process may recursively navigate down each level in the tree until no match is found (at which point it can return the set of base data elements that exist in the subtree rooted at the current node) or until a base data element is reached (at which point it can return the base data element and any associated metadata).

[0116] Once the tree navigation process is complete, the results of the tree navigation process may be analyzed and screened using other criteria and requirements to determine what should be returned as the result of the overall content associative lookup. First, the base data element with the most leading bytes from its name in common with the candidate may be selected. Second, if a base data element or elements are returned by the tree navigation process, there may be an additional requirement that they share a certain minimum number of bytes with the name of the candidate element before they are eligible to be returned as the result of the content associative lookup (otherwise, the content associative lookup will return null). Another example of a screening requirement may be that if the tree navigation process completes without reaching any base data elements, such that multiple base data elements (rooted at the node where the tree navigation terminated) are returned as the result of the tree navigation process, these multiple base data elements are eligible to be returned as the result of the overall content associative lookup only if the number of these elements is below a certain specified limit, such as 4 to 16 elements (otherwise, the content associative lookup will return null). A combination of multiple requirements may be used to determine the result of the content associative lookup. If multiple candidates still remain, the navigation lookahead bytes and associated metadata may be examined to determine which base data element is the best fit. If the selection still cannot be narrowed to one base data element, multiple base data elements may be provided to the derivation function. In this way, the lookup process will either report "missing" or return one base data element, or if not one base data element, a set of base data elements that are likely to be a good starting point for deriving a candidate element.

[0117] The tree needs to be designed for efficient content-associative access. A balanced tree provides similar access depth for most of the data. It is expected that the upper levels of the tree will often reside in processor cache, the next few levels in fast memory, and subsequent levels in flash storage. For very large data sets, one or more levels may need to reside in flash storage or even on disk.

[0118] FIG. 4 shows an example of how 256 TB of elementary data may be organized into a tree, according to some embodiments described herein, and illustrates how the tree may be laid out in memory and storage. Assuming an average fanout of 64 (26) children per node, a reference to a elementary data element may be accessed by reaching a leaf node data structure (e.g., as shown in FIG. 3H ), which resides (on average) at the sixth level of the tree (i.e., after five link traversals or hops). Thus, after five hops, such a structure at the sixth level of the tree contains another 230 such nodes, each with an average of 64 children (these children are references to elementary data elements), thus containing approximately 64 billion elementary data elements. With a 4 KB element size, this accommodates 256 TB of elementary data elements.

[0119] The tree may be laid out to allow traversal of six levels of the tree as follows: three levels residing in on-chip cache (containing approximately 4000 "top-level" tree node data structures that specify transitions for links to approximately 256K nodes), two levels in memory (containing 16 million "mid-level" tree node data structures that specify transitions for links to approximately 1 billion leaf nodes), and a sixth level in flash storage (containing 1 billion leaf node data structures). The 1 billion leaf node data structures residing at this sixth level of the tree in flash storage provide references to 64 billion basic data elements (an average of 64 elements per leaf node).

[0120] In the example shown in FIG. 4, at the fourth and fifth levels, each node spends an average of 16 bytes / element (1 byte for the child ID, a 6-byte reference to, say, the PDE, and another byte for the byte count, and another 8 bytes on average to specify the actual transition bytes, and some metadata). At the sixth level, each leaf node spends an average of 48 bytes / element (1 byte for the child ID, 1 byte for the byte count, 8 bytes to specify the actual transition bytes, a 6-byte reference to the base data element, 1 byte for the count of derivatives from this base data element, 16 bytes for the navigation lookahead, 2 bytes for the size of the base data element, and 13 bytes of other metadata); therefore, the total capacity in flash storage required for the tree (including references to base data elements and including any metadata) is approximately 3 terabytes. The total capacity required for higher nodes in the tree is a fraction of this size (because there are fewer nodes, fewer bytes are needed to specify closer references to child nodes, and less metadata is needed per node). In this example, the upper tree nodes consume an average of 8 bytes / element (1 byte for the child ID, 1 byte for the byte count, and an additional 3-4 bytes on average to specify the actual transition bytes, and 2-3 bytes for the child node references). Overall, in this example, a synthetic dataset with 256TB of base data is sorted into 1 billion groups using an additional unit of 3TB (or 1.17% of 256TB).

[0121] In the example shown in Figure 4, where 256TB of basic data contains 64 billion basic data elements of 4KB each, an address of less than 5 bytes (or 36 bits) is required to fully distinguish between the 64 billion basic data elements. From a content associative perspective, if the data mix is ​​such that an average of 4 bytes of incremental names are consumed at each of the first three levels and 8 bytes are consumed at each of the next three levels, then a total of 36 bytes (288 bits) of names (on average) will distinguish all 64 billion basic data elements. These 36 bytes are less than 1% of the 4KB that make up each element. If a 4KB basic data element can be identified by 1% (or even 5-10%) of its bytes, then the remaining bytes (which make up the majority of the bytes) can tolerate fluctuations, and candidates with such fluctuations can still reach this basic data element and be considered for derivation from it.

[0122] Note that the number of bytes required on any given link (to distinguish between various subordinate subtrees) is governed by the actual data in the mix of elements that comprise the dataset. Similarly, the number of links leaving a given node will vary with the data. The self-describing tree node and leaf node data structures declare the actual number and value of bytes required per link, as well as the number of links emanating from any node.

[0123] Further controls can be applied to limit the amount of cache, memory, and storage consumed at various levels of the tree, allowing input to be sorted into as many distinct groups as possible within the allocated budget of incremental storage. To address situations where density and pockets of data exist that require very deep subtrees to fully distinguish elements from one another, such density can be efficiently addressed by grouping a large set of related elements into flat groups at a certain depth in the tree (e.g., the sixth level) and performing streamlined search and derivation on these (first examining navigation lookahead and metadata to determine the best base data element, or (as a fallback) by looking only for duplicates rather than the full derivation provided by the method for the remaining data). This avoids the creation of very deep trees. Another alternative is to allow deep trees (with many levels) as long as these levels fit into available memory. As soon as a deeper level spills to flash or disk, steps can be taken to flatten the tree from that level forward to minimize latency, which would otherwise be incurred by multiple successive accesses of deeper levels of tree nodes stored in flash or disk.

[0124] It is expected that in many cases, a relatively small fraction of the total bytes from the name of each base data element will be sufficient to identify it. Studies performed on various real-world datasets using the embodiments described herein have confirmed that a small subset of the bytes of the base data elements serves to order the majority of the elements and enable a solution. Such a solution is therefore efficient in terms of the amount of storage required for its operation.

[0125] In terms of the accesses required for the example of Figure 4, for each 4KB chunk of input (or candidate element) received, the scheme requires the following accesses to query the tree structure and reach the leaf node: three cache lookups, two memory lookups (or possibly multiple memory lookups), and one IO from flash storage to access the leaf node data structure. This single IO from storage fetches a 4KB page, which holds the leaf node data structure information for a group of approximately 64 elements, including 48 bytes spent on the target base data element. These 48 bytes contain metadata about the target base data element. This completes the tree lookup process. The number of IOs required then depends on whether the candidate element is a duplicate, a derivative, or a new base data element to be installed in the sieve.

[0126] A candidate element that is a duplicate of a base data element requires one IO to fetch the base data element and verify the duplicate. Once the duplicate is verified, there is another IO to update the metadata in the tree. Thus, fetching a duplicate element requires two IOs after the tree lookup, for a total of three IOs.

[0127] A candidate element that fails the tree lookup and is neither a duplicate nor a derivative requires one more IO to store the element as a new base data element in the sieve, and one more IO to update the metadata in the tree. Thus, fetching a candidate element that fails the tree lookup requires two IOs after the tree lookup, for a total of three IOs. However, for candidate elements for which the tree lookup process completes without requiring any storage IO, fetching such a candidate element requires only two IOs in total.

[0128] A candidate element that is a derivative (but not a duplicate) first requires one IO to fetch the base data elements needed to compute the derivation. In most cases, it is expected that the derivation will be from a single base data element (rather than multiple), so it is expected that only one IO will be needed to fetch the base data element. Following successful completion of the derivation, another IO is required to store the reconstruction program and derivation details in the entry created for the stored element, and yet another IO to update the metadata in the tree (such as counts) to reflect the new derivation. Therefore, fetching a candidate element that will be a derivative requires three more IOs after the first tree lookup, for a total of four IOs.

[0129] In summary, it takes approximately 3-4 IOs to ingest a candidate element and apply the Data Distillation™ method to it (while leveraging redundancy globally across a very large dataset). This is typically just one more IO per candidate element compared to what traditional data deduplication techniques require, in return for leveraging redundancy globally across the dataset at a granularity finer than the element itself.

[0130] A storage system offering 250,000 random IO accesses per second (representing 1 GB / sec random access bandwidth to 4 KB pages) can ingest and run the Data Distillation™ method on approximately 62,500 input chunks per second (250,000 divided by four IOs per input chunk, each with an average size of 4 KB). This allows for an ingest rate of 250 MB / sec while using up the full bandwidth of the storage system. Even if only half of the storage system's bandwidth is used (thus leaving the other half available for accessing stored data), such a Data Distillation™ system can still provide an ingest rate of 125 MB / sec. Thus, given sufficient processing power, a Data Distillation™ system can exploit redundancy globally across the entire data set (at a granularity finer than the element itself) with lean IO and provide data reduction at ingest rates of hundreds of megabytes per second for current storage systems.

[0131] Thus, as confirmed by test results, the embodiments described herein accomplish the complex task of searching vast data stores for elements from which input elements can be derived, with minimal storage required to specify the derivation, with no wasteful IO accesses and minimal incremental storage required on the device. Constructed in this way, this framework makes it feasible to find elements suitable for derivation using a smaller percentage of the element's total bytes, leaving a larger percentage of the bytes available for perturbation and derivation. The key insight that explains why this scheme works so effectively for most data is that the tree provides a convenient, fine-grained structure that can locate the distinguishing and identifying bytes that identify elements within a sieve; these bytes can be efficiently separated and stored within the tree structure, even though they are at different depths and locations within the data.

[0132] Figures 5A-5C show a practical example of how data can be organized using embodiments described herein. Figure 5A shows 512 bytes of input data and the results of factorization (e.g., the result of performing operation 202 of Figure 2). In this example, fingerprinting is applied to find breaks in the data, so that consecutive breaks identify candidate elements. Alternating candidate elements are shown using bold and regular font. For example, the first candidate element is "b8ac83d9dc7caf18f2f2e3f783a0ec69774bb50bbe1d3ef1ef8a82436ec43283 bc1c0f6a82e19c224b22f9b2", the next candidate element is "ac83d9619ae5571ad2bbcc15d3e493eef62054b0 5b2dbccce933483a6d3daab3cb19567dedbe33e952a966c49f3297191cf22aa3 1b98b9dcd0fb54a7f761415e", etc. The input of FIG. 5A is factorized into 12 variable-sized candidate elements as shown. The first byte of each chunk is used to order and organize the elements within the sheave. Figure 5B shows how the 12 candidate elements shown in Figure 5A can be organized into a tree as basic data elements within the sheave using their names and the tree structure described in Figure 3B. Each element has a distinct name constructed from the entire content of that element. In this example, fingerprinting is applied to find breaks between the 12 candidate elements, so the first byte of each candidate element is already aligned with the anchor fingerprint. Therefore, the first byte of each name is already constructed from the first dimension of the content anchored by this fingerprint. The first byte of the name organizes the various elements. For example, if the first byte of an element's name is equal to "0x22," then the top link is taken to select basic data element #1. Note that the various links in Figure 5B are distinguished using different numbers of bytes as described with reference to the tree data structure shown in Figure 3B.

[0133] Figure 5C shows how the 12 candidate elements shown in Figure 5A can be organized using the tree data structure described with reference to Figure 3D. Fingerprinting is further applied to the content of each element to identify secondary fingerprints within the element's content. Bytes of content extracted from the locations of the first fingerprint (already present at the boundary of each element) and the second fingerprint are concatenated to form the initial bytes of the name, which are used to organize the element. In other words, name elements are constructed as follows: bytes of data from two dimensions or fields (located by the anchor fingerprint and secondary fingerprint, respectively) are concatenated to form the initial bytes of the name, followed by the remaining bytes. As a result of this name construction choice, different sequences of bytes result in different basic data elements in Figure 5C (relative to Figure 5B). For example, to reach basic data element #4, the tree navigation process first takes the link corresponding to "46093f9d", the first byte of the field in the first dimension (i.e., the first fingerprint), and then takes the link corresponding to "c4", the first byte of the field located in the second dimension (i.e., the second fingerprint).

[0134] 6A-6C illustrate how a tree data structure may be used in the content association mappers 121 and 122 described with reference to FIGS. 1A-1C, respectively, according to some embodiments described herein.

[0135] Once the difficult problem of finding suitable base data elements (from which to attempt to derive candidate elements) is solved, the problem narrows to examining one or a small subset of the base data elements and optimally deriving candidate elements from them with the minimum storage required to specify the derivation. Other objectives include keeping the number of accesses to the storage system to a minimum and keeping derivation and reconstruction times acceptable.

[0136] The derivation unit must represent the candidate element as the result of transformations performed on one or more base data elements, and must specify these transformations as a reconstruction program that is used to recreate the derivation once the data is extracted. Each derivation may require its own program to be constructed. The function of the derivation unit is to identify these transformations and create the reconstruction program with the smallest footprint. A variety of transformations may be used, including arithmetic, algebraic, or logical operations performed on one or more base data elements or on specific fields of each element. Byte manipulation transformations, such as concatenation, insertion, substitution, and deletion of bytes in one or more base data elements, may also be used.

[0137] 7A provides an example of a transformation that may be specified in a reconstruction program according to some embodiments described herein. The transformation vocabulary specified in this example includes arithmetic operations on fields of specific lengths within elements, as well as insertion, deletion, addition, and substitution of bytes of a declared length at specified offsets within base data elements. Various techniques and operations may be used by the derivation unit to detect similarities and differences between a candidate element and one or more base data elements and to construct a reconstruction program. The derivation unit may utilize the vocabulary available in the underlying hardware to perform its functions. The end result of this work is to specify transformations in a vocabulary specified for the reconstruction program, and to do so in a manner that also enables high-speed data retrieval using a minimal amount of incremental storage.

[0138] The derivation unit can utilize the processing power of the underlying machine and work within its allocated processing budget to provide the best analysis possible within the system's cost-performance constraints. Given that microprocessor cores are more readily available and that IO access to storage is expensive, the Data Distillation™ solution is designed to utilize the processing power of current microprocessors to efficiently perform localized analysis and derivation of candidate element content from a few base data elements. The performance of the Data Distillation™ solution (for very large data) is limited in speed not by computation but by the IO bandwidth of a typical storage system. For example, two or three microprocessor cores are expected to be sufficient to perform the computations and analysis required to support an ingest rate of several hundred megabytes per second for a typical flash-based storage system supporting 250,000 IOs per second. Note that two such microprocessor cores from a current microprocessor, such as the Intel Xeon processor E5-2687W (10 cores, 3.1 GHz, 25 MB cache), is a small fraction (two-tenths) of the total computational power available from the processor.

[0139] FIG. 7B illustrates an example of the results of candidate elements being derived from a base data element, according to some embodiments described herein. Specifically, the data pattern "Elem" is a base data element stored in the base data sheave, and the data pattern "Cand" is a candidate element to be derived from the base data element. The 18 common bytes between "Cand" and "Elem" are highlighted. The reconstruction program 702 specifies how the data pattern "Cand" can be derived from the data pattern "Elem." As shown in FIG. 7B, the reconstruction program 702 illustrates how to derive "Cand" from "Elem" by using 1-byte substitution, 6-byte insertion, 3-byte deletion, and 7-byte bulk substitution. The cost of specifying the derivation is 20 bytes + 3-byte reference = 23 bytes, which is 65.71% of the original size. Note that the reconstruction program 702 shown is a human-readable representation of the program and may not be how the program is actually stored according to embodiments described herein. Similarly, other reconstruction programs based on arithmetic operations such as multiplication and addition are also shown in FIG. 7B. For example, if "Elem" is bc1c0f6a790c82e19c224b22f900ac83d9619ae5571ad2bbec152054ffffff83 and "Cand" is bc1c0f6a790c82e19c224b22f91c4da1aa0369a0461ad2bbec152054ffffff83, an 8-byte difference can be derived as shown using the multiplication (00ac83d9619ae557)*2a =

[00] 1c4da1aa0369a046. The cost of specifying the derivation is 4 bytes + 3-byte reference = 7 bytes, which is 20.00% of the original size.Alternatively, if "Elem" is bc1c0f6a790c82e19c224b22f9b2ac83ffffffffffffffffffffffffffffffb283 and "Cand" is bc1c0f6a790c82e19c224b22f9b2ac83000000000000000000000000000002426, then a 16-byte difference can be derived using addition, e.g., by adding 0x71a3 to the 16-byte region starting at offset 16 and truncating the carry, as shown. The cost of specifying the derivation is 5 bytes + 3-byte reference = 8 bytes, which is 22.85% of the original size. Note that the sample encoding in Figure 7A is chosen for illustrative purposes only. The example in Figure 7B has a data size of 32 bytes, so 5 bits are sufficient for the length and offset fields in the element. For larger elements (e.g., 4KB elements), the size of these fields needs to be increased to 12 bits. Similarly, the sample encoding accommodates a reference size of 3 bytes or 24 bits. This should make it possible to reference 16 million basic data elements. If a reference needs to be able to address any location within, say, 256TB of data, the reference needs to be 6 bytes in size. When such a data set is factored into 4KB elements, the 6 bytes required to specify a reference is a small fraction of the size of a 4KB element.

[0140] The size of the information required to specify a derived element (derived from one or more base data elements) is the sum of the size of the reconstruction program and the size of the references required to specify the required base data element(s). The size of the information required to specify a candidate element as a derived element is referred to as the candidate's distance from the base data elements. If the candidate can be feasibly derived from any one of multiple sets of base data elements, the set of base data elements with the shortest distance is selected as the target.

[0141] When a candidate element needs to be derived from two or more base data elements (by assembling the extractions derived from each of them), the derivation unit must consider the cost of additional accesses to the storage system and weigh it against the benefits of a smaller reconstruction program and smaller distance. Once the best reconstruction program for the candidate is created, its distance is compared to a distance threshold, and if the distance does not exceed the threshold, the derivation is accepted. If the derivation is accepted, the candidate element is reformulated as a derived element and replaced with the combination of the base data elements and the reconstruction program. The entry in the distilled data created for the candidate element is replaced with the reconstruction program and one or more references to the associated base data elements. If the distance for the best derivation exceeds the distance threshold, the derivation is not accepted.

[0142] To result in data reduction, the distance threshold must always be less than the size of the candidate element. For example, the distance threshold may be set to 50% of the size of the candidate element, such that a derivative is only accepted if its footprint is less than or equal to half the footprint of the candidate element, thereby ensuring a reduction of at least two times for each candidate element for which a suitable derivative exists. The distance threshold may be a predetermined percentage or fraction based on user-specified input or selected by the system. The distance threshold may be determined by the system based on static or dynamic parameters of the system.

[0143] 8A-8E illustrate how data reduction can be performed by factoring input data into fixed-size elements and organizing the elements into the tree data structure described with reference to FIGS. 3D and 3E, according to some embodiments described herein. FIG. 8A illustrates how input data can be simply factored into 32-byte chunks. Specifically, FIG. 8A shows the first 10 chunks and several more chunks that appear after, for example, the 42 million chunks. FIG. 8B illustrates the organization of basic data elements within a sieve using names constructed such that the first byte of the name is composed of content from three dimensions within the element's content (corresponding to the location of the anchor fingerprint, secondary fingerprint, and tertiary fingerprint). Specifically, in FIG. 8B, each 32-byte chunk becomes a 32-byte candidate element (a fixed-size block). A fingerprinting method is applied to the element's content. Each element has a name that is constructed as follows: bytes of data from the element's three dimensions or fields (located by the anchor fingerprint, secondary fingerprint, and tertiary fingerprint, respectively) are concatenated to form the first byte of the name, followed by the remaining bytes of the element. The names are used to organize elements into sieves. As shown in Figure 8B, the first 10 chunks contain no duplicates or derivatives and are installed sequentially into the sieve as elements. Figure 8B shows the sieve after the 10th chunk has been consumed. Figure 8C shows the contents of the sieve at a later point in time, after consuming millions more elements of data input, for example, after the next 42 million chunks have been presented. The sieve is examined for duplicates or derivatives. Chunks that are not derivable from elements are installed into the sieve. FIG. 8C shows the sieve after 42 million chunks have been consumed, containing, for example, 16,000,010 elements (logically addressable with a 3-byte reference address), with the remaining 26,000,000 chunks being derivatives.FIG. 8D shows an example of a new input that is subsequently presented to the sieve and identified as a duplicate of an entry in the sieve (shown as element number 24,789). In this example, the sieve identifies element 24,789 (chunk 9) as the best element for chunk 42,000,011. The derivation function determines that the new chunk is an exact duplicate and replaces it with a reference to element 24,789. The cost of representing the derivation is a 3-byte reference to the original 35B, which is 8.57% of the original size. FIG. 8D shows a second example of an input (chunk 42,000,012) that is converted into a derivation of an entry in the sieve (shown as element number 187,126). In this example, the sieve determines that there is no exact match. The sieve identifies elements 187,125 and 187,126 (chunks 8 and 1) as the best elements. The new element is derived from the best element. The derivation for element 187,125 and the derivation for element 187,126 are shown in FIG. 8D. The cost representing the derivation for element 187,125 is 39 bytes + 3 byte references = 42 bytes, which is 120.00% of the original size. The cost representing the derivation for element 187,126 is 12 bytes + 3 byte references = 15 bytes, which is 42.85% of the original size. The best derivation (for element 187,126) is selected. The reconstructed size is compared to a threshold. For example, if the threshold is 50%, this derivation (42.85%) is accepted. FIG. 8E provides two additional examples of data chunks derived from base data elements, including one where a derivation is actually created by derivation from two base data elements. In the first example, chunk 42,000,013 is presented. The sieve identifies element 9,299,998 (chunk 10) as the best fit. The derivation for element 9,299,998 is shown in Figure 8E. The cost to represent the derivation is 4 bytes + 3 byte reference = 7 bytes, which is 20.00% of the original size. The reconstructed size is compared to a threshold. For example, if the threshold is 50%, then this derivation (20.00%) is accepted.In a second example, chunk 42,000,014 is presented. In this example, chunk 42,000,014 is such that half of the chunk can best be derived from element 9,299,997, and the other half of the chunk can best be derived from element 9,299,998. Therefore, a multi-derived element is created to provide further data reduction. The multi-element derivation is shown in FIG. 8E. The cost of representing this multi-derived element is 3 byte lookup + 3 bytes + 3 byte lookup = 9 bytes, which is 25.71% of the original size. The reconstructed size is compared to a threshold, and if the threshold is, for example, 50%, then this derivation (25.71%) is accepted. Note that the best result from a single derived element would have been 45.71%.

[0144] 8A-8E illustrate a key advantage of the Data Distillation™ system: it can be effective at performing data reduction while consuming and producing fixed-size blocks. Note that fixed-size blocks are highly desirable in high-performance storage systems. Using the Data Distillation™ apparatus, a large incoming input file consisting of many fixed-size blocks can be factored into many fixed-size elements, with all basic data elements being fixed-size. Potentially variable-size reconstructed data for each derived element can be packed together and maintained inline within the distilled data file, which can then be chunked into fixed-size blocks. Thus, for all practical purposes, powerful data reduction can be performed while consuming and producing fixed-size blocks within the storage system.

[0145] Figures 9A-9C show an example of the Data Distillation™ scheme first shown in Figure 1C. This scheme uses a separate base reconstruction program sieve that can be accessed content-associatively. Such a structure allows for the detection of derivations that build reconstructed programs already present within the base reconstruction program sieve. Such derivations can be reformulated to reference existing reconstructed programs. This allows for the detection of redundancies between reconstructed programs. In Figure 9A, input data is taken in. A fingerprinting method is applied to the data, and chunk boundaries are established at the fingerprint locations. The input is factored into eight candidate elements (alternating chunks shown in bold and regular font in Figure 9A) as shown. In Figure 9B, the eight candidate elements are shown organized within a sieve. Each element has a distinct name constructed from the entire content of the element. In this example, the name elements are constructed as follows: bytes of data from two dimensions or fields (located by the anchor fingerprint and secondary fingerprint, respectively) are concatenated to form the first byte of the name, followed by the remaining bytes. This name is used to order the elements within the sieve and provides content-associative access to the sieve via a tree structure. Figure 9B further illustrates a second content-associative structure including a base reconstructor. Figure 9C illustrates a duplicate reconstruction. Assume that a 55-byte candidate element (shown in Figure 9C) arrives that is not a duplicate of any base data element. Element 3 is selected as the best fit; the first two dimensions are identical for PDEs 2 and 3, but the remaining bytes, beginning with 88a7, match element 3. A new input is derived from element 3 using a 12-byte reconstructor (RP). The encoding is as shown in Figure 7A. Note that for this example, the maximum element size is 64 bits, and all offsets and lengths are encoded as 6-bit values, as opposed to the 5-bit lengths and offsets shown in Figure 7A. The base reconstructor sieve is searched and the new RP is not found.This RP is inserted into the base reconfiguration program sieve and ordered based on its value. The new element is reformulated as a reference to base data element 3 in the base reconfiguration program sieve and a reference to the newly created base reconfiguration program at reference 4. The total storage size for this derived element is 3 byte PDE reference, 3 byte RP reference, 12 byte RP = 18 bytes, which is 31.0% of the size for storing it as a PDE. Suppose then that a copy of a 55 byte candidate element arrives. As before, a 12 byte RP is created based on element 3. The base reconfiguration program sieve is searched and an RP with base RP ID = 3, RP reference = 4 is found. This candidate element is represented in the system as a reference to base data element 3 and a reference to reconfiguration program 4. The total storage size added for this derived element is 3 byte PDE reference, 3 byte RP reference = 6 bytes, which is 10.3% of the size for storing it as a PDE.

[0146] FIG. 10A provides an example of how transformations specified in a reconstruction program are applied to base data elements to result in derived elements, according to some embodiments described herein. This example shows derived elements specified to be generated from a base data element numbered 187,126 (which is also shown in the sieve of FIG. 8C) by applying four transformations (insertion, substitution, deletion, and addition) to the base data element as specified by the reconstruction program shown. As shown in FIG. 10A, element 187,126 is loaded from the sieve, and the reconstruction program is executed to derive chunk 42,000,012 from element 187,126. FIGS. 10B-10C illustrate a data retrieval process according to some embodiments described herein. Each data retrieval request essentially takes the form of an element in the distilled data and is presented to the retrieval engine in a lossless reduced format. The lossless reduced format for each element includes a reference to the associated base data element and reconstruction program. The retrieval unit of the Data Distillation™ device fetches basic data elements and reconstruction programs and provides them to the reconstruction unit for reconstruction. After the associated basic data elements and reconstruction programs for elements of the distilled data are fetched, the reconstruction unit executes the reconstruction programs to generate the elements in their original, unreduced form. The effort required by the data retrieval process to perform the reconstruction is linear with respect to the size of the reconstruction programs and the size of the basic data elements. Therefore, a high data retrieval rate can be achieved by the system.

[0147] Clearly, to reconstruct an element from its losslessly reduced form in the distilled data back to its original, unreduced form, only the base data element and the reconstruction program specified for that element need to be fetched. Therefore, accessing or reconstructing other elements is not required to reconstruct a given element. This makes the Data Distillation™ apparatus efficient at servicing random sequences of reconstruction and retrieval requests. Note that traditional compression methods, such as the Lempel-Ziv method, require fetching and decompressing an entire window of data containing the desired block. For example, if a storage system compresses a 4 KB block of data using a 32 KB window using the Lempel-Ziv method and then fetches and decompresses a given 4 KB block, the entire 32 KB window must be fetched and decompressed. This imposes a performance penalty because more bandwidth is consumed and a larger amount of data must be decompressed to provide the desired data. The Data Distillation™ apparatus does not incur such a penalty.

[0148] The Data Distillation™ appliance can be integrated into a computer system in various ways to organize and store data in a manner that efficiently discovers and exploits redundancies globally across the data in the system. Figures 11A-11G illustrate systems including a Data Distillation™ mechanism (which may be implemented using software, hardware, or a combination thereof) according to some embodiments described herein. Figure 11A presents a general-purpose computing platform with software applications running on system software that executes on a hardware platform comprised of processors, memory, and data storage components. Figure 11B illustrates a Data Distillation™ appliance integrated into the platform's application layer, with each specific application using the appliance to exploit redundancies within the data set for that application. Figure 11C illustrates a Data Distillation™ appliance used to provide a data virtualization layer or service for all applications running on top of the data virtualization layer or service. Figures 11D and 11E illustrate two different integrations of the Data Distillation™ appliance with the operating system, file system, and data management services of a sample computing platform. Other integration methods include, but are not limited to, integration with embedded computational stacks in hardware platforms such as those used in flash-based data storage subsystems as shown in FIG. 11F.

[0149] Figure 11G presents further details of the integration of a Data Distillation™ apparatus with the sample computing platform shown in Figure 11D. Figure 11G shows the components of the Data Distillation™ apparatus, with a parser and factorizer, derivator, extractor, and reconstructor running as software on a general-purpose processor, and a content-associative mapping structure residing across several levels of the storage hierarchy. The underlying data sheave may reside in a storage medium (such as a flash-based storage drive).

[0150] FIG. 11H shows how the Data Distillation™ device can be interfaced with a sample general-purpose computing platform.

[0151] A file system associates files (e.g., text documents, spreadsheets, executable files, multimedia files, etc.) with identifiers (e.g., file names, file handles, etc.) and allows operations (e.g., read, write, insert, append, delete, etc.) to be performed on files using the identifiers associated with the files. The namespace implemented by a file system may be flat or hierarchical. A namespace may also be multi-layered, e.g., a top-layer identifier may be resolved into one or more identifiers at successively lower layers until the top-layer identifier is fully resolved. In this way, a file system provides an abstraction of the physical data storage devices and / or storage media (e.g., computer memory, flash drives, disk drives, network storage devices, CD-ROMs, DVDs, etc.) that physically store the contents of files.

[0152] The physical storage devices and / or storage media used to store information in a file system may use one or more storage technologies and may reside at the same network location or may be distributed across different network locations. Given an identifier associated with a file and one or more operations requested to be performed on the file, a file system can (1) identify one or more physical storage devices and / or storage media and (2) cause the physical storage devices and / or storage media identified by the file system to perform the operations requested to be performed on the file associated with the identifier.

[0153] Each time a read or write operation is performed within a system, different software and / or hardware components may be involved. The term “reader” may refer to the collection of software and / or hardware components within a system that are involved when a given read operation is performed within the system, and the term “writer” may refer to the collection of software and / or hardware components within a system that are involved when a given write operation is performed within the system. Some embodiments of the methods and apparatus for data reduction described herein may be utilized by or incorporated into one or more software and / or hardware components of a system that are involved when a given read or write operation is performed. Different readers and writers may utilize or incorporate different data reduction implementations. However, each writer that utilizes or incorporates a particular data reduction implementation corresponds to a reader that also utilizes or incorporates the same data reduction implementation. Note that some read and write operations performed in the system may not utilize or incorporate a data reduction device. For example, when a Data Distillation™ device or data reduction device 103 retrieves a basic data element or adds a new basic data element to the basic data store, the device can directly perform read and write operations without data reduction.

[0154] 11H, writer 150W may generally refer to the software and / or hardware components of the system involved in performing a given write operation, and reader 150R may generally refer to the software and / or hardware components of the system involved in performing a given read operation. As shown in FIG. 11H, writer 150W provides input data to Data Distillation™ device or data reduction device 103 and receives distilled data 108 from Data Distillation™ device or data reduction device 103. Reader 150R provides fetch requests 109 to Data Distillation™ device or data reduction device 103 and receives fetched data output 113 from Data Distillation™ device or data reduction device 103.

[0155] Implementation examples for Figure 11H include, but are not limited to, incorporating or utilizing the Data Distillation™ or data reduction device 103 into an application, operating system kernel, file system, data management module, device driver, or firmware of a flash or disk drive, which spans the various configurations and methods of use described in Figures 11B-11F.

[0156] FIG. 11I illustrates how the Data Distillation™ unit can be used for data reduction in a block-processing storage system. In such a block-processing system, data is stored in blocks, each identified by a logical block address, or LBA. Blocks are constantly being modified and overwritten, so that new data can be overwritten in a block identified by a particular LBA. Each block in the system is treated as a candidate element, and the Data Distillation™ unit can be used to reduce the candidate element to a lossless reduced form that includes references to base data elements (stored in a particular base data element block) and, in the case of derived elements, a reference to a reconstruction program (stored in a particular reconstruction program block). FIG. 11I introduces a data structure 1151 that maps the contents of a block identified by an LBA to the corresponding elements in the lossless reduced form. For each LBA, there will be an associated element specification. For systems using fixed-size blocks, it is convenient that the incoming block, base data element block 1152, and reconstruction program block 1153 are all fixed sizes. In this system, each base data element can be stored as a separate block. Multiple reconstruction programs may be packed into a reconstruction program block, which also has the same fixed size. Because the data structure further includes, for each elementary data element and reconstruction program, a reference to the count field and associated metadata present in the leaf node data structure, when a block is overwritten with new data, the previous data present in the LBA can be effectively managed. That is, the count fields of the existing elementary data elements and reconstruction programs (that are being overwritten) must be decremented, and similarly, the counts of the elementary data elements referenced by the data received into the LBA must be incremented.By maintaining a reference to the count field within this data structure 1151, overwrites can be managed quickly, enabling high performance block processing storage systems that take full advantage of the data reduction provided by the Data Distillation™ device.

[0157] FIG. 12A illustrates the use of a Data Distillation™ device for communication of data across a bandwidth-constrained communication medium, in accordance with some embodiments described herein. In the setup shown, communication node A creates a set of files to be sent to communication node B. Node A uses the Data Distillation™ device to convert the input file into distilled data or a distilled file that includes references to reconstruction programs for base data elements and derived elements, which are installed in a base data sieve. Node A then sends the distilled file along with the base data sieve to node B. (The base data sieve may be sent before, simultaneously with, or after sending the distilled file; further, the base data sieve may be sent on the same or a different communication channel than the communication file used to send the distilled file.) Node B installs the base data sieve in a corresponding structure at its end and subsequently sends the distilled file through the extraction and reconstruction units present in node B's Data Distillation™ device, resulting in the original set of files created by node A. Thus, bandwidth-constrained communications media are used more efficiently by using Data Distillation™ devices on both ends of the media to transmit only reduced data. Furthermore, by using Data Distillation™, redundancy can be exploited to a greater extent (beyond what is feasible using conventional techniques such as Lempel-Ziv), allowing for the efficient transmission of larger files or groups of files.

[0158] We now describe the use of the Data Distillation™ appliance in a wide area network installation where a workgroup collaboratively shares data distributed across multiple nodes. When data is first created, it can be reduced and communicated as shown in FIG. 12A. The wide area network maintains copies of the data at each site, allowing for rapid local access to the data. The use of the Data Distillation™ appliance can reduce the footprint of each site. Furthermore, subsequent ingestion of new data at any site can reduce the new data by exploiting any redundancies between the new data and the contents of the existing base data sheave.

[0159] In such an installation, any modification of data at any given site must be communicated to all other sites so that each site's base data sheave is maintained consistently. Thus, as shown in FIG. 12B , updates, such as the installation and removal of base data elements, as well as metadata updates, may be communicated to each site's base data sheave in accordance with some embodiments described herein. For example, when a new base data element is installed in a given site's sheave, the base data element must be communicated to all other sites. Each site can use the value of the base data element to content-associatively access the sheave and determine where new entries need to be added within the sheave. Similarly, when a base data element is deleted from a given site's sheave, all other sites must be updated to reflect this deletion. One way this can be achieved is by communicating the base data element to all sites, along with any necessary updates to associated links in the tree and the deletion of that base data element from the sheave, so that each site can use the base data element to content-associatively access the sheave and determine which entries in the leaf nodes need to be deleted. Another way is to communicate to all sites a reference to the entry for the base data element in the leaf node where it resides.

[0160] Thus, the Data Distillation™ device can be used to reduce the footprint of data stored across various sites in a wide area network and to make efficient use of the network's communication links.

[0161] 12C-12K illustrate various components of reduced data generated by the Data Distillation™ device for various usage models, according to certain embodiments described herein.

[0162] Figure 12C shows how the Data Distillation™ device 1203 takes in a set of input files 1201 and, after the distillation process is complete, produces a set of distilled files 1205 and a base data sieve or base data store 1206. The base data sieve or base data store 1206 of Figure 12C is itself made up of two components: a mapper 1207 and a base data element (or PDE) 1208, as shown in Figure 12D.

[0163] The mapper 1207 itself has two components internally: a set of tree node data structures that define the entire tree, and a set of leaf node data structures. A set of tree node data structures may be contained in one or more files. Similarly, a set of leaf node data structures may be contained in one or more files. In some embodiments, one file, called the tree node file, holds the entire set of tree node data structures for the trees created for the basic data elements for a given dataset (input files 1201), and another file, called the leaf node file, holds the entire set of leaf node data structures for the trees created for the basic data elements for that dataset.

[0164] 12D, basic data elements 1208 includes a set of basic data elements created for a given dataset (input file 1201). A set of basic data elements may be contained in one or more files. In some embodiments, one file, called a PDE file, holds the entire set of basic data elements created for a given dataset.

[0165] Tree nodes in a tree node file contain references to other tree nodes in the tree node file. The deepest (or lowest) level of tree nodes in a tree node file contains references to entries into leaf node data structures in a leaf node file. Entries into leaf node data structures in a leaf node file contain references to basic data elements in a PDE file.

[0166] The tree node files, leaf node files, and PDE files are shown in Figure 12E, which details all components created by the device. Figure 12E shows a set of input files 1201, including N files named File 1, File 2, File 3, ... File N, which are reduced by the Data Distillation™ device to generate a set of distilled files 1205 and various components of the basic data sieve: Tree Node File 1209, Leaf Node File 1210, and PDE File 1211. Distilled files 1205 include N files named file1.dist, file2.dist, file3.dist...fileN.dist. The Data Distillation™ device factors the input data into its constituent parts and creates two categories of data elements: Basic Data Elements and Derived Elements. The distilled files contain descriptions of data elements in a lossless reduction format and include references to the basic data elements in the PDE files. Each file in input files 1201 has a corresponding distilled file in distilled files 1205. For example, file1 1212 in input files 1201 corresponds to a distilled file 1213 named file1.dist in distilled files 1205. Figure 12R shows an alternative representation of an input dataset specified as a set of input files and a directory or folder.

[0167] Note that FIG. 12E illustrates various components created by the data distiller based on the organization of distilled data and basic data sieves according to FIG. 1A, with the reconstructed programs placed in the lossless reduced representation of the elements in the distilled file. Note that in some embodiments (according to FIG. 1B), the reconstructed programs can be placed in the basic data sieves and treated like basic data elements. The lossless reduced representation of the elements in the distilled file includes a reference to the reconstructed program in the basic data sieve (rather than including the reconstructed program itself). In these embodiments, the reconstructed program is treated like a basic data element and generated in PDE file 1211. In yet another embodiment, according to FIG. 1C, the reconstructed program is stored separately from the basic data elements in a structure referred to as a reconstructed program store. In such an embodiment, the lossless reduced representation of the elements in the distilled file includes a reference to the reconstructed program in the reconstructed program store. In such an embodiment, in addition to generating tree node files 1209, leaf node files 1210, and PDE files 1211 for the tree organization of basic data elements, as shown in FIG. 12F, the device generates a second set of tree and leaf node files called reconstructed tree node files 1219 and reconstructed leaf node files 1220, along with a file containing all the reconstruction programs called RP file 1221.

[0168] The Data Distillation™ apparatus shown in Figure 12E further stores configuration and control information that governs operations in one or more of tree node file 1209, leaf node file 1210, PDE file 1211, and distillation file 1205. Alternatively, a fifth component may be generated that contains this information. As with the apparatus shown in Figure 12F, the configuration and control information may be stored in one or more of the various components shown in Figure 12F, or it may be stored in a separate component generated for this purpose.

[0169] FIG. 12G illustrates an overview of the use of the Data Distillation™ device, in which a given dataset (input dataset 1221) is sent to the Data Distillation™ device 1203 and processed to generate a lossless reduced dataset (lossless reduced dataset 1224). The input dataset 1221 may consist of a collection of files, objects, blocks, chunks, or extracts from a data stream. Note that FIG. 12E illustrates an example in which the dataset consists of files. The input dataset 1221 in FIG. 12G corresponds to the input file 1201 in FIG. 12E1, and the lossless reduced dataset 1224 in FIG. 12G includes the four components shown in FIG. 12E: the distilled file 1205, tree node file 1209, leaf node file 1210, and PDE file 1211 in FIG. 12E. In FIG. 12G, the Data Distillation™ device exploits redundancy between data elements across the range of the input dataset presented to the device.

[0170] A Data Distillation™ device may be configured to exploit redundancy across subsets of an input dataset and provide lossless reduction for each subset of data presented to the device. For example, as shown in FIG. 12H, an input dataset 1221 may be partitioned into multiple smaller collections of data, each of which is referred to in this disclosure as a "lot" or "lot of data" or "data lot." FIG. 12H illustrates a Data Distillation™ device configured to take an input data lot 1224 and generate a lossless-reduced data lot 1225. FIG. 12H illustrates an input dataset 1221 comprised of multiple collections of data, data lot 1, ... data lot i, ... data lot n. This data is presented to the Data Distillation™ device one data lot at a time, exploiting redundancy across the range of each data lot to generate the lossless-reduced data lot. For example, data lot i 1226 from input dataset 1221 is sent to the device, and lossless-reduced data lot i 1228 is provided to lossless-reduced dataset 1227. Each data lot from input dataset 1221 is sent to the device and a corresponding lossless reduced data lot is provided to lossless reduced dataset 1227. After consuming and reducing all of data lot 1, ... data lot i, ... data lot n, input dataset 1221 is reduced to lossless reduced dataset 1227.

[0171] By design, Data Distillation™ devices are already efficient at exploiting redundancy across a global scope of data, but the techniques described above may be used to further speed up and improve the efficiency of the data reduction process. The throughput of the data reduction process can be increased by limiting the size of data lots to fit within the system's available memory. For example, an input data set many terabytes or even petabytes in size can be divided into multiple data lots, each 256 GB in size, and each data lot can be rapidly reduced. Using a single processor core (Intel Xeon E5-1650 V3, Haswell 3.5 GHz processor) with 256 GB of memory, such a solution exploiting redundancy across the entire 256 GB range has been realized in our laboratory, resulting in data ingest rates of hundreds of megabytes per second while providing 2-3 times reduction levels for a variety of data sets. Note that the 256 GB range is millions of times larger than the 32 KB window size within which the Lempel-Ziv algorithm provides ingest performance of 10 MB / s to 200 MB / s on modern processors. Therefore, by appropriately limiting the extent of redundancy, an increase in the speed of the data distillation process can be achieved, potentially at the expense of some reduction.

[0172] Figure 12I shows a variation of the setup of Figure 12H, showing multiple data distillation processes running on multiple processors that greatly increases the throughput of data reduction (and data reconstruction / retrieval) of the input dataset. Figure 12I shows an input dataset 1201 partitioned into x data lots, where the x independent data lots are fed into j independent processes running on independent processor cores (each process is allocated sufficient memory to accommodate any data lot fed into it), running in parallel, resulting in approximately j times speedup for both data reduction and reconstruction / retrieval.

[0173] FIG. 12H illustrates a Data Distillation™ device configured to take an input data lot 1224 and generate a lossless reduced data lot 1225. FIG. 12H illustrates an input data set 1221, consisting of multiple data collections: data lot 1, ... data lot i, ... data lot n. In some embodiments, alternative partitioning schemes may be used to partition the input data set into multiple data lots, with data lot boundaries dynamically defined to maximize available memory. Available memory may be utilized initially to hold all tree nodes, or available memory may be utilized to hold all tree nodes and all leaf nodes of a data lot, or finally available memory may be utilized to hold all tree nodes, leaf nodes, and all basic data elements. These three different options enable alternative operating points for the device. For example, dedicating available memory to tree nodes allows a much wider range of data to be accommodated in the data lot, but this dedication incurs additional latency because the device must fetch leaf nodes and associated basic data elements from storage when needed. Alternatively, dedicating available memory to accommodate both tree nodes and leaf nodes speeds up distillation but reduces the effective size of the tree and therefore the range of data that can be accommodated in a data lot. Finally, using available memory to hold all tree nodes allows for the fastest distillation with leaf nodes and basic data elements but minimizes the size of the data lot that can be supported as a single range. In all of these embodiments, the data lot will be dynamically closed the moment the memory limit is reached, and subsequent files from the input dataset become part of the new data lot.

[0174] Further improvements exist that can improve the efficiency of the device and speed up the reconstruction process. In some embodiments, a single unified mapper is used for distillation, but instead of maintaining the basic data elements in a single PDE file, the basic data elements are maintained across N PDE files. Thus, the previously single PDE file is partitioned into n PDE files, each smaller than a certain threshold size, and each partition is created during the distillation process when the PDE file exceeds the threshold size (as it grows with the installation of basic data elements). Distillation proceeds for each input file by consulting the mapper to content-associatively select appropriate basic data elements suitable for derivation, and then deriving the appropriate basic data elements, which are fetched from the appropriate PDE file in which they reside. Each distillation file is further refined to list all those PDE files (out of the n PDE files) that contain the basic data elements referenced by the particular distillation file. To reconstruct a particular distillation file, one simply loads up or opens and accesses the listed PDE files for reconstruction. This has the advantage that to reconstruct a single distillation file or several distillation files, only those PDE files containing the basic data elements required for that particular distillation file need to be accessed or activated, without requiring other PDE files to be kept or loaded into memory or a high speed tier of storage, thus making the reconstruction faster and more efficient.

[0175] The partitioning of a PDE file into n PDE files can be further guided by criteria that localize the reference patterns made to the base data during the reduction of any file in the dataset. The system can be enhanced with a counter that counts and estimates the density of references to elements in the current PDE file. If this density is high, the PDE file will not be partitioned or split, and will continue to grow as subsequent elements are installed. As the density of references from a given distillation file gradually decreases, the PDE file can then be split and partitioned if it grows beyond a certain threshold. Once partitioned, a new PDE file is opened, and subsequent installations from subsequent distillations will be made into this new PDE file. This configuration further speeds up reconstruction when only a subset of files from a data lot need to be reconstructed.

[0176] FIG. 12J illustrates various components of reduced data generated by the Data Distillation™ device for a usage model in which mappers may no longer be retained after reduction of the input data set. Examples of such usage models include certain data backup and data archiving applications. In such usage models, subsequent use of the reduced data is reconstruction and retrieval of the input data set from the reduced data set. In such scenarios, the footprint of the reduced data can be further reduced by not storing mappers after data reduction is complete. FIG. 12J illustrates input file 1201 being sent to the device, which generates distilled file 1205 and PDE file 1211, which components comprise the reduced data in this scenario. Note that input file 1201 can be completely regenerated and recovered using only distilled file 1205 and PDE file 1211. Recall that the element-by-element lossless reduced representation in the distilled file includes a reconstruction program, if necessary, and a reference to the basic data elements in the PDE file. When combined with the PDE file, this is all the information needed to perform the reconstruction. It is also worth noting an important advantage of this configuration for the performance efficiency of reconstructing and retrieving input data sets. In this embodiment, the device factorizes the input data set into distilled files and basic data elements contained in separate PDE files. During reconstruction, the PDE files may first be loaded from storage into available memory, and then the distilled files may be sequentially read from storage for reconstruction. During reconstruction of each distilled file, any basic data elements required to reconstruct the distilled file may be quickly retrieved from memory without incurring any additional storage access latency for reading those basic data elements. Once the reconstructed distilled file is complete, it may be written to storage. This configuration eliminates the need for random storage accesses that would otherwise adversely affect performance.In this solution, loading the PDE file from storage is a set of accesses to a series of contiguous chunks of bytes, reading each distilled file is also a set of accesses to a series of contiguous chunks of bytes, and finally, each reconstructed input file is written to storage as a set of accesses to a series of contiguous chunks of bytes. The storage performance of this configuration tracks more closely with the performance of reading and writing a series of contiguous chunks of bytes than the performance of solutions that incur multiple random storage accesses.

[0177] In some embodiments, the data distiller can be refined to further improve the memory and performance efficiency of reconstruction and retrieval. During the distillation process, metadata is collected for each base data element and saved as part of the reduced data footprint. This metadata identifies these base data elements from which either duplicates or derivatives are created during the distillation process. During reconstruction, only these base data elements need to be retained in memory until all elements that are either duplicates of these base data elements or derivatives derived from these base data elements have been reconstructed. The metadata also provides, for each base data element, a total number of duplicates and derivatives derived from this base data element. This total, called the reuse count, indicates the number of times this base data element is used to reconstruct either duplicates or derivatives. At the start of dataset reconstruction, this metadata is first fetched into memory. As reconstruction progresses, base data elements are fetched from storage and used to reconstruct elements and populate the reconstructed dataset. As they are fetched, base data elements (identified by metadata) with non-zero reuse counts are retained and allocated in memory so as to be readily available (without incurring storage IO) for use in reconstructing elements that are either duplicates or derivatives of them. As the reconstruction progresses, the reuse count of a given base data element is decremented upon the reconstruction of each element that is either a duplicate or derivative of the given base data element. Upon the reconstruction of elements that are either duplicates or derivatives of one or more base data elements, the reuse count of each of these base data elements is decremented. When the reuse count of a given base data element becomes zero, the base data element no longer needs to be retained in memory, thus reducing the memory required to retain the base data element during reconstruction.The memory may be organized as a cache, and a base data element whose reuse count reaches zero may be deallocated from the cache. The reuse count of a base data element provides information about the lifetime during which the base data element can be reused for the reconstruction of a duplication or derivation, and therefore the base data element is retained in memory for that lifetime. The metadata retained at the end of the distillation process for the purpose of describing the lifetime of a base data element used for duplication or derivation is called PDE reuse and lifetime metadata.

[0178] The PDE reuse and lifetime metadata can be further refined to include the size of each of the basic data elements for which either duplicates or derivatives are created during the distillation process. The sum of the sizes of all these basic data elements provides an upper bound on the amount of memory required to hold the basic data elements during reconstruction and is called the Coarse Grain Working Set of basic data elements for a given dataset.

[0179] Note that after the distillation process, there may be base data elements in the reduced data that are not used to create duplicates or derivatives. If a reuse count were generated for such base data elements, it would be zero. During reconstruction, such base data elements do not need to be kept in memory because they will not be referenced again for the purpose of reconstructing duplicates or derivatives after being fetched from storage and sent to the reconstructed output. In such a situation, the coarse-grained working set of base data elements for the dataset will be smaller than the sum of the sizes of all base data elements in the dataset. Therefore, keeping only the coarse-grained working set in memory during reconstruction results in memory savings compared to keeping all base data elements in memory.

[0180] In fact, the actual amount of memory required to hold the basic data elements needed to reconstruct the derivative during reconstruction may be even smaller than the coarse-grained working set due to the possibility that some basic data elements may be dynamically deallocated from memory (as their reuse counts are decremented to zero) before other basic data elements are referenced for the first time during reconstruction and therefore configured to be fetched and allocated in memory (and then used for subsequent reconstruction of the derivative). Because both of these categories of basic data elements do not need to be present in memory at the same time, the actual amount of memory required during reconstruction will be smaller than the coarse-grained working set.

[0181] The above configuration allows for more efficient use of memory required to hold basic data elements during dataset reconstruction, while still eliminating random accesses to storage during reconstruction. Again, loading PDE files from storage can be a set of accesses to a series of contiguous chunks of bytes, reading each distilled file can also be a set of accesses to a series of contiguous chunks of bytes, and finally, each reconstructed input file can be written to storage as a set of accesses to a series of contiguous chunks of bytes. The storage performance of this configuration more closely tracks the performance of reading and writing a series of contiguous chunks of bytes than the performance of solutions that incur multiple random storage accesses.

[0182] The above improvements can be made to any of the distillation apparatuses shown in Figures 1A-1G, which are further illustrated in Figures 1I-1P. The basic data elements in the reduced data can be streamed from storage, whether stored in a separate PDE file (shown in Figure 1J) or stored inline within the distillation data (shown in Figure 1P).

[0183] Figure 12S shows the various components of the reduced data generated by a data distillation apparatus that implements this improvement. Figure 12S shows input file 1201 fed into this apparatus, which produces distilled file 1205, PDE file 1211, and PDE reuse and lifetime metadata file 1215. Note that in some embodiments, the PDE reuse and lifetime metadata may be included in the PDE file. Figure 12T shows the components of PDE reuse and lifetime metadata file 1215. For each basic data element used to reconstruct duplicates or derivatives, the PDE reuse and lifetime metadata file includes a handle or identifier of the basic data element 1216 that locates the basic data element in the PDE file or inline in the distilled file (if the basic data element exists inline in the distilled data). For each entry in PDE reuse and lifetime metadata 1215, reuse count 1217 provides a count of the number of times the basic data element is used to reconstruct duplicates or derivatives, and basic data element size 1218 provides the size of that basic data element.

[0184] In some embodiments, rather than allocating memory for reconstruction based on the coarse-grained working set size of the dataset, the data distiller can be further refined to help determine a tighter bound on the amount of memory required to hold the basic data elements during reconstruction. This bound is called the fine-grained working set and is determined by examining the creation and use of all basic data elements and their spatial and temporal characteristics throughout the distillation task. Additional bookkeeping and finer-grained analysis can be performed during distillation to calculate this tighter bound. This is described below.

[0185] Consider an input dataset (of a given size and spanning a range of addresses) that is factored into N elements. Distillation of this dataset involves N distillation events, each of which transforms an input element into either a base data element or a derivative (or duplicate) of a previously created base data element. Conversely, reconstructing the entire dataset involves N reconstructing events, each of which either provides a base data element as an output data element or executes a reconstructing program on a base data element to deliver an output element. The N events comprise a sequence, and the lifetime of a base data element can be described for these events in the sequence during both distillation and reconstructing. Each event may be associated with the address of an element in the original input data stream being distilled (which is also the address of the same element in the output stream when regenerated by reconstructing). This address is called the event's locator address. The locator address of each event is retained throughout the distillation process and can be made available at the end of the distillation process for an inventory step to help determine the fine-grained working set.

[0186] During the distillation process, the distillation device is further refined to record, for each basic data element, the event at which the basic data element is first created and the event at which it is last used to create a duplicate or derivative. These two events effectively define the lifetime of this basic data element. During reconfiguration, this basic data element only needs to be retained in memory for this lifetime. During reconfiguration, after the end of any basic data element's lifetime, i.e., after the last use event, that basic data element is no longer needed for the remaining reconfiguration and can be deallocated from the memory left for reconfiguration. For each basic data element, the "first created" field and the "last used" field store or record the locator address of the event at which the basic data element was first created or last used, respectively.

[0187] At each step in the distillation process (from the first distillation event to the last or Nth distillation event), the records of the "first created" and / or "last used" fields for one or more basic data elements involved in the event are updated. By the end of the distillation process, the records of the "first created" and "last used" entries are complete for all basic data elements. After distillation is complete, a new inventory analysis is performed by stepping through the locator addresses of each distillation event and determining the memory required to hold the basic data elements after each event. For any given event, the dynamic set of basic data elements that need to be held in memory for subsequent distillations is calculated as the difference between all "basic data elements created so far" and all "basic data elements retrieved so far." The "basic data elements created so far" is calculated as the sum of the sizes of all basic data elements whose "first created" value is less than or equal to this event's address locator value. The "basic data elements retrieved so far" is calculated as the sum of the sizes of all (previous) basic data elements whose "last used" value is less than this event's address locator value. The dynamic set is calculated for each distillation event. The maximum value of the dynamic set across all events is defined as the fine-grained working set, which is an upper bound on the size of memory that needs to be allocated during a reconstruction to ensure that all basic data elements needed during the reconstruction are dynamically available in memory without having to incur any random IO to fetch them.

[0188] In some embodiments, rather than performing the dynamic set calculation at each event (and therefore at each locator address), the calculation may be performed all at once for several consecutive events or ranges of locator addresses. This will reduce the amount of calculation and iteration required during the inventory step. This will produce a more conservative estimate of the fine-grained working set, but with a smaller impact on performance. In actual real-world situations, this estimate has been found to approach the actual fine-grained working set, and therefore, one can gain the benefit of using less memory (as dictated by the actual dynamic working set requirements of the dataset) while incurring a negligible performance loss due to the need to calculate a fine-grained working set estimate at the end of the distillation process.

[0189] Figure 12U illustrates the concept of PDE lifetime during the distillation of a sample dataset. This figure details the first seven PDEs (out of a total of N PDEs) and their lifetimes during the first 23 distillation events of the dataset. Each distillation event is indicated by a vertical dotted line intersecting the x-axis (indicating time). The lifetime of each PDE is indicated by a horizontal line, beginning with a large dot indicating when it was first created and ending with a large dot indicating when it was last used during the distillation of the entire dataset. While the remaining distillation events after the first 23 events are not shown in the figure, the ending dot indicating the "last use" of any PDE shown in this figure encompasses the distillation of the entire dataset. Thus, when the event for the last utilization of a given PDE is shown, this is the last use of that PDE during the distillation of the entire dataset. In Figure 12U, PDE1 is first created by the first distillation event 1288 (first vertical dotted line). The same PDE1 is used again to create overlaps or derivatives in the second, sixth, seventh, and eighth events. The sixth event 1290 indicates the use of PDE1, either as a derivation or a duplicate. The eighth event 1291 is the last use of PDE1. PDE1's lifetime extends from the first event 1288, when it is first created, to the eighth event 1291, when it is last used. The third distillation event creates PDE2, which is the only time this base data element is used (so there is no reuse and the reuse count is zero). Furthermore, the fourth event 1289 is the event when PDE3 is first created. Figure 12V shows the bookkeeping performed during distillation for the purpose of calculating the fine-grained working set as a stricter bound on the memory required to hold the base data elements during the reconstruction of this sample data set. When initially created, the base data elements are entered into a data structure along with their handle or identifier 1216, their originally created locator 1222 (which is the address of the input element in the input data being distilled), and the size 1218 of the base data element.The reuse count 1217 starts out initialized to zero, but is incremented each time a base data element is used to create a derivative or duplicate. Also, for all elements that are derived from (or are duplicates of) a given base data element, the locator address for that element is placed in the last reuse locator field 1223 of that base data element. After distillation is complete, an inventory step can be performed, and for each event, the locator address can be used to examine the data structure to determine the dynamic set for that event. After repeating this calculation across all events, the maximum value of the dynamic set can be set to the fine-grained working set.

[0190] In the example shown in Figures 12U and 12V, it can be seen that the PDE store, or base data store, for the dataset contains all seven base data elements PDE1 through PDE7, and therefore all 26,836 bytes of these base data elements. However, because PDE2 and PDE6 are never reused, the coarse-grained working set contains only five elements (PDE1, PDE3, PDE4, PDE5, and PDE7), and only 20,865 bytes of these. This is a reduced number of bytes. The fine-grained working set for the dataset contains only 10,177 bytes, which is the sum of PDE3 (6081 bytes) and PDE4 (4096 bytes), which is part of the dynamic set at event 11, the largest dynamic set across these 23 distillation events. These calculations demonstrate how additional improvements can be made to save on the memory required to hold the base data elements during reconstruction. Memory only needs to accommodate the fine-grained working set.

[0191] The fine-grained working set calculated at the end of the inventory step following the distillation of the dataset can be recorded and saved in the reduced data. Before invoking reconfiguration, the reduced data can be queried by the user to obtain the value of the fine-grained working set for this reduced data. The user can then invoke reconfiguration on the reduced data to provide the task with memory (specified via the reconfiguration memory budget control) equal to or greater than the fine-grained working set.

[0192] The apparatus described thus far enables more efficient use of memory during reconstruction by deallocating basic data elements from memory when their reuse counts are exhausted. By calculating the fine-grained working set during distillation, a tighter bound on the memory required during reconstruction can be obtained, and after querying the reduced data to discover this value, the reconstruction memory budget can be set accordingly. A further improvement can be made to allow the user to specify a "predicted reconstruction memory budget" a priori (at the start of distillation) to act as a bound that imposes a constraint on the distillation process, so that the fine-grained working set never exceeds the predicted reconstruction memory budget during reconstruction. Given such constraints, rather than simply discovering the fine-grained working set by performing an inventory step only at the end of the distillation of the data set, the distillation process is refined to perform repeated inventory steps at intervals during the distillation process to compute a conservative estimate of the fine-grained working set up to the time of the interval. At the moment this estimate of the fine-grained working set approaches the predicted reconstruction memory budget, rather than continuing distillation for the next interval, the data lot is closed and a new scope is opened for the subsequent distillation. In this scenario, the estimate of the fine-grained working set is conservative and may result in a value larger than the actual fine-grained working set. This is because the entire distillation process is not complete at the time each estimation is called. In this approach, given a predicted reconfiguration memory budget, distillation can first proceed until the sum of all basic data elements (the size of the PDE store) reaches the predicted reconfiguration memory budget. At this point, the inventory and estimation step can be called for the first time. The estimate can count the sum of all basic data elements with non-zero reuse counts into the estimated fine-grained working set; at this point, it is not known which elements were last used, and therefore the "most recently used" element cannot be subtracted. Elements with zero reuse counts can be removed from the estimate.Note the difference between the predicted reconstruction memory budget and the estimated fine-grained working set, and distillation can proceed to take in as many bytes of input as this difference, after which estimation is invoked again. The reason for this reduced interval is that each input element can potentially lead to an increase in the reuse count of existing elements with a reuse count of 0, bringing them into the fine-grained working set estimate. Again, estimation can be invoked, and the same process can be repeated. When the interval in the forward progression becomes smaller than a certain threshold, data lots (ranges where redundancy is exploited) can be closed, and further distillation of the dataset can proceed by opening new data lots.

[0193] In this way, distillation may be constrained to ensure that the actual memory required to hold basic data elements for reconstruction never exceeds the predetermined memory budget. Prior to reconstruction, it is no longer necessary to query the reduced data to source the actual fine-grained working set; rather, the reconstruction memory budget can simply be set to the value of the predicted reconstruction memory budget set at distillation time. This method may be employed in environments where there is little control over how reconstruction is performed and the user is certain whether or not to query the reduced data to determine the value of the fine-grained working set. This method of specifying the predicted reconstruction memory budget often results in a conservative estimate of the fine-grained working set in practice, leading to premature closure of data lots when the budget is about to be exceeded. An alternative approach is to not use the predicted reconstruction memory budget at distillation time, but simply specify the reconstruction memory budget at reconstruction time. When the amount of memory used by reconstruction reaches the budget, the implementation can switch to treating the memory like a cache, making space for incoming basic data elements, and the implementation can make decisions about which existing basic data elements to deallocate, thereby resulting in occasional random IO as they are returned from storage when needed. The gradual loss of performance with these IOs can be traded for the benefit of being able to perform distillation over a wider range.

[0194] Note that the discussion so far has described a method for accelerating reconstruction by using memory as the fastest storage tier available for reconstruction and keeping basic data elements in memory. Note that the above features can be used to keep basic data elements in the fastest storage tier available, even if that tier is not DRAM.

[0195] Note that FIG. 12J illustrates various components created by the data distillation device based on the organization of the distilled data and basic data sieves according to FIG. 1A, where the reconstruction programs are placed in the lossless reduced representation of the elements in the distilled file. Note that in some embodiments (according to FIG. 1B), the reconstruction programs can be placed in the basic data sieves and treated similarly to basic data elements. The lossless reduced representation of the elements in the distilled file includes a reference to the reconstruction program in the basic data sieve (rather than including the reconstruction program itself). In these embodiments, the reconstruction program is treated similarly to the basic data elements and generated in the PDE file 1211. In yet another embodiment, according to FIG. 1C, the reconstruction program is stored separately from the basic data elements in a structure referred to as the reconstruction program store. In such an embodiment, the lossless reduced representation of the elements in the distilled file includes a reference to the reconstruction program in the reconstruction program store. In such an embodiment, in addition to generating the PDE file for the basic data elements, the device also generates a file containing all the reconstruction programs, referred to as the RP file. This is illustrated in FIG. 12K, which illustrates the components of the reduced data for a usage model, where the mapper may no longer be maintained. FIG. 12K shows the reduced data components including the distillation file 1205, the PDE file 1211, and the RP file 1221.

[0196] 12L-12P illustrate how a distillation process according to some embodiments described herein can be deployed and executed on a distributed system to accommodate very large data sets at very fast ingestion rates.

[0197] The distributed computing paradigm involves the distributed processing of large data sets by programs running on multiple computers. Figure 12L shows a number of computers networked together in an organization called a distributed computing cluster. While Figure 12L shows point-to-point links between the computers, it will be understood that any communication topology, such as a hub-and-spoke topology or a mesh topology, may be used instead of the topology shown in Figure 12L. In a given cluster, one node is designated as the master node, which distributes tasks to the slave nodes and controls and coordinates their overall operation. The slave nodes perform tasks as directed by the master node.

[0198] The data distillation process may be distributed across multiple nodes of a distributed computing cluster to utilize the overall computational, memory, and storage capacity of the many computers in the cluster. In this setup, a master distillation module on the master node interacts with slave distillation modules running on slave nodes to achieve distributed data distillation. To facilitate this distribution, the device's basic data sieve may be partitioned into multiple independent subsets or subtrees that may be distributed across multiple slave modules running on slave nodes. Recall that in a data distillation device, basic data elements are organized in a tree form based on their names, which are derived from their content. The basic data sieve may be partitioned into multiple independent subsets or child sieves based on the first byte of the name of an element within the basic data sieve. There may be multiple ways to partition the namespace across multiple subtrees. For example, the value of the first byte of an element's name may be partitioned into multiple subranges, and each subrange may be assigned to a child sieve. Since there may be as many subsets or partitions created as there are slave modules in the cluster, each independent partition is deployed on a specific slave module. With the child sieves deployed, each slave module is designed to perform a data distillation process on the candidate elements it receives.

[0199] FIG. 12M shows a sample partitioning of a basic data sieve into four basic or child data sieves labeled PDS_1, PDS_2, PDS_3, and PDS_4 to be deployed on four slave modules running on four nodes. The partitioning is based on the first byte of the name of the basic data element. In the example shown, the first byte of the name of all elements in PDS_1 is in the range A to I, and sieve PDS_1 has a name A_I marked with a range of values ​​toward it. Similarly, the first byte of the name of all elements in PDS_2 is in the range J to O, and child sieve PDS_2 has a name J_O marked with a range of values ​​toward it. Similarly, the first byte of the name of all elements in PDS_3 is in the range P to S, and child sieve PDS_3 has a name P_S marked with a range of values ​​toward it. Finally, the first byte of the name of all elements in PDS_4 is in the range T to Z, and child sieve PDS_4 has a name T_Z marked with a range of values ​​toward it.

[0200] In this setup, a master module running on a master node receives an input file, performs lightweight parsing and factorization of the input file to split it into a sequence of candidate elements, and then directs each candidate element to a suitable slave module for further processing. The lightweight parsing may include parsing each candidate element against a schema or may include applying fingerprinting to the candidate element to determine the dimensions that make up the first bytes of the candidate element's name. The parsing in the master is limited to identifying only the number of bytes necessary to determine which slave module should receive the candidate element. Based on the value in the first bytes of the candidate element's name, the candidate is forwarded to a slave module in the slave node that holds a child sheave corresponding to this particular value.

[0201] As data accumulates within the sieves, the partitions may be continually reconsidered and rebalanced. The partitioning and rebalancing functions may be performed by a master module.

[0202] Upon receiving a candidate element, each slave module performs a data distillation process, beginning with creating a name for the candidate element by fully parsing and examining the candidate element. Using this name, the slave module performs a content-associative lookup of the child sieve and performs a distillation process to convert the candidate element into a lossless reduced representation of the element for that child sieve. The lossless reduced representation of the element in the distilled file is enriched with a field called SlaveNumber to identify the slave module and the corresponding child sieve to which the element was reduced. The lossless reduced representation of the element is sent back to the master module. If the candidate element is not found in the child sieve or cannot be derived from a base data element in the child sieve, a new base data element is identified to be assigned to the child sieve.

[0203] The master module continues to route all candidate elements from the input file to the appropriate slave modules, accumulating the element descriptions it receives (in a lossless reduced representation) until it has received all elements for the input file. At that point, a global commit communication is issued to all slave modules so that each child sieve can be updated with the results of their individual distillation processes. The distilled file for the input is stored in the master module.

[0204] In some embodiments, rather than waiting for the entire distilled file to be ready before any slave can update its child sieves with new base data elements or metadata, the update of the child sieves may be completed as candidate elements are processed in the slave modules.

[0205] In some embodiments, each child sieve includes basic data elements and reconstruction programs according to the description of FIGS. 1B and 1C. In such embodiments, the reconstruction programs are stored in the child sieve, and the lossless reduction representation includes references to both the basic data elements and the reconstruction programs (if necessary) in the child sieve. This further reduces the size of the elements and, therefore, the size of the distillation files that need to be stored in the master module. In some embodiments, the basic reconstruction program sieve in each child sieve includes the reconstruction programs used to create derivations from the basic data elements present in that child sieve. In such cases, the basic reconstruction programs are available locally at the slave node, enabling rapid derivation and reconstruction without any of the delays that would otherwise occur to fetch the basic reconstruction programs from remote nodes. In other embodiments, the basic reconstruction program sieve is distributed globally across all nodes to leverage the total capacity of the distributed system. The lossless reduction representation is refined with a second field that identifies the slave node or child sieve that contains the basic reconstruction program. In such embodiments, the solution incurs additional delays to fetch the basic reconstruction programs from remote nodes to generate the final reconstruction programs by derivation or to reconstruct the elements. The overall method distributes files across all nodes based on the content of each chunk or candidate element within each file, leveraging the combined storage capacity of all slave nodes.

[0206] Data retrieval is also coordinated by the master module. The master module receives the distilled file and looks up the lossless reduction specification for each element in the distilled file. The master module extracts the field "SlaveNumber" which indicates which slave module will reconstruct the element. The element is then sent to the appropriate slave module for reconstruction. The reconstructed element is then sent back to the master module. The master module assembles the reconstructed elements from all slaves and forwards the reconstructed file to the consumers requesting the file.

[0207] Figure 12N illustrates how a data distiller can be deployed and executed in a distributed system. An input file 1251 is sent to a master module, which parses and identifies the first byte of the name of each candidate element in the file. The master module directs the candidate elements to one of four slave modules. Slave module 1 in slave node 1, which holds PDS_1, or a child sieve with name A_I containing a base data element with a name leading byte having a value in the range A to I, receives candidate element 1252 with name BCD..., which is determined to be a duplicate of an element already present in the child sieve with name A_I. Slave module 1 returns a lossless reduced representation 1253 that includes an indicator that the element is base and present in slave 1 at address refPDE1. The master sends all candidate elements to the associated slave modules, which assemble, collect, and finally store the distilled file, as shown in Figure 12N.

[0208] Figure 12O shows a variation of the scheme shown in Figure 12N. In this variation, in the lossless reduction representation of each element in the distillation file, the field identifying the particular Child_Sieve to which the element was reduced contains the name of that Child_Sieve instead of the number of the module or node in which that Child_Sieve resides. Thus, the field SlaveNumber is replaced by the field Child_Sieve_Name. This includes the associated Child_Sieve. , has the advantage of referencing the Child_Sieve by its virtual address rather than the number of the module or physical node in which it resides. Thus, as can be seen in FIG. 12O, slave module 1 in slave node 1 holding PDS_1, or a child sieve with name A_I containing a base data element with a first byte of its name having a value in the range A to I, receives a candidate element 1252 with name BCD... that is determined to be a duplicate of an element already present in the child sieve with name A_I. Slave module 1 returns a lossless reduced representation 1254 that includes an indicator that the element is base and is present in the Child_Sieve with name A_I at address refPDE1.

[0209] Note that by using the configurations described in Figures 12L-12O, the overall throughput rate of the data distillation process can be increased. Throughput at the master is thus limited by lightweight parsing and dispatching of candidate elements from the master module. Distillation for multiple candidate elements is performed in parallel as long as their content directs them to separate slave modules.

[0210] To further increase overall throughput, the tasks of lightweight parsing and factoring the input stream to identify which Child_Sieve should receive the candidate element can be parallelized. This task can be partitioned by the master module into multiple concurrent tasks performed in parallel by slave modules running on multiple slave nodes. This can be accomplished by looking ahead in the data stream and slicing it into multiple overlapping segments. These segments are sent by the master to each of the slave modules, which perform lightweight parsing and factorization in parallel and send the factorization results back to the master. After solving the factorization across each segment boundary, the master routes the candidate element to the appropriate slave module.

[0211] Figures 12L through 12O illustrated a configuration in which the data distillation apparatus operated in a distributed manner, using one master distillation module running on the master node and multiple slave distillation modules running on the slave nodes. The master module was responsible for partitioning the basic data elements across the various child sieves. In the configuration shown, all input files to be ingested were ingested by the master module, and the losslessly reduced distilled files were maintained in the master module, while all basic data elements (and any basic reconstruction programs) resided in the child sieves in the various slaves. Data retrieval requests for files were also handled by the master, and the reconstruction of the corresponding distilled files was coordinated by the master. Figure 12P illustrates a variation in which input files can be ingested by any of the slave distillation modules (and the corresponding distilled files maintained in those modules), and data retrieval requests can be processed by any of the slave distillation modules. Because the master module continues to partition the basic data elements across the child sieves in the same manner, the distribution of the basic data elements across the child sieves is the same as the distribution in the configuration shown in Figures 12L through 12O. However, in the new configuration shown in Figure 12P, each slave module is aware of the partitioning because it can both import and export data. Furthermore, all modules are aware of the existence and location of the distilled files that are created and stored in each of the modules when data is imported by those modules. This allows any slave module to fulfill a data retrieval request for any of the files stored throughout the system.

[0212] As shown in FIG. 12P, each of the slave modules can ingest and retrieve data from the distributed storage system. For example, slave distillation module 1 1270 ingests input file I 1271, performs lightweight parsing to factorize input file I, and routes the candidate elements to modules containing child sieves corresponding to the names of each candidate element from input file I. For example, candidate element 1275 from input file I is sent to slave distillation module 2 1279. Similarly, slave distillation module 2 1279 ingests input file II, performs lightweight parsing to factorize input file II, and routes the candidate elements to modules containing child sieves corresponding to the names of each candidate element from input file II. For example, candidate element 1277 from input file II is sent to slave distillation module 1 1270. Each of the slave distillation modules processes the candidate elements it receives, completes the distillation process for those child sieves, and returns a lossless reduced representation of the candidate element to the initiating module that ingested the data. For example, in response to receiving candidate element 1275 from input file I from slave distillation module 1 1270, slave distillation module 2 1279 returns lossless reduction element 1276 to slave distillation module 1 1270. Similarly, in response to receiving candidate element 1277 from input file II from slave distillation module 2 1279, slave distillation module 1 1270 returns lossless reduction element 1278 to slave distillation module 2 1279.

[0213] In this configuration, data retrieval can be satisfied at any slave module. The module receiving the retrieval request must first determine where the distilled file for the requested file resides and fetch the distilled file from the corresponding slave module. The initiating slave module must then coordinate a distributed reassembly of the various elements in the distilled file to result in the original file and serve it to the requesting application.

[0214] In this way, the data distillation process can be distributed across multiple nodes of a distributed system to more effectively utilize the overall computational, memory, and storage capacity of the many computers in the cluster. All nodes in the system can be utilized to ingest and retrieve data. This should enable ultra-fast data ingestion and retrieval while making full use of the combined total storage capacity of the nodes in the system. It also allows an application running on any node in the system to query at the local node any data stored anywhere in the system and obtain the answer to that query efficiently and seamlessly.

[0215] In the configuration described in Figures 12M through 12P, the partitioning of data across child sieves residing in various nodes of the system was based on the names of elements in a globally visible namespace, from which elements are extracted by factoring input files. In an alternative configuration, data lots, or entire groups of files sharing certain metadata, can be assigned and stored on specific nodes. Thus, the primary partitioning of the entire data is based on data lots and is performed and managed by the master. All slave modules remain aware of the assignment of data lots to modules. A data lot resides entirely on a given slave node. The child sieve on the distillation slave module running on that slave node contains all basic data elements belonging to this data lot. In other words, the entire tree of all basic data elements for a given data lot resides entirely on one child sieve within one slave distillation module. All distillation files for a given data lot also reside on the same slave distillation module. With this configuration, input files can still be ingested by any of the slave distillation modules, and data retrieval requests can still be processed by any of the slave distillation modules. However, the entire data distillation process for a given data lot is performed entirely on the module containing that data lot. Data ingestion and retrieval requests are routed from the initiating module to the specific slave module designated to hold the particular data lot. This solution has the advantage of reducing communication overhead in a distributed environment when factoring and distilling data lots. Redundancies are no longer utilized across the global data footprint, but are utilized very efficiently locally within the data lot. This solution still uses the combined storage capacity of the distributed system and provides a seamless ability to query, ingest, and retrieve any data from any node in the system.

[0216] Thus, using the many techniques described above, data distillation is performed at ultra-high speeds on very large datasets in a resource-efficient manner in distributed systems.

[0217] The Data Distillation™ method and apparatus may be further improved to facilitate efficient movement and migration of data. In some embodiments, the lossless reduced data set may be provided in the form of multiple containers or parcels to facilitate data movement. In some embodiments, one or more reduced data lots may fit into a single container or parcel, or a single reduced data lot may be converted into multiple parcels. In some embodiments, a single reduced data lot is provided as a single self-describing parcel. Figure 12Q shows the same structure of such a parcel. The parcel 1280 of Figure 12Q can be viewed as a single file or as a contiguous set of bytes containing the following components concatenated together sequentially: (1) Header 1281, which is a parcel header containing, first, a parcel length 1282 specifying the length of the parcel, and, second, an offset identifier identifying the offset where the distillation file, PDE file, and various manifests are located within the parcel; (2) Distillation File 1283, which is a distillation file for a concatenated lot of data, where the length of each distillation file is specified first, followed by all the bytes that comprise the distillation file; (3) PDE File 1284, which is a PDE file that begins with a length identifier for the PDE file, followed by the body of the PDE file containing all the basic data elements; and (4) Source Manifest 1285, which is a source manifest that describes the structure of the input dataset and identifies the unique directory structure, pathname, and filename of each file in the parcel. The source manifest also includes a listing of each node in the (reduced and parceled) input data lot along with metadata associated with each node; (5) a destination manifest and mapper, Destination Manifest and Mapper 1286, which provides the intended mapping of each input node and file into the target destination directory and file structure or target bucket / container and object / blob structure in the cloud.This manifest facilitates the movement, reconfiguration, and relocation of various components within a parcel to their final destinations after the data movement, and this destination mapper section can be independently modified to retarget the destinations to which the data within a parcel will be transferred and reconfigured.

[0218] In this way, a lossless reduced representation of a data lot is provided as a parcel in a format that is self-describing and suitable for moving and rearranging data.

[0219] The effectiveness of the embodiments described herein was determined by performing data reduction on various real-world datasets. The real-world datasets considered included the Enron corpus of corporate emails, various U.S. government records and documents, U.S. Department of Transportation records entered into a MongoDB NOSQL database, and publicly available corporate PowerPoint presentations. Using the embodiments described herein, factoring the input data into variable-sized elements (bounded by fingerprinting) with an average of 4KB achieved an average data reduction of 3.23x across these datasets. A 3.23x reduction means that the size of the reduced data is equal to the size of the original data divided by 3.23x, resulting in a reduced footprint of 31% compression. Traditional data deduplication techniques were found to provide a 1.487x data reduction on these datasets using comparable parameters. Using the embodiments described herein, factoring the input data into fixed-size elements of 4KB achieved an average data reduction of 1.86x across these datasets. Legacy data deduplication techniques were found to provide a 1.08x data reduction for these datasets using comparable parameters. Thus, the Data Distillation™ solution was found to provide significantly better data reduction than legacy data deduplication solutions.

[0220] Test runs also confirmed that a small subset of the bytes of the basic data elements allows for ordering of the majority of elements within a sieve, thereby enabling a solution with minimal incremental storage for its operations.

[0221] These results confirm that the Data Distillation™ apparatus efficiently enables exploiting redundancy between data elements globally across an entire dataset, at a granularity finer than the element itself. The lossless data reduction provided by this method is achieved with streamlined data access and IO, using data structures that themselves require minimal incremental storage, and using a small fraction of the total computational power available on modern multi-core microprocessors. The embodiments described in the previous section feature systems and techniques that perform lossless data reduction on large and very large datasets while providing high speed data ingestion and retrieval, and that do not suffer from the drawbacks and limitations of conventional techniques.

[0222] Performing content-associative search and retrieval on losslessly reduced data by deriving the data from basic data elements present in the basic data sieve The data distillation apparatus described above and illustrated in Figures 1A through 12P can be enhanced with certain features to efficiently perform multidimensional search and content-associative retrieval of information from data stored in a lossless, reduced format. Such multidimensional search and data retrieval are important building blocks of analytics or data warehousing applications. These enhancements are now described.

[0223] FIG. 13 shows a leaf node data structure similar to the structure shown in FIG. 3H. However, in FIG. 13, the entry in the leaf node data structure for each base data element is modified to include references (also referred to as backward references or backward links) to all elements in the distilled data that contain a reference to that particular base data element. Recall that the data distillation scheme factors data from the input file into a sequence of elements, which are placed in a distilled file in a reduced format using a specification such as that described in FIG. 1H. There are two types of elements in the distilled file: base data elements and derived elements. The specification of each of these elements in the distilled file includes a reference to the base data element that exists in the base data sieve. For each of these references (from an element in the distilled file to the base data element in the base data sieve), there will be a corresponding backward link or backward reference installed in the leaf node data structure (from the entry for the base data element in the leaf node data structure to the element in the distilled file). The backward reference determines the offset in the distilled file that marks the start of the lossless reduced representation of the element. In some embodiments, the backward reference includes the name of the distilled file and an offset within that file that identifies the location of the start of the element. As shown in Figure 13, along with the backward reference for each element in the distilled file, the leaf node data structure further maintains an indicator that identifies whether the referenced element in the distilled file is a base data element (prime) or whether it is a derived element (deriv). During the distillation process, when an element is placed into the distilled file, a backward link is installed in the leaf node data structure.

[0224] The backward reference or backward link is designed as a universal handle that can reach all elements in all distilled files that share a basic data sieve.

[0225] Because data element sizes are expected to be chosen so that each reference is a small fraction of the data element's size, adding backward references is not expected to significantly affect the data reduction achieved. For example, consider a system in which each derived element is constrained to be derived from no more than one base data element (so that multiple derived elements are not allowed). The total number of backward references across all leaf node data structures is equal to the total number of elements across all distilled files. Assume a sample input data set 32 ​​GB in size is reduced to 8 GB of lossless reduced data, using an average element size of 1 KB, resulting in a 4x reduction ratio. There are 32M elements in the input data. If each backward reference is 8 KB in size, the total space occupied by the backward references is 256 MB, or 0.25 GB. This is a small increase for the 8 GB footprint of reduced data. The new footprint is 8.25 GB, and the effective reduction achieved is 3.88x, representing a 3% reduction loss. This is a small price to pay for the benefits of powerful content-associative data retrieval over the reduced data.

[0226] As discussed earlier in this document, the distiller can use a variety of methods to determine the location of various components of the skeletal data structure within the content of a candidate element. The various components of an element's skeletal data structure can be thought of as dimensions, and the concatenation of these dimensions, followed by the remaining content of each element, is used to create the name of each element. The names are used to order and organize the basic data elements into a tree.

[0227] In a usage model where the structure of the input data is known, a schema defines the various fields or dimensions. Such a schema is supplied by the analytics application using this content-associative data retrieval device and provided to the device via an interface to the application. Based on the declarations in the schema, the distillation device's parser can parse the content of the candidate element to detect and locate the various dimensions and create a name for the candidate element. As described above, elements with the same content in fields corresponding to dimensions are grouped together along the same leg of the tree. For each basic data element installed in the sieve, information about the dimensions can be stored as metadata in the entry for that basic data element in the leaf node data structure. This information can include the location, size, and value of the content in each declared dimension and can be stored in a field called "Other Metadata for Basic Data Element" in FIG. 13.

[0228] FIG. 14A illustrates a sample schema providing a description of the structure of an input dataset and a description of the correspondence between the structure and dimensions of the input dataset, according to some embodiments described herein. Structure description 1402 is an excerpt or portion of a more complete schema describing the complete structure of the input data. Structure description 1402 includes a listing of keys (e.g., “PROD_ID,” “MFG,” “MONTH,” “CUS_LOC,” “CATEGORY,” and “PRICE”), followed by the value types corresponding to the keys. A colon symbol “:” is used as a delimiter separating keys from value types, and a semicolon symbol “;” is used as a delimiter separating separate pairs of keys from their corresponding value types. Note that the complete schema (of which structure 1402 is a part) may specify additional fields identifying the start and end of each input, and possibly other fields outside the dimensions. Dimension mapping description 1404 describes how the dimensions used to organize the basic data elements map to key values ​​in the structured input dataset. For example, the first line in dimension mapping description 1404 specifies that dimension 1 is created using the first four bytes of the value corresponding to the key "MFG" in the input data set (because the first line ends with the text "prefix=4"). The remaining lines in dimension mapping description 1404 describe how to create the other three dimensions based on the structured input data. In mapping keys to this dimension, the order of the keys as they appear in the input does not necessarily match the order of the dimensions. Using the provided schema description, the parser can recognize these dimensions in the input data and create names for candidate elements. For example, in Figure 14A, using dimension mapping description 1404, names for candidate elements are created as follows:(1) The first four bytes of the name are the first four bytes from the value corresponding to the key "MFG", which is declared as dimension 1; (2) the next four bytes of the name are the first four bytes from the value corresponding to the key "CATEGORY", which is declared as dimension 2; (3) the next three bytes of the name are the first three bytes from the value corresponding to the key "CUS_LOC", which is declared as dimension 3; (4) the next three bytes of the name are the first three bytes from the value corresponding to the key "MONTH", which is declared as dimension 4; (5) the next set of bytes of the name consists of the concatenation of the remaining bytes from the dimensions; and (6) finally, after all the bytes of the dimensions have been used up, the remaining bytes of the name are created from the concatenation of the remaining bytes of the candidate elements.

[0229] The schema provided by the application driving the device may specify the number of first and second dimensions. Information about all of these first and second dimensions may be maintained in metadata within the leaf node data structure. The first dimension forms the main axis along which elements are sorted and organized within the sieve. If the first dimension is exhausted and subtrees with large membership still remain, the second dimension may also be used deeper in the tree to subdivide elements into smaller groups. Information about the second dimension is maintained as metadata and may also be used as a secondary criterion to distinguish elements within the leaf node. In some embodiments that provide content-associative multidimensional search and retrieval, a requirement may be imposed that all incoming data must contain keys and valid values ​​for each dimension declared by the schema. This provides the system with a way to ensure that only valid data is placed into the desired subtree within the sieve. Candidate elements that do not contain all fields specified as dimensions or that contain invalid values ​​in the values ​​corresponding to the fields for the dimensions are sent down a different subtree, as previously shown in Figure 3E.

[0230] The data distiller is constrained in one additional way to comprehensively support content-associative search and retrieval of data based on the content within a dimension. When a derived element is created from a base data element, the derivation unit is constrained to ensure that both the base data element and the derivation have the exact same content in the value field for each of the corresponding dimensions. Thus, while the derivation is being created, the reconstruction program is not allowed to perturb or change the content in the value field corresponding to any of the base data element's dimensions in order to construct the derived element. Given a candidate element, upon sieve lookup, if the candidate element has different content in any dimension compared to the corresponding dimension of the target base data element, a new base data element must be installed instead of accepting the derivation. For example, if this subset of the first dimension has sorted the elements into distinct groups in the tree such that a candidate element arrives at a leaf node and finds a base data element that has the same content in the subset of the first dimension but different content in the remaining first or second dimensions, then instead of creating a derivative, a new base data element needs to be installed. This feature ensures that all data can be retrieved using a dimension by simply querying the base data sieve.

[0231] The derivation unit may use various implementation techniques to enforce the constraint that the candidate element and the base data element must have exactly the same content in the value field for each corresponding dimension. The derivation unit may extract information from the base data element's skeletal data structure, including the location, length, and content of the fields corresponding to the dimensions. Similarly, this information may be received from the parser / factorizer or calculated for the candidate element. Corresponding fields for the dimensions from the candidate element and the base data element may then be compared for equality. If equality is determined, the derivation unit may proceed with the remainder of the derivation. If not equal, the candidate element is installed into the sieve as a new base data element.

[0232] The above limitations are not expected to significantly hinder the degree of data reduction for most usage models. For example, if the input data consists of a set of elements, each of which is a data warehouse transaction 1000 bytes in size, and a set of six primary dimensions and 14 secondary dimensions is specified by a schema, each of which has, for example, 8 bytes of data per dimension, then the total bytes occupied by content in the dimensions is 160 bytes. No fluctuations are allowed on these 160 bytes when creating derivations. This leaves ample opportunity to exploit redundancy, as the remaining 840 bytes of candidate element data remain available for fluctuation to create derivations, while at the same time allowing content-associative searching and retrieval of data from the data warehouse using the dimensions.

[0233] To perform a search query for data containing a particular value for a field in a dimension, the device can traverse the tree to reach a node in the tree that matches the specified dimension, and all leaf node data structures below that node can be returned as lookup results. A reference to a base data element present in the leaf node can be used to fetch the desired base data element, if necessary. A reverse link allows retrieval of input elements (in a lossless reduced format) from the distilled file, if desired. The elements can then be reconstructed to yield the original input data. Thus, the improved device can perform all searches against data in the base data sieve (which is a small subset of the total data), and still reach and retrieve all derived elements as needed.

[0234] The improved apparatus can be used to perform search and lookup queries for powerful search and retrieval of relevant subsets of data based on content within dimensions specified by the query. A content-associative data retrieval query has the form "fetch(dimension 1, dimension 1 value; dimension 2, dimension 2 value; ...)." The query specifies the dimensions to be involved in the search and the values ​​to be used for each of the specified dimensions for the content-associative search and lookup. The query may specify all dimensions or only a subset of the dimensions. The query may specify complex conditions based on multiple dimensions as search and retrieval criteria. All data within the sieve that have the specified values ​​of the specified dimensions will be retrieved.

[0235] A variety of fetch queries are supported and can be utilized by analytics applications using this content-associative data retrieval device. Such queries are supplied from an application to the device via an interface. The interface provides queries from the application to the device and returns query results from the device to the application. First, a query, FetchRefs, can be used to fetch a reference or handle to the leaf node data structure of FIG. 13 (along with the entry's child ID or index) for each basic data element that matches the query. A second form of query, FetchMetaData, can be used to fetch metadata (including the skeletal data structure, information about dimensions, and a reference to the basic data element) from an entry in the leaf node data structure of FIG. 13 for each basic data element that matches the query. A third form of query, FetchPDEs, fetches all basic data elements that match the search criteria. Another form of query, FetchDistilledElements, fetches all elements in the distilled file that match the search criteria. Yet another form of query, FetchElements, fetches elements in the input data that match the search criteria. Note that for the FetchElements query, the device It first fetches the distillation elements, then reconstructs the related distillation elements into elements from the input data and returns them as the query results.

[0236] In addition to such multidimensional content associative fetch primitives, the interface may further provide applications with the ability to directly access base data elements (using base data element references) and elements in the distilled file (using element reverse references). Additionally, the interface may provide applications with the ability (given a distilled element reference) to reconstruct a distilled element in the distilled file to provide that element as it existed in the input data.

[0237] Appropriate combinations of these queries can be used by analytics applications to perform searches, find relevant unions and intersections, and glean important insights.

[0238] Figure 14B, described below, shows an example of an input data set having the structure described in structural description 1402. In this example, the input data contained in file 1405 contains e-commerce transactions. Using the schema and dimension declarations of Figure 14A, the input data is converted by a parser in the data distiller into a set of candidate elements 1406. Note that the first byte of each candidate element's name is composed of content from the dimension. For example, the first byte of name 1407 for candidate element 1 is PRINRACQNYCFEB. These names are used to organize the candidate elements into a tree. After data reduction is complete, the distilled data is placed into distilled file 1408.

[0239] FIG. 14C, described below, shows how the dimension mapping description 1404 can be used to parse the input data set shown in FIG. 14A according to the structure description 1402, determine the dimensions according to the dimension mapping description 1404, and organize the base data elements into a tree based on the determined dimensions. In FIG. 14C, the base data elements are organized in a master tree using a total of 14 characters taken from four dimensions. Within the master tree, portions of the leaf node data structures for various base data elements are shown. Note that for clarity, the complete leaf node data structures of FIG. 13 are not shown. However, FIG. 14C shows the path information or name of each entry in the leaf node data structure, the child ID, an indicator of whether the element in the distilled file is "prime" (denoted by P) or "deriv" ​​(denoted by D), all backward references or links from the base data elements to elements in the distilled file, and further references to the base data elements. FIG. 14C shows seven elements in the distilled file that map to five base data elements in the master tree. In Figure 14C, backward link A for the base data element with name PRINRACQNYCFEB refers back to element 1 in the distillation file, while the base data element with name NIKESHOELAHJUN has backward links B, C, and E to elements 2, 3, and 58, respectively. Note that elements 3 and 58 are derivatives of element 2.

[0240] Figure 14D shows an auxiliary index or auxiliary tree that is created from a dimension to improve search efficiency. In this example, the auxiliary mapping tree that is created is based on dimension 2 (which is CATEGORY). By directly traversing this auxiliary tree, all elements of a given CATEGORY in the input data can be found without a more expensive traversal of the master tree that might otherwise be required. For example, traversing down the leg indicated by "SHOE" directly yields the two base data elements for shoes, which are ADIDSHOESJCSEP and NIKESHOELAHJUN.

[0241] Alternatively, such auxiliary trees may be based on a second dimension and used to help quickly focus a search using that dimension.

[0242] The following are examples of queries executed on the device shown in Figure 14D. The query FetchPDEs(dimension 1, NIKE;) returns two basic data elements named NIKESHOELAHJUN and NIKEJERSLAHOCT. The query FetchDistilledElements(dimension 1, NIKE;) returns element 2, element 3, element 58, and element 59, which are the distilled elements in the lossless reduction format. The query FetchElements(dimension 1, NIKE; dimension 2, SHOE) returns transaction 2, transaction 3, and transaction 58 from the input data file 1405. The query FetchMetaData(dimension 2, SHOES) returns the metadata stored in the leaf node data structure entries for each of the two basic data elements named ADIDSHOESJCSEP and NIKESHOELAHJUN.

[0243] The apparatus described thus far can be used to support searches based on content specified in fields called dimensions. It can also be used to support searches based on keyword listings that are not included in the dimension listings. Such keywords can be provided to the apparatus by an application, such as a search engine, that is driving the apparatus. Keywords can be specified to the apparatus through a schema declaration, or can be passed via a keyword list that includes all keywords, each separated by a declaration separator (such as a space, comma, or newline). Alternatively, all keywords can be specified comprehensively using both the schema and the keyword list. Numerous keywords may be specified, and the apparatus imposes no limit on the number of keywords. These search keywords are called keywords. The apparatus can maintain an inverted index for searches using these keywords. The inverted index includes, for each keyword, a listing of backward references to elements in the distilled file that contain this keyword.

[0244] Based on the keyword declarations or keyword lists in the schema, the parser of the distillation device can parse the contents of the candidate element to detect and locate various keywords (if and where found) in the incoming candidate element. The candidate element is then converted by the data distillation device into a base data element or derived element and placed as an element in the distilled file. The inverted index for keywords found in this element can be updated with the backward reference of this element in the distilled file. For each keyword found in an element, the inverted index is updated to include the backward reference of this element in the distilled file. Recall that elements in the distilled file are lossless reduced representations.

[0245] When a search query of the data is performed using a keyword, the inverted index is consulted to find and extract backward references of elements in the distilled file that contain that keyword. Such backward references of elements can be used to retrieve a lossless reduced representation of the element, allowing the element to be reconstructed. The reconstructed element can then be provided as a result of the search query.

[0246] The inverted index may be modified to include information identifying the location of the offset of the keyword within the reconstructed element. Note that because the parser may determine the offset or location of each keyword detected in the candidate element, this information may also be recorded in the inverted index when the backward reference of the element in the distilled file is entered into the inverted index. When a search query is executed, after the backward reference of the element in the distilled file containing the associated keyword is retrieved by consulting the inverted index, and after the element is reconstructed, the recorded offset or location of the keyword in the reconstructed element (which is the same as the original input candidate element) can be used to identify the exact location within the input data or input file where the keyword is present.

[0247] Figure 15 shows an inverted index that facilitates keyword-based searching. For each keyword, the inverted index contains a pair of values, the first being a reverse reference to the lossless reduction element in the distilled file that contains the keyword, and the second value being the offset of the keyword within the reconstructed element.

[0248] Dimensions and keywords have different implications for the basic data sieves within the data distiller. Dimensions are used as the main axes along which the basic data elements are organized within the sieves. Dimensions form the skeletal data structure of each element within the data. Dimensions are declared based on knowledge of the structure of the incoming data. The derivation unit is constrained such that any derived element created must have the exact same content as the basic data element within the values ​​of the fields for each corresponding dimension.

[0249] These properties need not hold for keywords. In some embodiments, there is no a priori requirement that keywords exist in the data at all, the base data elements may not be organized based on keywords, and the derivation unit is not constrained in terms of derivations involving content that contains keywords. The derivation unit is free to create derivations from base data elements by modifying the keyword values ​​as needed. The locations of keywords are simply recorded where found during a scan of the input data, and the inverted index is updated. When a content associative search based on keywords is performed, the inverted index is queried to obtain all locations of the keyword.

[0250] In other embodiments, the keyword need not be present in the data (the absence of a keyword in the data does not invalidate the data), but the base data sieve must contain all elements that contain the keyword, and the derivation section is constrained with respect to derivations involving content that contains the keyword; i.e., derivations are not allowed except to reduce duplication. The objective of these embodiments is that all distinct elements that contain any keyword must be present in the base data sieve. This is an example where the rules governing the selection of base data are conditioned by the keyword. In these embodiments, a modified inverted index may be created that contains, for each keyword, a backward reference to each base data element that contains that keyword. These embodiments provide powerful keyword-based search capabilities, where searching only the base data sieve is as effective as searching the entire data set.

[0251] Other embodiments may exist in which the derivation unit is constrained such that the reconstruction program is not allowed to perturb or change the content of any keywords found in a base data element in order to formulate the candidate element as a derived element of that base data element. Keywords must propagate unchanged from the base data element to the derivation. If the derivation unit needs to change the bytes of any keywords found in the base data element in order to successfully formulate the candidate as a derived element of this base data element, the derivation may not be accepted and the candidate must be installed in the sieve as a new base data element.

[0252] The derivation can be constrained in various ways with respect to derivations involving keywords, so that the rules governing the selection of the underlying data are conditioned by the keywords.

[0253] The apparatus for searching data using keywords may accept updates to the keyword listing. Keywords may be added without any changes to the data stored in lossless reduced form. When new keywords are added, new incoming data may be parsed against the updated keyword list, and the inverted index updated with the incoming data is then stored in lossless reduced form. When existing data (already stored in lossless reduced form) needs to be indexed against the new keywords, the apparatus may incrementally read through the distilled files (one or more distilled files at a time, or one losslessly reduced data lot at a time), reconstruct the original files (but without disturbing the losslessly reduced stored data), parse the reconstructed files, and update the inverted index. All the while, the entire data repository may continue to be stored in lossless reduced form.

[0254] Figure 16A shows a schema declaration that is a variation of the schema shown in Figure 14A. The schema in Figure 16A includes a declaration of a second dimension 1609 and a listing of keywords 1610. Figure 16B shows an example of an input data set 1611 having a structure described in structure description 1602 that is parsed and converted into a set of candidate elements with names based on the declared first dimension. The candidate elements are converted to elements in a distillation file 1613. The declaration of the second dimension "PROD_ID" imposes a constraint on the derivation such that candidate element 58 cannot be derived from the base data element "NIKESHOELAHJUN with PROD_ID=348," and therefore one additional base data element, "NIKESHOELAHJUN with PROD_ID=349," is created in the base data set. The input data set is the same as that shown in Figure 14B, but the result of distillation results in seven distilled elements but six base data elements. FIG. 16C shows the distillation file, master tree, and base data elements created as a result of the distillation process.

[0255] Figure 16D shows the auxiliary tree created for the second dimension, "PROD_ID." Traversing this tree with a particular PROD_ID value yields the base data element with that particular PROD_ID. For example, the query FetchPDEs(dimension 5, 251) for the base data element with PROD_ID=251, or alternatively the query FetchPDEs(PROD_ID, 251), yields the base data element WILSBALLLAHNOV.

[0256] Figure 16E shows an inverted index (labeled Inverted Index for Keywords 1631) created for three keywords declared in structure 1610 of Figure 16A. These keywords are FEDERER, LAVER, and SHARAPOVA. The inverted index is updated after parsing and consuming input dataset 1611. The query FetchDistilledElements(keyword Federer) utilizes the inverted index (rather than the master tree or auxiliary tree) to return element 2, element 3, and element 58.

[0257] FIG. 17 shows a block diagram of an overall apparatus improved for content-associative data retrieval. A content-associative data retrieval engine 1701 provides a data distillation apparatus with a schema 1704, or structure definition including the dimensions of the data. The engine 1701 also provides the apparatus with a keyword list 1705. The engine 1701 issues queries 1702 for searching and retrieving data from the distillation apparatus and receives the query results as results 1703. The derivation unit 110 is improved to recognize the declaration of dimensions when creating derivations and to prohibit content changes at dimensional locations. Note that backward references from entries in the leaf node data structure to elements in the distilled file are stored in the leaf node data structure in the base data sieve 106. Similarly, auxiliary indexes are also stored in the base data sieve 106. Also shown is an inverted index 1707, which is updated with backward references 1709 by the derivation unit 110 while elements are written to the distilled data. This content-associative data retrieval engine interacts with other applications (such as analytics, data warehousing, and data analysis applications) and provides them with the results of queries they perform.

[0258] In summary, the improved data distiller enables powerful multidimensional content-associative search and retrieval of data stored in a lossless reduced form.

[0259] The Data Distillation™ device can be used to losslessly reduce audio and video data. The data reduction achieved by this method is achieved by deriving components of the audio and video data from basic data elements present in a content-associative sieve. The application of this method for such purposes is now described.

[0260] Figures 18A-18B show block diagrams of an encoder and decoder for compressing and decoding audio data according to the MPEG1 Layer 3 standard (also known as MP3). MP3 is an audio coding format for digital audio that compresses input audio using a combination of lossy and lossless data reduction techniques. MP3 compresses compact disc (CD) audio from 1.4 Mbps to 128 Kbps. MP3 exploits the limitations of the human ear to suppress audio components that are inaudible to most people. To achieve this, a set of techniques collectively known as perceptual coding techniques are used, which lossily but imperceptibly reduce the size of snippets of audio data. Perceptual coding techniques are lossy; information lost during these steps cannot be recovered. These perceptual coding techniques are complemented by Huffman coding, a lossless data reduction technique described earlier in this document.

[0261] In MP3, an input audio stream is compressed into a sequence of small data frames, each containing a frame header and compressed audio data. The original audio stream is periodically sampled to produce a sequence of audio snippets, which are then compressed using perceptual coding and Huffman coding to produce a sequence of MP3 data frames. Both perceptual coding and Huffman coding techniques are applied locally within each snippet of audio data. Huffman coding exploits redundancy locally within a snippet of audio, but not globally across the audio stream. Thus, MP3 technology does not exploit redundancy globally, either across a single audio stream or between multiple audio streams. This provides an opportunity for further data reduction beyond what MP3 can achieve.

[0262] Each MP3 data frame represents a 26-ms audio snippet. Each frame stores 1152 samples and is subdivided into two granules containing 576 samples each. As can be seen in the encoder block diagram of Figure 18A, when encoding a digital audio signal, time-domain samples are taken and converted into 576 frequency-domain samples through a process of filtering and application of a modified discrete cosine transform (MDCT). Perceptual coding techniques are applied to reduce the amount of information contained in the samples. The output of perceptual coding is nonuniformly quantized granules 1810 containing reduced information per frequency line. Huffman coding is then used to further reduce the size of the granules. The 576 frequency lines of each granule may use multiple Huffman tables for their encoding. The output of Huffman coding is the main data component of the frame, including scale factors, Huffman coding bits, and auxiliary data. Side information (used to characterize and locate various fields) is placed in the MP3 header. The output of the coding is an MP3-encoded audio signal. At a bit rate of 128 Kbps, an MP3 frame is 417 or 418 bytes in size.

[0263] FIG. 18C illustrates how the data distiller first shown in FIG. 1A may be improved to perform data reduction on MP3 data. The method illustrated in FIG. 18C factors MP3 data into candidate elements and exploits redundancy between the elements at a finer granularity than the elements themselves. For MP3 data, granules are selected as elements. In one embodiment, non-uniformly quantized granules 1810 (as shown in FIG. 18A) may be treated as elements. In another embodiment, an element may consist of a concatenation of a quantization frequency line 1854 and a scale factor 1855.

[0264] In FIG. 18C, a stream of MP3-encoded data 1862 is received by a data distiller 1863 and reduced to a stream of distilled MP3 data 1868, which is stored in a lossless reduced form. The input stream of MP3-encoded data 1862 consists of an MP3 header and a sequence of MP3 data pairs. The MP3 data includes a CRC, side information, primary data, and auxiliary data. The output distilled MP3 data produced by the device consists of a similar sequence of pairs (each pair is a distilled MP3 header followed by an element specification in lossless reduced form). The distilled MP3 header contains all components of the original frame except the primary data, namely, the MP3 header, CRC, side information, and auxiliary data. The element field in this distilled MP3 data contains the granules specified in the lossless reduced form. A parser / factorizer 1864 performs a first decoding of the incoming MP3-encoded stream (including performing Huffman decoding) to extract quantized frequency lines 1851 and scale factors 1852 (shown in FIG. 18B) and generate audio granules 1865 as candidate elements. The first decoding steps performed by the parser / factorizer are identical to the synchronization and error checking 1851, Huffman decoding 1852, and scale factor decoding 1853 steps of FIG. 18B. These steps are performed in any standard MP3 decoder and are well known in the art. A basic data sieve 1866 contains granules as basic data elements organized for content-associative access. When a granule is installed in the basic data sieve, the contents of the granule are used to determine where in the sieve the granule should be installed and to update the skeletal data structure and metadata in the appropriate leaf node of the sieve. The granules are then Huffman coded and compressed so that they can be stored in the sieve with a footprint similar to the one they occupied when they were in the MP3 data. Each time a granule in the sieve is needed by the derivation unit as a basic data element, the granule is decompressed and then provided to the derivation unit.Using this data distiller, input audio granules are derived from the basic data elements (also audio granules) present in the sieve by the derivation unit 1870, and a lossless reduced or distilled representation of the granule is created and placed in the distilled MP3 data 1868. This distilled representation of the granule is placed in an element field that replaces the Huffman-encoded information originally present in the primary data field of the MP3 frame. The distilled representation of each element or granule is encoded using the format shown in FIG. 1H. Each element in the distilled data is either a basic data element (with a reference to the basic data element or basic granule in the sieve attached) or a derived element (with a reference to the basic data element or basic granule in the sieve attached plus a reconstruction program that generates the derived element from the referenced basic data element). During the derivation step, a threshold for accepting the derivation can be set to be a fraction of the size of the original Huffman-encoded information present in the primary data field of the frame being reduced. Therefore, the derivation will not be accepted unless the sum of the reconstructed program and the references to the elementary data elements is less than this fraction of the size of the corresponding main data field of the MP3 encoded frame (which contained the Huffman encoded data).If the sum of the reconstructed program and the references to the elementary data elements is less than this fraction of the size of the existing main data field of the encoded MP3 frame (which contained the Huffman encoded data), then a decision can be made to accept the derivation.

[0265] The above-described method allows redundancy to be exploited at a global scope across multiple audio granules stored in the device. MP3-encoded data files can be converted to distilled MP3 data and stored in a lossless, reduced form. When retrieval is required, a data retrieval process (using an retrieval unit 1871 and a reconstructor 1872) can be invoked to reconstruct the MP3-encoded data 1873. In the device shown in Figure 18C, the reconstructor is responsible for executing a reconstruction program to generate the desired granules. The reconstructor is further modified to perform the Huffman encoding step (shown as Huffman encoding 1811 in Figure 18A) required to generate the MP3-encoded data. This data can then be fed to a standard MP3 decoder to reproduce the audio.

[0266] In this manner, a data distiller can be adapted and used to further reduce the size of MP3 audio files.

[0267] In another variation of the described scheme, upon receiving an MP3 encoded stream, the parser / factorizer takes the entire main data field as a candidate element for derivation or as a base data element for installation in a base data sieve. In this variation, all elements would continue to be Huffman coded, and the reconstruction program would operate on elements that have already been Huffman coded. This variation of the data distiller may be used to further reduce the size of MP3 audio files.

[0268] In a manner similar to that described in the previous section and illustrated in FIGS. 18A-18C, a Data Distillation™ device can be used to losslessly reduce video data. The data reduction achieved by this method is achieved by deriving components of the video data from basic data elements present in a content-associative sieve. A video data stream includes an audio component and a video component. The distillation method for the audio component has already been described. We now address the video component. A video component is typically organized as a series of images. The images begin with an I-frame, typically followed by a number of predicted frames (referred to as P-frames and B-frames). I-frames are typically bulky and contain complete snapshots of an image, while predicted frames are derived after using techniques such as motion prediction on the I-frame or on other derived frames. Some embodiments of the Data Distillation™ device extract I-frames from the video data as elements and perform a data distillation process on them, thereby retaining certain I-frames as basic data elements present in the content-associative sieve, while the remaining I-frames are derived from the basic data elements. The above method allows for the exploitation of redundancy at a global scope across multiple I-frames, both within a video file and across multiple video files. Because I-frames are typically bulky components of video data, this approach results in a reduction in the footprint of the video component. Applying the distillation technique to both the audio and video components results in a lossless reduction in the overall size of the video data.

[0269] FIG. 19 illustrates how the data distiller first shown in FIG. 1A can be improved to perform data reduction on video data. A stream of video data 1902 is received by a data distiller 1903 and reduced to a stream of distilled video data 1908, which is stored in a losslessly reduced form. The input stream of video data 1902 includes two components: compressed video data and compressed audio data. The output distilled video data produced by the device also includes two components: compressed video data and compressed audio data, but these components are further reduced in size by the data distiller 1903. A parser / factorizer 1904 extracts the compressed video data and compressed audio data from the stream of video data 1902 and extracts intra-frames (I-frames) and predicted frames from the compressed video data (including performing any necessary Huffman decoding). The I-frames are used as candidate elements 1905 to perform a content-associative lookup in a base data sieve 1906. The set of elementary data elements (also I-frames) returned by the elementary data sieve 1906 is used by the derivation unit 1910 to generate a lossless reduced or distilled representation of the I-frame, and the lossless reduced I-frame is placed in the distilled video data 1908. The distilled representation is encoded using the format shown in FIG. 1H, where each element in the distilled data is either a elementary data element (with a reference to the elementary data element in the sieve attached) or a derived element (with a reference to the elementary data element in the sieve attached as well as a reconstruction program that generates the derived element from the referenced elementary data element). During the derivation step, a threshold for accepting the derivation can be set to be a fraction of the size of the original I-frame. Thus, the derivation is not accepted unless the sum of the reconstruction program and the references to the elementary data elements is less than this fraction of the size of the corresponding I-frame. If the sum of the reconstruction program and the references to the elementary data elements is less than this fraction of the size of the original I-frame, a decision can be made to accept the derivation.

[0270] The above-described method allows redundancy to be exploited at a global scope across multiple I-frames of multiple video data sets stored in the device. When retrieval is required, a data retrieval process (using a retrieval unit 1911 and a reconstructor 1912) can be invoked to reconstruct video data 1913. In the device shown in Figure 19, the reconstructor is responsible for executing a reconstruction program to generate the desired I-frames. The reconstructor is further enhanced to combine compressed audio data with compressed video data (essentially the reverse of the extraction operation performed by the parser and factorizer 1904) to generate video data 1913. This data can then be fed to a standard video decoder to reproduce the video.

[0271] In this way, a data distiller can be adapted and used to further reduce the size of the video file.

[0272] The above description is presented to enable those skilled in the art to make and use the embodiments. Various modifications to the disclosed embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments and applications without departing from the spirit and scope of the present disclosure. Thus, the present invention is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and features disclosed herein.

[0273] The data structures and codes described in this disclosure may be partially or completely stored on computer-readable storage media and / or hardware modules and / or hardware devices. Computer-readable storage media include, but are not limited to, volatile memory, non-volatile memory, magnetic and optical storage devices such as disk drives, magnetic tape, CDs (compact discs), DVDs (digital versatile discs or digital video discs), or other media now known or later developed that can store code and / or data. Hardware modules or devices described in this disclosure include, but are not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated or shared processors, and / or other hardware modules or devices now known or later developed.

[0274] The methods and processes described herein may be embodied partially or completely as code and / or data stored on a computer-readable storage medium or device, so that when a computer system reads and executes the code and / or data, the computer system performs the associated methods and processes. The methods and processes may also be embodied partially or completely in hardware modules or devices, so that when the hardware modules or devices are activated, the associated methods and processes are performed. Note that the methods and processes may also be embodied using a combination of code, data, and hardware modules or devices.

[0275] The foregoing descriptions of embodiments of the present invention have been presented for purposes of illustration and description only. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Accordingly, many modifications and variations will be apparent to those skilled in the art. Also, the above disclosure is not intended to limit the invention.

Claims

1. 1. A method comprising: a processor factoring the data set into a series of data elements; The processor processes the losslessly reduced data set by, for each data element in the series of data elements: (1) converting the data element to a base data element and setting a creating locator address and a last used locator address associated with the base data element equal to the address of the data element within the data set; (2) if the data element is a duplicate of a first previously created base data element, replacing the data element with a reference to the first previously created base data element and setting a last used locator address associated with the first previously created base data element equal to the address of the data element within the data set; or (3) if the data element is not a duplicate of any previously created base data element, replacing the data element with a reference to a second previously created base data element and a series of transformations which, when applied to the second previously created base data element, result in the data element, and setting a last used locator address associated with the second previously created base data element equal to the address of the data element within the data set; generating the image by either and the processor determines an amount of memory expected to be used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements, the amount of memory allowing basic data elements to be streamed from storage and held in memory as needed for reconstruction, thereby eliminating the need for random storage accesses to fetch basic data elements during reconstruction.

2. determining an amount of memory used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements, Stepping through the locator addresses, for a given locator address: determining a first set of basic data elements by calculating the difference between (1) a second set of basic data elements whose associated creating locator addresses are less than or equal to said given locator address and (2) a third set of basic data elements whose associated last used locator addresses are less than said given locator address; determining a value for the amount of memory associated with the given locator address by summing the sizes of the first set of elementary data elements; and 2. The method of claim 1, further comprising determining a maximum memory amount value from the memory amount values ​​associated with the locator addresses.

3. determining an amount of memory used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements, dividing the locator addresses into a set of locator address ranges; and stepping through the set of locator address ranges, wherein for a given locator address range: determining a first set of basic data elements by calculating the difference between (1) a second set of basic data elements whose associated creating locator addresses are less than or equal to the end address of said given locator address range, and (2) a third set of basic data elements whose associated last used locator addresses are less than the start address of said given locator address range; determining a value for the amount of memory associated with the given locator address range by summing the sizes of the first set of elementary data elements; and 2. The method of claim 1, further comprising determining a maximum memory amount value from the memory amount values ​​associated with the set of locator address ranges.

4. a step of allocating memory of a size equal to or larger than the determined amount of memory by the processor; 2. The method of claim 1, further comprising the step of: the processor reconstructing the data set from the losslessly reduced data set using the allocated memory; wherein basic data elements are streamed from storage to the allocated memory as needed; and, if a given basic data element is used to reconstruct a data element, the given basic data element is deleted from the allocated memory after reconstructing the data element if the last used locator address of the given basic data element is equal to the address of the reconstructed data element.

5. each base data element is associated with a reuse count, and the reuse count is incremented each time the base data element is reused to losslessly reduce the data element by either replacing the data element with a reference to the base data element, or a reference to the base data element and a series of transformations, the method comprising: the processor allocating memory of a size equal to or greater than the determined amount of memory; 2. The method of claim 1, further comprising: the processor reconstructing the data set from the losslessly reduced data set using the allocated memory; wherein basic data elements are streamed from storage to the allocated memory as needed; a reuse count associated with a given basic data element is decremented each time the given basic data element is reused to reconstruct a data element; and the given basic data element is deleted from the allocated memory if the reuse count associated with the given basic data element is equal to zero.

6. 2. The method of claim 1, wherein the address of the data element is the sequence number of the data element in the series of data elements.

7. The method of claim 1, further comprising: receiving, by the processor, a memory budget value that is a maximum amount of memory that can be used during reconstruction of the data set from the losslessly reduced data set; During the step of generating the lossless reduced data set, the processor periodically estimating a partial amount of memory expected to be used during reconstruction of the data set from the losslessly reduced data set to date based on a creation locator address and a last used locator address; responsive to determining that the partial amount of memory is within a threshold distance from the memory budget value, creating a losslessly reduced data lot corresponding to a portion of the data set that has been losslessly reduced so far; and resuming the step of generating the losslessly reduced data set using a new data lot starting at the current position within the data set.

8. A computer readable program having instructions stored thereon, the instructions, when executed by a computer, causing the computer to: factoring the data set into a set of data elements; and generating a losslessly reduced data set, the losslessly reduced data set comprising, for each data element in the set of data elements: (1) converting the data element to a base data element and setting a creating locator address and a last used locator address associated with the base data element equal to the address of the data element within the data set; (2) if the data element is a duplicate of a first previously created base data element, replacing the data element with a reference to the first previously created base data element and setting a last used locator address associated with the first previously created base data element equal to the address of the data element within the data set; or (3) if the data element is not a duplicate of any previously created base data element, replacing the data element with a reference to a second previously created base data element and a series of transformations which, when applied to the second previously created base data element, result in the data element, and setting a last used locator address associated with the second previously created base data element equal to the address of the data element within the data set; and the instructions further cause the computer to: a computer-readable program that causes a computer to determine an amount of memory expected to be used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements, the amount of memory enabling basic data elements to be streamed from storage as needed and held in memory for reconstruction, thereby eliminating the need for random storage accesses to fetch basic data elements during reconstruction;

9. determining an amount of memory used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements; Stepping through the locator addresses, for a given locator address: determining a first set of basic data elements by calculating the difference between (1) a second set of basic data elements whose associated creating locator addresses are less than or equal to said given locator address and (2) a third set of basic data elements whose associated last used locator addresses are less than said given locator address; determining a value for the amount of memory associated with the given locator address by summing the sizes of the first set of elementary data elements; and 9. The computer readable program of claim 8, further comprising determining a maximum memory amount value from the memory amount values ​​associated with the locator addresses.

10. determining an amount of memory used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements; dividing the locator addresses into a set of locator address ranges; and stepping through the set of locator address ranges, wherein for a given locator address range: determining a first set of basic data elements by calculating the difference between (1) a second set of basic data elements whose associated creating locator addresses are less than or equal to the end address of said given locator address range, and (2) a third set of basic data elements whose associated last used locator addresses are less than the start address of said given locator address range; determining a value for the amount of memory associated with the given locator address range by summing the sizes of the first set of elementary data elements; and 9. The computer readable program of claim 8, further comprising determining a maximum memory amount value from the memory amount values ​​associated with the set of locator address ranges.

11. and storing further instructions that, when executed by the computer, cause the computer to: allocating memory of a size equal to or greater than the determined amount of memory; and reconstructing the data set from the losslessly reduced data set using the allocated memory, wherein basic data elements are streamed from storage to the allocated memory as needed, and wherein if a given basic data element is used to reconstruct a data element, the given basic data element is deleted from the allocated memory after reconstructing the data element if the last used locator address of the given basic data element is equal to the address of the reconstructed data element.

12. each base data element is associated with a reuse count, and each time the base data element is reused to losslessly reduce the data element by replacing the data element with either a reference to the base data element or a reference to the base data element and a series of transformations, the reuse count is incremented; and the computer readable program further stores further instructions that, when executed by the computer, cause the computer to: allocating memory of a size equal to or greater than the determined amount of memory; and reconstructing the data set from the losslessly reduced data set using the allocated memory, wherein basic data elements are streamed from storage to the allocated memory as needed, a reuse count associated with a given basic data element is decremented each time the given basic data element is reused to reconstruct a data element, and the given basic data element is deleted from the allocated memory if the reuse count associated with the given basic data element is equal to zero.

13. 9. The computer readable program of claim 8, wherein the address of the data element is a sequence number of the data element in the series of data elements.

14. and storing further instructions that, when executed by the computer, cause the computer to: receiving a memory budget value that is a maximum amount of memory that can be used during reconstruction of the data set from the losslessly reduced data set; During generation of said lossless reduced data set, periodically estimating a partial amount of memory expected to be used during reconstruction of the data set from the losslessly reduced data set to date based on a creation locator address and a last used locator address; responsive to determining that the partial amount of memory is within a threshold distance from the memory budget value, creating a losslessly reduced data lot corresponding to a portion of the data set that has been losslessly reduced so far; 9. The computer readable program of claim 8, further comprising: restarting generating the losslessly reduced data set using a new data lot starting at a current position within the data set.

15. 1. An apparatus comprising: a processor; a non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by the processor, causing the processor to: factoring the data set into a set of data elements; and generating a losslessly reduced data set, the losslessly reduced data set comprising, for each data element in the set of data elements: (1) converting the data element to a base data element and setting a creating locator address and a last used locator address associated with the base data element equal to the address of the data element within the data set; (2) if the data element is a duplicate of a first previously created base data element, replacing the data element with a reference to the first previously created base data element and setting a last used locator address associated with the first previously created base data element equal to the address of the data element within the data set; or (3) if the data element is not a duplicate of any previously created base data element, replacing the data element with a reference to a second previously created base data element and a series of transformations which, when applied to the second previously created base data element, result in the data element, and setting a last used locator address associated with the second previously created base data element equal to the address of the data element within the data set; and the instructions further include causing the processor to: determining an amount of memory expected to be used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements, the amount of memory enabling basic data elements to be streamed from storage and held in memory for reconstruction as needed, thereby eliminating the need for random storage accesses to fetch basic data elements during reconstruction.

16. determining an amount of memory used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements; Stepping through the locator addresses, for a given locator address: determining a first set of basic data elements by calculating the difference between (1) a second set of basic data elements whose associated creating locator addresses are less than or equal to said given locator address and (2) a third set of basic data elements whose associated last used locator addresses are less than said given locator address; determining a value for the amount of memory associated with the given locator address by summing the sizes of the first set of elementary data elements; and 16. The apparatus of claim 15, further comprising determining a maximum memory amount value from the memory amount values ​​associated with the locator addresses.

17. determining an amount of memory used during reconstruction of the data set from the losslessly reduced data set using creation locator addresses and last used locator addresses associated with the basic data elements; dividing the locator addresses into a set of locator address ranges; and stepping through the set of locator address ranges, wherein for a given locator address range: determining a first set of basic data elements by calculating the difference between (1) a second set of basic data elements whose associated creating locator addresses are less than or equal to the end address of said given locator address range, and (2) a third set of basic data elements whose associated last used locator addresses are less than the start address of said given locator address range; determining a value for the amount of memory associated with the given locator address range by summing the sizes of the first set of elementary data elements; and 16. The apparatus of claim 15, further comprising determining a maximum memory amount value from the memory amount values ​​associated with the set of locator address ranges.

18. The non-transitory computer-readable medium stores further instructions that, when executed by the processor, cause the processor to: allocating memory of a size equal to or greater than the determined amount of memory; 16. The apparatus of claim 15, further comprising: reconstructing the data set from the losslessly reduced data set using the allocated memory; wherein basic data elements are streamed from storage to the allocated memory as needed; and wherein, if a given basic data element is used to reconstruct a data element, the given basic data element is deleted from the allocated memory after reconstructing the data element if the last used locator address of the given basic data element is equal to the address of the reconstructed data element.

19. each base data element is associated with a reuse count, and each time the base data element is reused to losslessly reduce the data element by replacing the data element with either a reference to the base data element or a reference to the base data element and a series of transformations, the reuse count is incremented; and the non-transitory computer readable medium stores further instructions that, when executed by the processor, cause the processor to: allocating memory of a size equal to or greater than the determined amount of memory; and reconstructing the data set from the losslessly reduced data set using the allocated memory, wherein basic data elements are streamed from storage to the allocated memory as needed, a reuse count associated with a given basic data element is decremented each time the given basic data element is reused to reconstruct a data element, and the given basic data element is deleted from the allocated memory if the reuse count associated with the given basic data element is equal to zero.

20. 16. The apparatus of claim 15, wherein the address of the data element is a sequence number of the data element in the series of data elements.

21. The non-transitory computer-readable medium stores further instructions that, when executed by the processor, cause the processor to: receiving a memory budget value that is a maximum amount of memory that can be used during reconstruction of the data set from the losslessly reduced data set; During generation of said lossless reduced data set, periodically estimating a partial amount of memory expected to be used during reconstruction of the data set from the losslessly reduced data set to date based on a creation locator address and a last used locator address; responsive to determining that the partial amount of memory is within a threshold distance from the memory budget value, creating a losslessly reduced data lot corresponding to a portion of the data set that has been losslessly reduced so far; 16. The apparatus of claim 15, further comprising: restarting generating the losslessly reduced data set using a new data lot starting at a current position within the data set.

Images