Bandwidth sensitive data compression and decompression

Abstract

Data compression and decompression methods for compressing and decompressing data based on an actual or expected throughput (bandwidth) of a system. In one embodiment, a controller tracks and monitors the throughput (data storage and retrieval) of a data compression system and generates control signals to enable / disable different compression algorithms when, e.g., a bottleneck occurs so as to increase the throughput and eliminate the bottleneck.

Description

CROSS-REFERENCE TO RELATED APPLICATION[0001]This application is based on U.S. provisional application Ser. No. 60 / 268,394, filed on Feb. 13, 2001, which is fully incorporated herein by reference.BACKGROUND[0002]1. Technical Field[0003]The present invention relates generally to data compression and decompression and, in particular, to a system and method for compressing and decompressing data based on an actual or expected throughput (bandwidth) of a system that employs data compression. Additionally the present invention relates to the subsequent storage, retrieval, and management of information in data storage devices utilizing either compression and / or accelerated data storage and retrieval bandwidth.[0004]2. Description of Related Art[0005]There are a variety of data compression algorithms that are currently available, both well defined and novel. Many compression algorithms define one or more parameters that can be varied, either dynamically or a-priori, to change the performanc...

AI Technical Summary

Benefits of technology

"The present invention is a system and method for compressing and decompressing data based on its actual or expected throughput. The system includes a data compression system, compression routines, and a controller for tracking system throughput and selecting the appropriate compression routine based on the data type. The system can also compress and store data to a storage device, as well as transmit it over a communication channel. The technical effect of the invention is to optimize data compression and improve throughput performance in a data processing system."

Problems solved by technology

Indeed, a large dictionary may be employed to yield very good compression ratios but the algorithm may take a long time to execute.

If speed were more important than compression ratio, then the algorithm can be limited by selecting a smaller dictionary, thereby obtaining a much faster compression time, but at the possible cost of a lower compression ratio.

Thus, one challenge in employing data compression for a given application or system is selecting one or more optimal compression algorithms from the variety of available algorithms.

Algorithms that compress particularly well usually take longer to execute whereas algorithms that execute quickly usually do not compress particularly well.

Yet another problem within the current art is data storage and retrieval bandwidth limitations.

Limitations in process technologies currently prohibit placing a sufficient quantity of onboard memory for most applications.

Thus one additional problem with the existing art is that memory storage devices severely limit the performance of consumer, entertainment, office, workstation, servers, and mainframe computers for all disk and memory intensive operations.

For example, magnetic disk mass storage devices currently employed in a variety of home, business, and scientific computing applications suffer from significant seek-time access delays along with profound read / write data rate limitations.

Another problem within the current art is that emergent high performance disk interface standards such as the Small Computer Systems Interface (SCSI-3), iSCSI, Fibre Channel, AT Attachment UltraDMA / 100+, Serial Storage Architecture, and Universal Serial Bus offer only higher data transfer rates through intermediate data buffering in random access memory.

These interconnect strategies do not address the fundamental problem that all modern magnetic disk storage devices for the personal computer marketplace are still limited by the same typical physical media restriction.

Thus another problem with modern data storage systems utilizing RAID systems is that a linear increase in data bandwidth requires a proportional number of added disk storage devices.

Another problem with most modern mass storage devices is their inherent unreliability.

More complex techniques are also employed that support various trade-offs between data bandwidth and data reliability.

A failure of a single drive results in a complete loss of all data.

Thus another problem with RAID systems is that low cost improved bandwidth requires a significant decrease in reliability.

This tends to decrease data bandwidth from even that of a single comparable disk drive.

Hence, another problem with RAID systems is the high cost of increased reliability and associated decrease in performance.

Thus another problem with redundant modern mass storage devices is the degradation of data bandwidth when a storage device fails.

Additional problems with bandwidth limitations and reliability similarly occur within the art by all other forms of sequential, pseudo-random, and random access mass storage devices.

Yet another problem within the current art is the application and use of various data compression techniques.

Lossy data compression techniques provide for an inexact representation of the original uncompressed data such that the decoded (or reconstructed) data differs from the original unencoded / uncompressed data.

For example, lossy data compression of visual imagery might seek to delete information content in excess of the display resolution or contrast ratio of the target display device.

One fundamental problem with almost all modern approaches is the compression ratio to encoding and decoding speed achieved.

As previously stated, the current theoretical limit for data compression is the entropy limit of the data set to be encoded.

However, in practice, many factors actually limit the compression ratio achieved.

This is followed in turn by dictionary compression, Huffman coding, and run-length coding with respectively decreasing execute times. What is not apparent from these algorithms, that is also one major deficiency within the current art, is knowledge of their algorithmic efficiency.

This is almost never achieved in practice due to limitations within modern optimizing software compilers.

Yet another problem within the current art relates to data management and the use of existing file management systems.

Furthermore, these bad sector replacement maps are definable on a disk by disk basis.

Practical limitations in the size of the data required to both represent and process an individual data block address, along with the size of individual data blocks, governs the type of file allocation tables currently in use.

Thus one limitation within the current art is that disk file allocation tables and associated file management systems are a compromise between efficient data storage, access speed, and addressable data space.

Thus another significant problem within the current art is that file systems are not able to randomly access compressed data in an efficient manner.

Further aggravating this problem is the fact that when data is decompressed, processed and recompressed it may not fit back into its original disk space, causing disk fragmentation or complex disk space reallocation requirements.

For very small files this technique is often adequate, however for larger files the compression and decompression times are too slow, resulting in inadequate system level performance.

In addition, the ability to access randomly access data within a specific file is lost.

Thus file by file compressed data management is not an adequate solution.

A group of blocks is compressed and decompressed as a unit, thus there exists practical limitations on the size of GOBs.

If GOBs are small compression ratios are low and processing time short.

Conversely, when GOBS are large compression ratios are higher and processing time is longer.

Several problems exist within the current art for the management of compressed blocks.

However as files are processed within the computers, files may grow or shrink in size.

Inefficient disk storage results when a substantial file size reduction occurs.

Conversely when a file grows substantially, the additional space required to store the data may not be available contiguously.

The result of this process is substantial disk fragmentation and slower access times.

One problem with this method is that average file access times are substantially increased by this technique due to the random data storage.

Peak access delays may be reduced since the statistics behave with a more uniform white spectral density, however this is not guaranteed.

A further layer of complexity is encountered when compressed information is to be managed on more than one data storage device.

Images