Methods of creating a dictionary for data compression

Abstract

Some aspects of the invention provide methods, systems, and computer program products for creating a static dictionary in which longer byte-strings are preferred. To that end, in accordance with aspects of the present invention, a new heuristic is defined to replace the aforementioned frequency count metric used to record the number of times a particular node in a data tree is visited. The new heuristic is based on counting the number of times an end-node of a particular byte-string is visited, while not incrementing a count for nodes storing characters in the middle of the byte-string as often as each time such nodes are visited. The result is an occurrence count metric that favours longer byte-strings, by being biased towards not incrementing the respective occurrence count values for nodes storing characters in the middle of a byte-string.

Description

FIELD OF THE INVENTION [0001] The invention relates to data compression and, in particular, to methods, systems, and computer program products for creating a dictionary for use in data compression and expansion. BACKGROUND OF THE INVENTION [0002] Data compression methods can be divided into two broad categories: namely, “lossy” and lossless data compression methods. Lossy data compression methods result in a loss of some information during compression. On the other hand, lossless data compression refers to the ability to change a data set, without losing information, so that the data set can be stored within less space after compression as compared to before compression. Lossy compression methods are often employed in situations where a loss of information is tolerable (e.g. compression of audio and video data). By contrast, lossless compression methods are preferably employed in situations where a loss of information is undesirable and fidelity is a priority (e.g. compression of te...

AI Technical Summary

Benefits of technology

The invention is a method for creating a static dictionary for use in lossless data compression and expansion. The method involves creating a plurality of data trees, each tree containing a root node and child nodes. Each child node stores a unique binary pattern and an occurrence count value associated with it. The data trees are then joined to form a single string, which contains the binary patterns in a direct path from the root node to the child nodes. The occurrence count value of the string is the occurrence count value of the child node. The method also includes incrementing the occurrence count value when the child node is visited and removing binary pattern strings that have low occurrence count values. The system includes a computer program product that implements the method. The technical effects of the invention include improved data compression and expansion capabilities, as well as improved efficiency and flexibility in the creation of the static dictionary.

Problems solved by technology

Lossy data compression methods result in a loss of some information during compression.

However, adding and maintaining additional blocks of memory adds capital and operating costs.

In the context of large database systems, such as those employed in the financial services sector, such capital and operating cost increases can make database management very expensive.

However, there are a number of problems associated with this.

The most frequently occurring byte-strings are typically quite short, which means that the longest byte-strings that could be used may not be stored for use in the static dictionary since the dictionary is biased towards retaining shorter more frequently occurring byte-strings.

Images