Stream compression and decompression

Abstract

A method for compressing a sequence of records, each record comprising a sequence of fields, comprises steps of buffering a record in a line of a matrix, reordering the lines of the matrix according to locality sensitive hash values of the buffered records such that records with similar contents in corresponding fields are placed in proximity, and consolidating fields in columns of the matrix into a block of codes. In this, consolidating yields codes of one of a first type comprising a sequence of individual fields and a second type comprising a sequence of fields with at least one repetition. The second type of code comprises a presence field indicating repeated fields and an iteration field indicating a number of respective repetitions. Decompression of the records from the block codes compressed above is also described.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS[0001]This application is based upon and claims priority from prior U.S. patent application Ser. No. 13 / 553,457, Attorney Docket No. CH920110018US1, filed on Jul. 19, 2012, which in turn is based upon and claims priority from prior European Patent Application No. 11175194.4, filed on Jul. 22, 2011, the entire disclosure of each are hereby incorporated by reference in their entirety.FIELD OF THE INVENTION[0002]The present invention relates to a system and a method for data stream compression and another system and another method for data stream decompression.[0003]The invention also relates to a computer program product for carrying out one of the methods.BACKGROUND OF THE INVENTION[0004]In many technical applications, a stream of data is generated which must be recorded for later analysis. Such applications may be found in domains such as network forensics and cyber-security, scientific data analysis (e.g. archiving and analysis of astronomical...

AI Technical Summary

Benefits of technology

[0008]The resulting codes may advantageously provide a high compression ratio especially for data that comprises fixed fields such as header fields of TCP packets, time-tagged data or categorized measurements. Furthermore, the resulting codes may allow simplified access to the data without the requirement of decompressing all blocks containing data of interest.

[0027]Both the compression and the decompression apparatus may easily be integrated into a communication or measurement system where large volumes of data need to be archived and processed at a later time. Efficiency of data analysis may thus be improved which may lead to cost and advantages and improved control over the data. Especially in data analysis that comprises access to a varying number of records per block of codes, the decompression method and the decompression apparatus may be advantageously used. Such analysis often must be performed in “needle in a haystack” type of searches, which are common for network forensics.

Problems solved by technology

Often, the amount of data generated per time is very large so that extremely fast processing is mandatory to prevent the processing systems from clogging up with data.

In known archiving solutions, accessing the archived data may be slow because searching and decompressing the requested data is a computationally intensive task.

Images