Automatic detection of separators for compression

Active Publication Date: 2011-11-01
AT&T INTPROP I L P +1
View PDF8 Cites 10 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Benefits of technology

An advance in the art is realized with a method that accepts a data file, iteratively tests different information units as record delimiters and field delimiters, and chooses as the data files record delimiter, R, and field delimiter, F, the information units that result in the lowest variability that is computed on fields created by use of the chosen delimiter pair R,F. In an illustrative embodiment, processing time is reduced by first identifying a subset of likely record delimiters, and restricting the search to that subset.

Problems solved by technology

Unfortunately, in applications such as data compression and data structure discovery, one is often presented with a data file without any extra information.
Such approaches are not effective for handling large volumes of data, so there is a pressing need for tools and techniques to automate the structure-extraction process.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Automatic detection of separators for compression
  • Automatic detection of separators for compression
  • Automatic detection of separators for compression

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

An underlying assumption of this invention is that a presented data file is a relational table. Such a table typically contains some fields in which data are restricted and have low variability relative to all data or even to data in other fields. For example, take a relational data file with a field consisting of social security numbers among other fields containing other information such as job titles or home addresses. There is great variability when all data of a record are considered together. However, the social security field consists of precisely 9 digits. Likewise, even though a field of encrypted passwords may contain random characters with large variability, the length of an instance might not be fewer than some preordained number and typically not many more, and the alphabet would not include unprintable characters.

The following assumes the use of some function E(s) that can compute the variability of a collection of characters “s”. Such a function may be customizable to...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

A method that accepts a data file, iteratively tests different information units as record delimiters and field delimiters, and chooses as the data files record delimiter, R, and field delimiter, F, the information units that result in the lowest generalized entropy that is computed on fields created by use of the chosen delimiter pair R,F.

Description

BACKGROUNDThis relates to data files and, more particularly, to large files.At times one is faced with some large file whose structure is known only to the extent that it consists of records, that the records are separated by some (unknown) record delimiter, that each record comprises fields, and that the fields are separated by some (unknown) field delimiter. These files, which are often produced by databases and information processing systems, are called relational tables.Modern information systems routinely generate and store massive relational tables. In the context of IP networks, for example, this includes a wealth of different types of collected data, including traffic (e.g., packet of low level traces), control (e.g., router forwarding tables, BGP and OSPF updates) and management (e.g., fault, SNMP traps) data. It is beneficial to process this type of data into forms that enhance data storage, access, and transmission. In particular, good compression can help to significantl...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F17/30908G06F16/80
InventorVO, KIEM-PHONGSEN, SUBHABRATA
OwnerAT&T INTPROP I L P