Lossless data compression

Abstract

Method for improved implementation of a system comprising an analyzer which includes a data acquisition system configured to generate a stream of digital data comprising N numbers of size n corresponding to a size of a measured variable, in which at least some of the numbers include at least one zero, and wherein the system includes a processing device connected to a data acquisition system, the method comprising: supplying the processing device with an initial stream of digital data from the data acquisition system; causing the processing device to segment at least a portion of the data stream to form at least one segmented data array; causing the processing device to transpose the at least one segmented data array to form at least one transposed data array;Causing the processing device to split the at least one transposed data array to form at least one nested data array; causing the processing device to perform run-length encoding of leading zeros of the numbers to form compressed data in order to reduce the amount of memory required to store the data, and storing the compressed data in memory.

Description

Prior ArtUS 5,678,043 discloses a method of compressing and storing data in relational databases in which data sets are treated as numbers, are sorted up or down according to a predetermined sorting rule, then for each data set the difference between the number of the data set and another number of another data set is calculated, the data sets then being represented by the corresponding difference. By representing the differences between the data sets, the data sets are smaller, so that less memory is needed and compression is achieved.Object of the InventionIt is an object of the invention to provide a data compression system and method using a lossless data compression technique that can be implemented under a minimum of hardware and software resources.Brief Summary of the InventionOne aspect of the present disclosure is an analysis apparatus having a data acquisition system that generates data. The data acquisition system may comprise a spectrometer or other device. The analyzer includes a data compression system and method that uses a lossless data compression technique that can be implemented under a minimum of hardware and software resources. The method can be implemented such that it can be divided into many parallel operations. The method may be implemented using software and / or processing devices, such as field programmable gate arrays (FPGHs) or graphics processing units (GPUs).Another aspect of the present disclosure is a method that brings similarity across different dimensions of a dataset. These similarities may exist in temporal and / or spatial dimensions. However, the present invention is not limited to these dimensions.Another aspect of the present disclosure relates to a method for improving the operation of a system comprising an analyzer having a data acquisition system configured to generate a stream of digital data comprising N numbers of size n. The numbers correspond to a magnitude of a measured variable, where at least some of the numbers include at least one 0. The system includes a processing device operatively connected to the data acquisition system. The data acquisition system and processing device may be housed within the analyzer, or the system may have separate components operatively connected using "hard" lines, wireless communication devices, or other suitable arrangement. The method includes providing the processing device with an original stream of digital data from the data acquisition system and causing the processing device to segment the stream of data to form at least one segmented data array. The method also includes causing the processing device to transpose the at least one segmented data array to form at least one (segmented) transposed data array. The method may include causing the processing device to share the at least one transposed data array to form at least one interleaved data array. The processing apparatus operates run-length encoding of leading zeros of the numbers to form compressed data and reduce the amount of memory required to store the data. The compressed data is then stored in memory. The stream of data may be segmented into n x n segmented data arrays. The segmented data arrays can be transformed into at least one interleaved data array of size (2n) x (n / 2). The transposed data array can optionally be reversed. Also, numbers that do not include at least one leading zero may not be compressed, and uncompressed data may be stored with the compressed data in a storage unit. The compressed and / or uncompressed data may be transferred (via a wire / wire or wirelessly) to a computer having a processor configured to decompress the compressed data.Another aspect of the present disclosure is an analyzer having a data acquisition system configured to generate a stream of digital data comprising N numbers of size n corresponding to a size of a measured variable in which at least some of the numbers include at least one zero. The analyzer includes a processing device configured to receive a stream of data from the data acquisition system. The processing device is configured to segment the stream of data to form at least one segmented data array, transpose the at least one segmented data array to form at least one transposed data array. The processor is also configured to share at least one transposed data array to form at least one interleaved data array and to run-length encode leading zeros of the numbers to form compressed data and reduce the amount of memory required to store the compressed data. The processing device is also configured to store the compressed data in memory. The processing device and the memory can be accommodated within the analysis device. Optionally, these components may be separate components operatively connected to the data acquisition system.These and other features, advantages and objects of the present invention will be made apparent to those skilled in the art upon reference to the following description, claims and drawings.Brief Description of the DrawingsFIG. 1 is a block diagram illustrating an analyzer including a data acquisition system according to an aspect of the present disclosure; FIG. 2 is a flow chart illustrating the operating system of FIG. 1; and FIG. 3 shows the processing steps for a segmented data array.Detailed DescriptionFIG. 1 shows a system 10 with an analysis device 20, which can be operatively connected to a computer device, such as a PC 30. The analyzer 20 may comprise a spectrometer, such as an incandescent atom emission spectrometer. Analyzer 20 may also include other devices, such as a gravimetric analyzer, an ash fusion measurement device, a calorimeter, an elemental analyzer, a digital microscope, a mass spectrometer, an audio device, or other device that generates and / or processes data. The analyzer 20 includes a data acquisition system 22 configured to generate a stream of digital data comprising N numbers of size n corresponding to a size of a measured variable in which at least some of the numbers include at least one zero. The data produced by the data acquisition system 22 may be in the form of a stream of data in which many, most or all of the individual numbers include at least one leading zero. For example, if the individual numbers are 16-bit numbers corresponding to the size of a measured variable, the numbers may include one or more leading zeros if the size of the measurement variable is less than a maximum possible size. Here, it is to be understood that the numbers are formatted in a reverted row such that the zeros appear as trailing zeros corresponding to less than a maximum possible size than leading zeros. As used herein, the terms "leading zero" and "leading zeros" generally refer to any numerical format in which numbers that are less than a maximum possible value include at least one zero.The system 10 also includes a processing device 24. the processing device 24 may be a field programmable array (FBGA), a graphics processing unit (GPU), or another suitable processor. It should be understood that the processing device 24 could include virtually any type of processor. The processing device 24 may be housed within the analyzer 20. Alternatively, the processing device 24 may be located separately from the analyzer 20 and / or data acquisition system 22.Accordingly, it should be understood that the arrangement of FIG. 1 is not necessarily limited to a specific physical configuration.The processing device 24 may be programmed to execute an algorithm for compressing and / or decompressing data generated by the data acquisition system 22. In FIG. 1, a compression / decompression algorithm is denoted as "26A" and a compression algorithm is denoted as "26B". However, these determinations are provided herein for purposes of explanation, and the present disclosure is not necessarily limited to separate compression and decompression algorithms / processes. The processing device 24 is connected to the memory 28. The memory 28 may be part of the data acquisition system 22, or may include a separate memory connected to the processing device 24. Generally, the memory 28 may include a hard disk, a semiconductor memory that is portable or fixed, an optical storage device (e.g., optical disk unit and optical disks / CD / DVD, etc.), or other suitable memory.Analyzer 20 may optionally be linked to a computer, such as a PC 30, by line 32. In this case, the line 32 can have a wireless connection, cable or the like data transfer arrangement, such as a flash memory, an optical disk or the like, which can be used for the data transfer from the analysis device 20 to the PC 30. As will be described in more detail below, the PC 30 may include a decompression algorithm 26C and a compression / decompression algorithm 26A. It is understood that these algorithms, processors are not necessarily separate and the terms "26A" and "26C" are provided for purposes of description. Further, analyzer 20 may include only compression algorithms 26B, and computer 30 may also include only one decompression algorithm 26C. The PC 30 includes storage capacity such as a mass storage unit 34. the mass storage unit 34 may include a floppy disk drive, a semiconductor memory, or a movable data storage such as a disk drive / drive, flash memory, or the like.Referring to FIG. 2, the compression / decompression method 40 generally starts at step 42. At step 42, an incoming raw data stream is received by a processor, such as a processor device 24. The incoming raw data stream may comprise a group of N numbers, each number comprising n bits. For example, each number may comprise 4, 8, 16, 32, 64, or more bits. At step 44, the incoming raw data stream is segmented to form segmented arrays. For example, 16 x 16-bit segmented arrays are formed for 16-bit data. At step 46, each segmented array is transposed. At step 48, the series of numbers is optionally reversed to form a reversed segmented array.At step 50, the segmented (and optionally reversed) array is divided or transformed into an interleaved array. For example, a 16 x 16 bit array becomes a 32 x 8 bit interleaved array.At step 52, the algorithm determines whether a first data location of each number is 0. For example, if a 32 x 8 bit interleaved array is formed at step 50, the algorithm determines at step 52 whether each 8 bit number includes a leading 0. As described above, the "first" data location would not necessarily require a leading zero to be present if the numbers were formatted such that there are trailing zeros if an individual number was less than a maximum possible value. In general, the first data location zero condition could be whenever a single digit is less than a maximum possible value for the number, regardless of the format of the digits.If the first data location is zero (step 52), then the run length algorithm encodes the leading zeros as shown at step 54, and the run length data and remaining data (i.e., the non-zero portion of each number) is then formed into an outgoing data stream of the compressed data as shown at step 56. At step 58, the compressed data stream from step 56 is combined with uncompressed data from step 52 and stored in memory. The uncompressed data of step 52 includes numbers that do not have a null as the first data location.At step 60, the compressed data is decompressed and collected (i.e., combined) with the uncompressed data previously stored at step 58 to form the original raw data stream. As noted above, the stored data (step 58) may be stored in a memory 28 associated with the analyzer 20, and / or stored on a mass storage unit 34 of a PC 30 or the like. Similarly, the data may be decompressed by the processor device 24 associated with an analyzer 20, and / or the data may be decompressed using a personal computer 30 or other suitable device.Generally, decompression involves separation of the compressed data from the uncompressed data, and steps 44, 46, 48, 50, 54, and 56 are then reversed and inverse operations are performed. For example, run length decoding may be performed to generate numbers that include one or more zeroes, and interleaved arrays may be formed. The interleaved arrays are transformed into transposed groups of size n x n. If the array row was reversed during compression, the array is reversed during the decompressing operation to reform the transposed segmented array. The transposed segmented array is then transposed to form segmented fields which are then combined (collected) to reform the original raw data stream.FIG. 3 is an example to illustrate a process of FIG. 2 The example uses a data array of 16-bit numbers of size N. Typically, this data is streamed from a device, such as an analog / digital converter (ADC). The data may comprise virtually any type of data. For example, the data may include measurements generated by devices such as glow discharge atomic emission spectrometers, thermal gravimetric analyzers, ash fusion measurement devices, calorimeters, elemental analyzers, digital microscopes, mass spectrometers, audio devices, or virtually any other types of devices that measure, generate, and / or process data. For example, the system / apparatus / processor of the present disclosure may be implemented in conjunction with spectrometers according to U.S. Patents 9,425,034; 9,406,493; 9,312,119; 9,048,080; 7,385,187; 6,023,330.The data may contain significant numbers of zeroes. For example, in an audio-based system, the volume of the sound may be represented by an n-bit (e.g., 16-bit) number. In many cases, a majority of a measured sound may not be exceptionally loud, such that all or most of the measured data includes at least one zero, indicating that the measured audio signal has not reached the maximum value.In the following example, each data point is represented using a 16-bit number. However, it should be understood that more or fewer bits may be used as required for a particular application. For example, each data point may have an 8-bit, 32-bit, 64-bit number. Further, it is to be understood that the following example is not limited to any particular file type.Table 1 represents a number (stream) of data (16-bit numbers) of an N size: The first step of the compression method is to segment the array (stream) of raw data into at least one n x n segmented array. In the case of 16 digits, each of 16-bit size, 16 x 16 segmented arrays are formed as shown in Table 2: Because the method uses smaller portions of the raw data stream, there is typically no need to store the total number of N sizes before the compression process begins.The next step is the transposition of the n x n (e.g., 16 x 16) segmented array by each bit location according to Table 3: This results in binding or positioning all zeros on one side of the array.As discussed below in connection with FIG. 3, the series of numbers may be reversed to shift zeros to the other side of the array to form a reversed transposed array.The data constituting the transposed (reversed) array is then broken into 8-bit pairs to form an interleaved data array as shown in Table 4:07,156,155,154,153.152,151.150,15115,1514,1513.1512,1511.1510,159,158,1527,146,145,144,143,142,141.140,14315,1414,1413,1412,1411,1410,149,148.1447,136,135,134,133.132,131,130.13...3115,014,013,012,011,010.09.08.0Breaking the bits into 8-bit pairs (interleaved array) as shown in Table 4 saves a single byte that can be used to store the run length information. The interleaved array is generally in the form of a (2n) x (n / 2) array.Next, the method involves run length encoding only the first data (row) in the array according to Table 5:0RLL7RLL6RLL5RLL4RLL3RLL2RLL1RLL01UD0,7UD0,6UD0,5UD0,4UD0.3UD0,2UD0,1UD0,02UD1,7UD1.6UD1,5UD1,4UD1,3UD1.2UD1,1UD1,03UD2,7UD2,6UD2,6UD2,4UD2,3UD2,2UD2,1UD2,0...Data that is deeper in the field is typically "feature richer" and less compressible. To compress this data, it is determined whether the first data location is 0. If it is not zero, no compression will take place. In the example of Table 5, the RLL (run length) may be up to 32 bytes. In Table 5, du represents the remaining uncompressed data.Figure 3 is an embodiment illustrating the processing steps for a segmented array of data. In Figure 3, the method starts with 16 numbers as shown in column 1 ("raw data"). In column 1, the raw data is segmented to form a 16 x 16 bit array. The raw data is then transposed bit by bit as shown in column 2 ("transposed"). The row of numbers is then reversed, as shown in column 3 ("reversed array"). As discussed above, reversion of the array is optional. The data (reversed array) is then converted from 16 x 16-bit words to 32 x 8-bit bytes to form an interleaved array as shown in column 4 ("interleaved array"). The leading zeros are then run length encoded to form the final compression, as shown in column 5 ("RLL").Decompression may be performed by reversing the steps noted above using inverse functions to reproduce the uncompressed data as shown in column 6 ("uncompressed"). More specifically, for decompressing the data of column 5, an inverse run length encoding step is first performed to form an interleaved array (column 4). The 32 x 8-bit bytes (interleaved array) are then converted to 16 x 16-bit words (reversed array). The row of the reversed array is then reversed (if the reversion occurs during compression) to form a transposed array. The transposed array is then transposed to form the original segmented 16 x 16 bit raw data array. Individual segmented arrays may then be assembled to form the original N-size data array.Referring again to columns 4 and 5 (FIG. 1 ), in the illustrated example, sub-array 16 is integer. The first byte (column 5) is the number of zeroes in the sub-array (column 4). Thus, in the illustrated embodiment, the first byte (10) in column 5 corresponds to the number of zeros (10) at the top of the interleaved grouping of column 4. the first byte (column 5) may be subtracted from the total to determine the number where the next sub-array begins.The above-described process / algorithm was tested using data from various data acquisition systems. In the cases tested, the compression ratio was more than 20 percent. Adding further steps may result in higher ratios, but would require more resources. Here, it is to be understood that the format may be changed to 32-bits, 64-bits, 8-bits, etc. for a particular application.Another aspect of the present disclosure includes exploiting the similarity of varying dimensions of a dataset. The initial expansion in dimensionality of the lossless data compression approach takes advantage of the similarity between a dataset in time. Considering the differences between two observations of such a record in time, not all of the bits will be used, thus enabling compression with lossless data compression as described above.The following describes this (optional) aspect of a method or process according to the present disclosure. In the following discussion, Y represents observations and is of the size T x N.T represents the total number of samples in time and N represents the length of each individual data set. Y(0,n) represents the first data set in time at location n, Y(l,n) represents the second data set in time at location n, and Y(T,n) represents the last data set in time at location n.D(t,n) represents the difference between the two data sets at time t at location n. Note the exception to the first value D.The data contained in D thus have both positive and negative numbers, since Y does not necessarily have to be monotone. To preserve the sign of the difference, another parameter is used with S.Since the sign information is preserved, the absolute value of D can be taken, thus eliminating the need for storing a two's complement for negative numbers. This operation results in zeros present in the upper bits of the D values, resulting in highly compressible data using the method described above in connection with Figure 2 and Tables 1-5.D(1,1:n) is compressed for each observation at time t based on the initial description of the method above in connection with Figure 2 and Tables 1-5. This is ensured by storing the character information between the RLL and UD data as shown in Table 6:0RLL7RLL6RLLSRLL4RLL3RLL2RLL1RLLO1UD0,7UD0,6UD0,5UD0,4UD0,3UD0,2UD0,1UD0,02UD1,7UD1,6UD1,5UD1,4UD1,3UD1,2UD1,1UD1,03UD2,7UD2,6UD2,6UD2,4UD2,3UD2,2UD2,1UD2,0...Table 6 is substantially the same as Table 5 (above). The run length (RLL) may be up to 32 bytes in this example. The remaining uncompressed data (UD) is shown in Table 6.The following is used to store the character information of each difference over the SD field:RLL - The run length may be up to 32 bytes in this example.SD - Each bit represents the sign of a difference at location n. This is stored as a 16-bit number based on the example using a 16-element sub-array.UD - is the remaining uncompressed data as shown in Table 7:0RLL7RLL6RLLSRLL4RLL3RLL2RLL1RLLO1SD15SD14SD13SD12SD11SD10SD9SD82SD7SD6SD5SD4SD3SD2SD1SDO3UD0,7UD0,6UD0,5UD0,4UD0,3UD00, 2UD0,1UD0,04UD1,7UD1,6UD1,5UD1,4UD1,3UD1,2UD1,1UD1,05UD2,7UD2,6UD2,6UD2,4UD2,3U02,2U02,1UD2,0...Note that if t=0, all of the character bits will be zero, since there is actually no difference.Decompression of a sub-array is performed using the RLL and UD fields as described above in connection with Tables 1-5 and Figures 1 and 2. However, the SD field must be preserved such that it can be used to determine whether the data at time t is a sum or difference of the data at time t-1 for each data point n. As an example:Decompressing D(0,n) gives Y(0,n)In decompression D(1,n), the value of Y(1,n) is found by the following equation:Y(1,n)=Y(0,n)+D(1,n) when SD(1,n) is 0Y(1,n)=Y(0,n)-D(1,n) when SD(1,n) is 1The above method / algorithm provides lossless data compression that yields compressibility simultaneously in multiple dimensions. The first dimension is the spatial dimension of the data that results in the fact that most of the data are at a same intensity. The second dimension brings the fact that the data has little variability over time. The combination of these two methods results in data compressions of more than 50%.As set forth above in connection with Tables 1-5 and Figs. 1 and 2, the data format is not limited to a size of 16 bits. Similarly, the method described above in connection with Tables 6 and 7 is not limited to initial 1 x N data sets, nor 1 x N data sets observed at T times, i.e., T x N size data sets. The methods of the present invention can be applied to M x N size data sets, such as images or M x N x T size data sets, such as videos.As indicated above, the methods of the present invention can be used in conjunction with a variety of devices. Such devices may include analog / digital converters that generate a stream of digital data that may be compressed using the methods described above. According to compression and / or data processing, the data may be used to form a data display on a screen. For example, the data may be presented in graphical format on the screen of an analysis instrument having a data collection system / device or a laptop or other suitable device screen.The methods described herein may be performed by a processing device 24 executing the instructions of the algorithms 26A and / or 26B described above. These instructions may be stored in a non-transitory computer readable medium, such as a memory 28.Further, it should be understood that the data is initially compressed using a device such as an FPGA or GPU of a data collection system / device, and the data may be later decompressed by the FPGA or GPU, or the data may be decompressed using a laptop or other suitable device.

Claims

A method for improved performance of a system including an analyzer including a data acquisition system configured to generate a stream of digital data comprising N numbers of size n corresponding to a size of a measured variable in which at least some of the numbers include at least one zero, and wherein the system includes a processing device connected to a data acquisition system, the method comprising: applying an original stream of digital data from the data acquisition system to the processing device; causing the processing device to segment at least a portion of the data stream to form at least one segmented data array; causing the processing device to transpose the at least one segmented data array to form at least one transposed data array; causing the processing device to share the at least one transposed data array to form at least one interleaved data array; causing the processing device to run-length encode leading zeros of the numbers to form compressed data to reduce the amount of memory required to store the data; and storing the compressed data in memory.The method of claim 1, comprising: causing the processing device to segment the at least a portion of the data stream into at least one n X n segmented data array.The method of claim 2, including causing the processing device to transform the at least one segmented data array into at least one interleaved data array of size (2n) X (n / 2).The method of any of claims 1 to 3, wherein the processing device is caused to reverse the transposed data array.The method of any one of claims 1 to 4, wherein the stored compressed data is transferred to a computer having a processor configured to decompress the compressed data.The method of claim 5, characterized in that the processor of the computer is configured to decompress the compressed data by the steps of: inverse run length coding the compressed data to form at least one interleaved data array; converting the at least one interleaved data array into at least one transposed data array; transpose the at least one transposed data array to form at least one segmented data array.The method of claim 6, characterized in that the processor of the computer is configured to collect a number of segmented data arrays to form a data stream substantially identical to the original stream of digital data.The method of any one of claims 1 to 7, characterized byusing a data acquisition system selected from the group consisting of glow discharge atomic emission spectrometers, thermal gravimetric analyzers, ash melt measurement devices, calorimeters, elemental analyzers, digital microscopes, mass spectrometers, and audio devices to provide the original stream of digital data to the processing device.Method according to any one of claims 1 to 8, characterised in that the processing device comprises at least one field programmable gate array and a graphic processing unit.The method of any one of claims 1 to 9, wherein the method comprises: providing a PC; and transferring the compressed data from the data acquisition system to the PC.The method of any of claims 1 to 10, wherein the method comprises causing the processing device to determine whether a first data location is zero, and transpose the at least one segmented data array only if the first data location is zero.The method of claim 11, wherein the method comprises causing the processing device to generate a stream of data comprising uncompressed data and compressed data stored in memory.The method of any of claims 1 to 12, characterized in that the original stream of digital data has N numbers comprising measured data taken at discrete time intervals, each number comprising a magnitude of a measured variable.A method according to any one of claims 1 to 13, characterized by causing the processing device to 1) determine a difference between two data sets at a time and location; 2) provide a sign of a difference using a sign parameter; 3) determine an absolute value of the difference; 4) compress the difference; and 5) store the compressed difference and the sign parameter.An analyzer comprising: a data acquisition system configured to generate a stream of digital data comprising N numbers of size n corresponding to a size of a measured variable in which at least some of the numbers include at least one zero, the system including: a processing device configured to receive a data stream from the data acquisition system, the processing device configured to cause the processing device to segment at least a portion of the data stream to form at least one segmented data array; causing the processing device to transpose the at least one segmented data array to form at least one transposed data array; Causing the processing device to share the at least one transposed data array to form at least one interleaved data array, causing the processing device to run length encode leading zeros of the numbers to form compressed data, reducing the memory requirement required for storing the data, and storing the compressed data in memory.The analyzer of claim 15, characterized in that the processing device is configured to reverse the transposed data array.An analyser according to any one of claims 15 and 16, characterised in that the processing device is configured to segment the at least a portion of the stream of data into at least one segmented array of data of size n X n.Analysis device according to claim 17, characterised in that the processing device is configured such that it transforms the at least one segmented data array into a (2n) X (n / 2) interleaved data array.The analyzer according to any one of claims 15 to 18, characterized in that the analyzer includes at least one storage unit including a hard disk and / or a semiconductor storage disk, and wherein the processing device is configured such that compressed data is stored on the storage unit.The analyzer according to any one of claims 15 to 19, characterized in that the analyzer is selected from the group comprising glow discharge atomic emission spectrometers, thermal gravimetric analyzers, ash melt measurement devices, calorimeters, elemental analyzers, digital microscopes, mass spectrometers and audio devices.

Images