Maximum error bound lossy compression method for time-series

Abstract

In sum, the disclosed invention constitutes a method and a system for the lossy compression of time series consisting of real numbers, depicted using the IEEE Standard for Floating-Point Arithmetic (IEEE 754). It provides lossy compression of time series using a user-defined maximum tolerable loss per value, aiming to enhance storage and transmission efficiency. This way, it allows the storage of real numbers' time series using the least memory / storage space possible in a computer system, as well as the transmission of real numbers' time series with minimum bandwidth requirements. It allows for case-by-case determination of the acceptable loss so that the outcome reflects the distinct requirements of each case and exploits the similarity of consecutive data points to ensure optimal resources management. The technical result of the invention is the significant economization of storage space and bandwidth as the present invention readily outperforms state-of-the-art methods, leading to more efficient applications.

Description

[0001] DESCRIPTION

[0002] MAXIMUM ERROR BOUND LOSSY COMPRESSION METHOD FOR TIME-SERIES

[0003] TECHNICAL FIELD

[0004] The present invention falls within the broader field of data compression and, more specifically, it pertains to the lossy compression of time series. In general, the invention is related to a wide range of scientific fields in the broader Information Technology (IT) landscape, Data Science and Analytics. Particularly as a lossy data compression solution, it is closely connected to time series analysis (mainly, real numbers' time series analysis), computer science and software engineering, database technology (in particular, real numbers' time series databases), Internet of Things (loT), and computer-implemented algorithms.

[0005] TECHNICAL PROBLEM

[0006] Contemporary technology has led to an ever-growing demand for efficient storage management. The present invention solves the technical problem of data compression and the resulting resources economy, both in terms of storage space as well as in terms of bandwidth during data transmission. It provides a response to the requirements for data storage rationalization, by minimizing storage footprints, and optimization of memory / storage place allocation. More specifically, modern applications generate vast streams of data, often depicted as sequences of timestamped records. These sequences are called time series. Such applications face significant challenges in terms of storing the generated data, stemming from the urgently emerging need for the effective use of scarce resources, such as computer memory and bandwidth. Thus, applications increasingly rely upon the effectiveness of compression techniques to minimize storage requirements. Depending on the specific requirements of each application, one can opt for either lossless or lossy compression methods. In a nutshell, lossless techniques reduce data size without loss of information, while lossy methods achieve much more substantial space savings while tolerating a predefined maximum error bound.

[0007] Lossless compression methods are more suitable for applications in which data precision is crucial, such as the monitoring of critical infrastructure, e.g., nuclear power plants or rocket launchers. These methods guarantee that the compressed representation will not cause loss of data. However, the compression ratio that lossless methods yield is limited. On the other hand, employing lossy methods enables substantial space savings; however, at the expense of precision in the compressed data. They are divided into those that enforce quantifiable error bounds to the compressed data and those that do not. In the former case, the space-savings grow with the specified maximum error that can be tolerated. These approaches are preferred in applications in which the precision is not crucial, for example in compressing multimedia data (e.g., audio, images or video), and especially when fast transmission is also needed (e.g., in streaming media).

[0008] The present invention belongs to the class of maximum error-bounded lossy compression. Through innovative techniques that produce large identical sequences of bits in the binary representations of consecutive data points, as well as methods that utilize the similarity of a data point with its immediate previous one, the invention achieves impressive space savings.

[0009] STATE OF THE ART

[0010] Several attempts have been made to solve the technical problem presented above:

[0011] Lossless methods with XOR

[0012] Document CN116192153 discloses a method that involves sequencing time data points to be compressed, performing XOR calculation on consecutive data points and exploiting the leading and trailing zero bits of the resulting values to come up with a lossless compressed representation.

[0013] Document CN116208168 discloses a method that performs XOR on the value to be compressed and a preset value to generate index and default bytes that represent the original value in a lossless compressed format.

[0014] Document CN116089385 discloses a lossless compression and decompression method for high-precision floating point type time series data, which comprises the following steps of: sorting the data, performing XOR on consecutive data points to obtain a corresponding XOR value, removing leading zeros in the XOR values of the time sequence data to generate corresponding effective bits, and determining a corresponding effective bit number according to the effective bit of each piece of time sequence data.

[0015] Reference 1 discloses FPC, which is a lossless compression algorithm that applies a bitwise XOR between the next value to be compressed and a predicted value it generates by employing two predictors. In the context of time series datasets, the resulting values usually exhibit long runs of leading and trailing zeros. FPC uses the best value of the two predictors, in terms of the leading zero count of the resulting XORed value. The compressed representation uses one bit to specify the predictor and exploits the presence of multiple leading zeros in the resulting XORed value to reduce the bits that need to be stored.

[0016] Reference 2 discloses Gorilla, which is based on FPC and improves its efficiency by discarding its expensive prediction schemes. More specifically, Gorilla simply compares the value to be compressed with the immediate previous one, by performing a bitwise XOR operation. If the resulting XORed value is zero, only one flag bit is used to represent the result. Otherwise, Gorilla stores two flag bits, that signify whether a previous state can be used, and the zero-trimmed center bits of the resulting XORed value. Depending on the previous state, the output may also include information about a new state that needs to be specified. Reference 3 discloses Chimp, which is based on Gorilla and introduces two novel techniques to provide improved space efficiency. First, it uses four encoding formats instead of three, that better capture and exploit the properties of real- world floating point datasets. Second, it compares the value to be compressed with the best of the last 128 values, by performing a bitwise XOR operation. Through the use of a ring buffer, only one XOR operation is actually performed. The compressed representation includes a seven-bit index that specifies the value used in the comparison.

[0017] Lossy methods

[0018] Reference 4 discloses BUFF that uses a byte-oriented columnar storage representation for compressing bounded, low-precision floating values. BUFF truncates floating point values by zeroing some of its least significant mantissa bits. Additionally, BUFF adjusts the number of bits used to specify the exponent of a floating-point value according to the range of values of the particular dataset to be compressed.

[0019] Reference 5 discloses Elf which is very similar to BUFF as it also truncates floating point values by zeroing some of its least significant mantissa bits. Using an exhaustive search Elf is able to find how many bits can be zeroed without loss, for bounded precision values.

[0020] Reference 6 discloses PMC-MR, an optimal algorithm for approximating sensor data using Piecewise Constant Approximation. A cache filter is used, which predicts that the next data point will have a value within the error threshold from the previous one. Thus, a new data point and a respective timestamp is recorded only when it violates the error constraint. The values in between are approximated using the last recorded value; however, the algorithm does not store the number of data points between recorded values, providing, instead, a continuous signal. Reference 7 discloses Swing and Slide, which are a joint- and a disjoint-segment PLA algorithm, respectively. Swing constructs the longest possible line segment starting from a fixed origin point by adjusting two bounding slope lines until reaching a break-up point, which will generate a new joint-segment, starting from the last point of the previous segment. Slide filters are different as they generate disjoint segments as an approximation for the original data points. This gives them more flexibility at the expense of having to store an additional value for each segment. Updating the bounding slope lines can be optimized with the use of the convex hull of the observed data points, which allows for checking only against the points of the convex hull, instead of the significantly larger number of all the data points observed in the current filtering interval.

[0021] Reference 8 discloses Sim-Piece, a compression approach that employs PLA to approximate a time series and then groups the generated line segments to further reduce the storage requirements.

[0022] The present invention features an XOR comparison of the value to be compressed with a previous value of the dataset or a predicted value to come up with a resulting value that is more compressible, as is also the case with FPC, Gorilla and Chimp lossless compression algorithms. However, in the context of the present invention the value to be compressed is adjusted prior to its compression so that the space-savings are increased.

[0023] The present invention is similar to BUFF and Elf in terms of providing lossy or bounded precision lossless compression, as well as manipulating the least significant mantissa bits of floating point values. However, the present invention does not perform a zeroing of these bits as BUFF and Elf do. Instead, it substitutes these bits with the respective bits of the previously stored values to achieve increased space-savings.

[0024] Similarly to PMC-MR, Swing, Slide and Sim-Piece, the present invention also provides maximum error-bounded lossy compression. However, the present invention operates strictly on values, instead of pairs of data points and timestamps, and its representation allows for restoring the exact number of values originally compressed.

[0025] BRIEF DISCLOSURE OF THE INVENTION

[0026] Briefly, the present invention is a method and a system for the lossy compression of time series, depicted using the IEEE Standard for Floating-Point Arithmetic (IEEE 754), which is a technical standard for floating-point computation and is currently the most common representation for real numbers on computers. It focuses on lossy compression of time series consisting of real numbers with a user-defined maximum tolerable loss per value, aiming to enhance storage and transmission efficiency. Applying computer-implemented algorithms, the invention allows the storage of real numbers' time series using the least memory / storage space possible in a computer system, as well as the transmission of real numbers' time series with minimum bandwidth requirements. The maximum acceptable loss per value allowed during compression is defined by the user or application depending on the accuracy required in each case, ensuring that the outcome reflects the distinct requirements of each particular case.

[0027] The invention exploits the similarity of consecutive data points often observed in real-world time series datasets and uses a single bit to record a data point if its value is within the user-defined error threshold from the previous one. Otherwise, the invention substitutes the less significant bits of the current value with those of the previous value. The number of bits that are substituted are specified through the user-defined error threshold, so that the resulting value is within the acceptable error bound. Then, a bitwise XOR operation is applied between the new value and the previous one. Depending on the number of leading and trailing zeros in the resulting value, the latter is stored using the smallest of two possible output formats. The technical result of the invention is the significantly smaller size (number of bits) of the time series processed by the algorithm according to the present invention, compared to its original representation based on the IEEE 754 standard representation in the computer system. Typical applications of the invention are time series management and processing applications, for example in Internet of Things applications or in time series storage systems (time series databases). In these applications the use of the present invention results in memory savings and significantly lower requirements, with up to +820% relative improvement over the state-of-the-art method as shown in the embodiments of the invention, for storing or transmitting data over a network when required. The relative improvement has been calculated as (Compression Ratio according to the present invention - Compression Ratio according to the prior art) / Compression Ratio according to the prior art. These savings in resources in a computing system lead to faster and more efficient applications.

[0028] The decompression of the time series follows an inverse logic, which is depicted in Figure 3.

[0029] LIST OF FIGURES

[0030] Figure 1 (la and lb) illustrates a general view of the system architecture for performing the method according to the present invention. Figure la illustrates the system architecture within the local paradigm, while Figure lb depicts the architecture within the distributed paradigm, in particular the cloud-edge continuum.

[0031] Figure 2 presents a detailed view of the functioning of the method according to the present invention.

[0032] Figure 3 depicts the decompression process. DETAILED DISCLOSURE OF THE INVENTION WITH REFERENCE TO THE FIGURES

[0033] The invention is set out in the claims.

[0034] The invention is described below by means of non-limiting examples and with reference to the attached drawings.

[0035] Figures la and lb present a general view of the system's architecture, where the individual subsystems are depicted. However, the invention can be implemented in any possible computing system architecture within the local or distributed system paradigms, for instance peer-to-peer, client-server, grid computing and the like. Figure la concentrates on the compression functionality itself, irrespective of the origin of the data. Figure lb depicts different possible cloudbased implementations. The data origins as well as the data paths are indicative and not limiting. Furthermore, Figure lb should not be interpreted as implying that the compression process should exclusively take place at the edge / locally. Even if the compression functionality is depicted associated within different loT contexts (for instance, residential or industrial), it is readily conceivable that the compression computations can take place anywhere in the cloud.

[0036] Figure 2 presents individual steps according to the inventive method:

[0037] The invention receives as input a sequence of single- or double-precision IEEE 754 floating point values, as well as the maximum acceptable error e. The compression of the sequence is achieved as described below with reference to Figure 2.

[0038] The first step is the initialization of the maximum acceptable error e (Block 1).

[0039] Then, the invention attempts to read the next data point of the input sequence of floating-point values (Block 2).

[0040] If there exists a next point, then the invention checks whether no previous value has been saved before and this point is the first value to be processed (Block 3). If this is the case two simple steps follow. First, the invention initializes the state of the previous value maintained, using the current value (Block 4). Second, the procedure writes the value as is to the result, i.e., no compression is applied on the first value (Block 5). The invention then returns to Block 2. If a previous value has been processed, then we check whether the current value is within the user- defined error threshold from this previous value (Block 6). In this case, a single step follows. In particular, the invention writes to the result, as a compressed code description, a single zero bit, that is enough to represent the current value, as the latter is within the error threshold from the previous value stored (Block 7). The invention then returns to Block 2. In this way, instead of 4 or 8 bytes (i.e., 32 or 64 bits) required to represent real numbers with the IEEE 754 standard, the value now occupies only 1 bit of space. Otherwise, i.e., if the current value is not within the user-defined error threshold from the previous value, the following steps are executed. First, the invention substitutes as many less significant bits from the current value as possible, using the corresponding bits of the previous value (Block 8). To do this, the procedure calculates the position in the 32-bit (single precision) or 64-bit (double-precision) IEEE 754 value that corresponds to the first decimal digit of the value. For single precision values this is equal to: decimal_point_position = 23 - (value « 1 »> 24) - 127, where « denotes a signed left shift operation and »> denotes an unsigned right shift operation.

[0041] Using this result, the invention comes up with the position of the most significant bit that can be safely substituted while keeping the value within the user-defined error threshold as follows: substitution_position = decimal_point_position + [log2(e)J

[0042] The substitution is performed for all the bits from this position to the least significant bit of the current value, using the corresponding bits of the previous value as follows: value = value » substitution_position « substitution_position, where » denotes a signed right shift operation and « denotes a signed left shift operation. value = value | (previous_value & (substitution_position2- 1)), where | denotes a bitwise OR operation and & denotes a bitwise AND operation. These operations guarantee that the resulting value is within the user-defined error threshold, and has as many identical less significant bits with the previous value as this threshold allows for.

[0043] The next step is to apply an XOR operation between this resulting value and the previous value, which is very likely to produce a value with many leading and trailing zeros, due to the similarity of consecutive values and the substitution operation applied, respectively (Block 9).

[0044] Next, the previous value is updated (Block 10).

[0045] Then, the invention examines whether the number of leading zeros in the resulting value to be compressed is larger or equal to the previous value of leading zeros, and the number of trailing zeros in the resulting value to be compressed is larger or equal to the previous value of trailing zeros. (Block 11)

[0046] If both these criteria are true, then the numbers of leading and trailing zeros is not necessary as the previous numbers can be used and the procedure simply writes: a) two flag bits '10', and b) the center part of the resulting value, i.e., the bits that remain after removing the leading and trailing bits specified by the previous number of leading and trailing zeros, respectively (Block 12). The invention then returns to Block 2.

[0047] The second case is that either of the number leading or trailing zeros of the resulting value are less than the number of those previously stored. Moreover, the second case is also applied when no previous value has already been stored. First, the previous values of leading and trailing zeros are updated. (Block 13).

[0048] Then, the actions taken in this second case are the following (Block 14): a) two flag bits 'll' are written, b) the number of leading zeros is written using 5 bits, c) the number of center bits of the resulting value, i.e., the bits that remain after removing the leading and trailing zeros, is written using 6 bits, and d) the center bits are written.

[0049] The invention then returns to Block 2. In both cases, the sequences of zeros that have emerged are represented in the compressed form using fewer bits. Therefore the present invention produces a representation that is considerably smaller than the original one.

[0050] An example of the entire process is given below:

[0051] Assuming that we need to compress the sequence of values <25.2, 25.3, 25.6> using a maximum error threshold e=0.2.

[0052] First, value 25.2 is encoded. As the previous value has not been set, the 'compare' procedure will lead to the 'encode_first_value' procedure, which will store the value without applying compression and will initialize the previous value to be equal to 25.2.

[0053] Then, value 25.3 is encoded. As 25.3 is within 0.2 from the previous value stored, i.e., 25.2, the 'compare' procedure will lead to the 'encode_same_value' procedure, which will store a single '0' bit to represent the value as equal to the previous one.

[0054] After that, value 25.6 is encoded. As 25.6 is not within 0.2 from the previous value stored, i.e., 25.2, the 'compare' procedure will lead to the 'encode_using_xor' procedure, which will substitute as many less significant bits of the binary representation of 25.6 as possible with the corresponding bits of the binary representation of 25.2, while remaining within 0.2 from the original 25.6 value.

[0055] Using the formula the resulting substitution position is: substitution_position = 23 - ((01000001 11001100 11001100 110011012« 1 »> 24) - 127) + [log2(0.2)J = 23 - (131 - 127) + (-3) = 16

[0056] Thus, the procedure will substitute the bits below this position with the corresponding bits from the previous value:

[0057] 25.6 => 01000001 11001100 11001100 11001101

[0058] 25.2 => 01000001 11001001 10011001 10011010

[0059] Resulting value => 01000001 11001100 10011001 10011010

[0060] Then, the procedure applies XOR between this resulting value and the previous value: 01000001 11001001 10011001 10011010 XOR

[0061] 01000001 11001100 10011001 10011010 =

[0062] 0000000000000101 00000000 00000000

[0063] The final value has 13 leading zero bits and 16 zero trailing bits. Thus, there are only 3 center bits that need to be explicitly stored. As there is no previous value stored using this procedure the number of leading zeros and center bits should also be explicitly stored. Overall, the procedure writes two flag bits 'll', then 5 bits for the number of leading zeros '01101', 6 bits representing the number of center bits '000011', and the 3 center bits '101', i.e., a total of 16 bits instead of 32. Finally, the procedure updates the number of previous leading and trailing zeros to be equal with 13 and 16, respectively. These values will help store less bits in future uses of this procedure.

[0064] Figure 3 depicts the logic followed for the decompression of the compressed time series. The decompression process is a straightforward implementation following the compression process.

[0065] The present invention achieves +650% relative improvement compared to the previous state-of-the-art for datasets with earth observational metrics, and up to +820% for datasets with stock prices, in terms of compression ratio.

[0066] EMBODIMENTS OF THE INVENTION

[0067] Non-limiting examples of embodiments of the invention are described below. More specifically, the following table features a comparison of the compression ratio (CR) achieved by the invention, against the CR achieved with the method of Reference 5, for different values of e and for three different datasets about measurable physical parameters, as well as a financial dataset. Air Pressure features barometric pressure measurements corrected to sea level and surface level. Wind direction features two-dimensional wind direction measurements. The third dataset comprises infrared biological temperature, i.e., surface temperature measurements. Finally, the last dataset features prices of USA stocks.

[0068] Those skilled in the art will appreciate that the inventive concept can be readily implemented irrespective of the specific microprocessor architecture employed. For instance, the compression algorithm runs on any current 32-bit or 64-bit microprocessor without altering any of the features of the inventive method as claimed.

[0069] The embodiments described above are provided for illustrative purposes only and should not be construed as limiting the invention. Those skilled in the art will readily recognize various modifications and changes that can be made to the present invention that fall within the scope of the disclosure and application of the present invention. REFERENCES

[0070] 1. Martin Burtscher and Paruj Ratanaworabhan. 2007. High Throughput Compression of Double-Precision Floating-Point Data. In 2007 Data Compression Conference (DCC 2007), 27-29 March 2007, Snowbird, UT, USA. IEEE Computer Society, 293-302. https: / / doi.org / 10.1109 / DCC.2007.44

[0071] 2. Tuomas Pelkonen, Scott Franklin, Paul Cavallaro, Qi Huang, Justin Meza, Justin Teller, and Kaushik Veeraraghavan. 2015. Gorilla: A Fast, Scalable, In-Memory Time Series Database. Proc. VLDB Endow. 8, 12 (2015), 1816-1827. https: / / doi.org / 10.14778 / 2824032.2824078

[0072] 3. Panagiotis Liakos, Katia Papakonstantinopoulou, Yannis Kotidis: Chimp: Efficient Lossless Floating Point Compression for Time Series Databases. Proc. VLDB Endow. 15(11): 3058-3070 (2022)

[0073] 4. Chunwei Liu, Hao Jiang, John Paparrizos, Aaron J. Elmore: Decomposed Bounded Floats for Fast Compression and Queries. Proc. VLDB Endow. 14(11): 2586-2598 (2021)

[0074] 5. Ruiyuan Li, Zheng Li, Yi Wu, Chao Chen, Yu Zheng: Elf: Erasing-based Lossless Floating-Point Compression. Proc. VLDB Endow. 16(7): 1763-1776 (2023)

[0075] 6. Lazaridis, I., Mehrotra, S.: Capturing sensor-generated time series with quality guarantees. In: Proc, of the 19th Int. Conf, on Data Engineering, March 5-8, 2003, Bangalore, India, pp. 429-440. IEEE Computer Society (2003). DOI 10.1109 / ICDE.2003.1260811

[0076] 7. Elmeleegy, H., Elmagarmid, A.K., Cecchet, E., Aref, W.G., Zwaenepoel, W.:

[0077] Online piecewise linear approximation of numerical streams with precision guarantees. Proc. VLDB Endow. 2(1), 145-156 (2009). DOI

[0078] 10.14778 / 1687627.1687645. http: / / www.yldb.org / pyldb / vol2 / yldb09-

[0079] 573.pd

[0080] 8. Xenophon Kitsios, Panagiotis Liakos, Katia Papakonstantinopoulou, Yannis Kotidis: Sim-Piece: Highly Accurate Piecewise Linear Approximation through Similar Segment Merging. Proc. VLDB Endow. 16(8): 1910-1922 (2023)

Claims

AMENDED CLAIMS received by the International Bureau on 02 June 2025 (02.06.2025)Claim 1A computer-implemented resource management method in which a time series of real numbers is converted to a compressed format, the method comprising the determination of a user-defined maximum acceptable error (e) (Block 1) storing the first number of the series (Blocks 3, 4, 5) if the difference between the current serial number and the previously stored serial number is less than or equal to the maximum acceptable error (Block 6), compress the current number by storing a single bit ('0') referring to the previously stored number (Block 7); if the difference between the current serial number and the previously stored serial number is greater than the maximum acceptable error (Block 6), then the current number is processed by determining the position of the most significant bit that can be replaced while keeping the value of the current number within the error limit, and replacing this and the other less significant bits of the current number with the corresponding bits of the previous number (Block 8) applying XOR between the resulting value and the previously stored value (Section 9) storing the XOR result as a lossy compressed representation of the current number using a number of flag bits, followed by a subset of the XOR operation bits (Block 10).Claim 2The method of Claim 1, where the compressed lossy representation uses two different storage formats, depending on the number of zeros that result at the beginning and end of the number (Block 11).AMENDED SHEET (ARTICLE 19)Claim 3The method of Claim 2, where one of the two different formats includes the flag bits "10" followed by the central non-zero bits of the result of the XOR operation (Block 12) and the other of the two different formats includes the flag bits "11", followed by 5 bits representing the number of leading zeros, 6 bits representing the number of centre non-zero bits of the result of the XOR operation and the centre non-zero bits of the result of the XOR operation (Block 14).Claim 4The method of any of Claims 1 to 3, further comprising decompressing the stored values of that compressed time series (Figure 3).Claim 5The method of any of Claims 1 to 4, where the compressed format is used to store and / or retrieve the time series, transmit the time series, or any combination thereof.Claim 6The method of any of Claim 5, where the compressed / decompressed formats are used to perform calculations.Claim 7The method of Claim 6, where said calculations involve training a machine learning system.Claim 8AMENDED SHEET (ARTICLE 19)A computer system comprising means operative to perform the method according to any of Claims 1 to 7.Claim 9A computer-readable medium storing a program which, when loaded and run on a computer system, performs the method of any of Claims 1 to 7.AMENDED SHEET (ARTICLE 19)

Images