Streaming error bounded floating point type data compression method

Abstract

The invention discloses a streaming error bounded floating point type data compression method. The method comprises the following steps: acquiring an original time sequence of a floating point type to be compressed; the quantization-based streaming error bounded floating point compression method comprises the following steps of: quantizing a floating point value in an original time sequence into an integer quantized value; encoding the quantized value to generate a compressed bit stream as a compression time sequence; the invention discloses a streaming error bounded floating point compression method based on XOR operation. The method comprises the following steps: displacing a floating point value in an original time sequence to obtain a displacement value; converting the displacement value into an approximate value; performing exclusive-or operation on each approximate value and the previous approximate value to obtain an exclusive-or value; and encoding the XOR value to generate a compressed bit stream as a compression time sequence. According to the method, the streaming error bounded floating point compression method based on quantization and XOR operation is designed to realize the streaming floating point compression with the error bounded, and the compression requirements of streaming time series data of the Internet of Things and the like can be better met.

Description

Technical Field

[0001] The invention relates to the technical field of streaming floating point data compression, and in particular to a streaming error-bounded floating point type data compression method. Background Art

[0002] In recent years, the Internet of Things (IoT) technology has made great progress. In a typical IoT scenario, a large amount of floating-point time series data (referred to as time series) is continuously generated by various sensors in a streaming manner and then transmitted to the server under limited bandwidth for further real-time analysis and processing. Figure 1 As shown in the figure, it has the following characteristics: 1) Limited bandwidth. The number of time series is huge, but the network bandwidth between sensors and servers is usually limited. For example, the widely used wireless network communication protocol Zigbee has a defined rate of only 250kbit / s. 2) Fault tolerance. On the one hand, there are certain measurement errors in the sensor itself; on the other hand, some applications do not actually require accurate readings from the sensor. For example, in daily life, we only care about temperature readings with up to two decimal places. Therefore, the compression processing of such data allows distortion within the specified error range. 3) High real-time requirements. Many critical applications rely on real-time data analysis, which means that each record should be sent to the server immediately once it is generated. For example, through the school bus tracking and monitoring system, parents, schools, and regulators can track the conditions and safety of the school bus in real time.

[0003] In view of the above characteristics, in order to improve efficiency, it is generally recognized that it is necessary to compress data before transmission.

[0004] Existing time series floating point data compression algorithms can be mainly divided into three categories: batch lossless compression, batch lossy compression and streaming lossless compression. Batch lossless compression algorithms can compress time series in batches without losing any information; batch lossy compression algorithms can usually achieve better compression ratios in batches at the expense of some data accuracy; streaming lossless compression algorithms can compress time series in streaming without losing any information.

[0005] However, the applicant found that the existing floating-point compression methods are either batch compression, which may cause long delays, or streaming lossless compression, and the compression rate is not ideal in application scenarios where certain errors are allowed. Specifically, the batch compression algorithm needs to wait for all data records in a batch (for example, 1,000 data records) to be ready before the compression operation can be actually performed. If the data sampling rate is not high, it will cause a large delay and fail to meet the real-time requirements. For example, GPS (Global Positioning System) sensors usually report their location every 1 to 10 seconds. If the batch size is set to 1,000, the delay will be as high as 1.7 to 16.7 minutes, which is not suitable for scenarios such as real-time bus tracking and monitoring. To reduce latency, one solution is to use small batch technology, such as setting the batch size to a value less than 50. However, since it is difficult to find shared information between data records in this case, the compression performance of most batch compression algorithms will not be good in small batches.

[0006] In terms of streaming compression, all existing methods are lossless compression, which can adapt to high-precision application scenarios such as scientific computing, but compared with lossy compression algorithms with bounded errors, these algorithms do not perform well in terms of compression ratio. Combined with the actual application scenarios and requirements in the field of the Internet of Things (with certain fault tolerance and high real-time requirements), the streaming lossy compression method with bounded errors can theoretically better meet the requirements. Therefore, how to design a streaming lossy compression method with bounded errors is a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the technical problem to be solved by the present invention is: how to provide a streaming error-bounded floating-point type data compression method, by designing a streaming error-bounded floating-point compression method based on quantization and XOR operation to achieve error-bounded streaming floating-point compression, compared with batch compression algorithms and streaming lossless compression algorithms, it can better meet the compression requirements of streaming time series data such as the Internet of Things.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] A streaming error-bounded floating-point data compression method, comprising:

[0010] S1: Get the original time series of floating point type to be compressed;

[0011] S2: Select the corresponding streaming error-bounded floating-point compression method according to the set error-bounded type: if the streaming error-bounded floating-point compression method based on quantization is selected, execute step S3; if the streaming error-bounded floating-point compression method based on XOR operation is selected, execute step S4;

[0012] S3: quantize the floating point values ​​in the original time series into integer quantized values ​​to generate a quantized value sequence; encode the quantized values ​​in the quantized value sequence to generate a compressed bit stream as a compressed time series;

[0013] S4: Shift the floating-point values ​​in the original time series to obtain shift values, and generate a shift value sequence; convert the shift values ​​in the shift value sequence into approximate values, and obtain an approximate value sequence; perform an XOR operation on each approximate value in the approximate value sequence and its previous approximate value to obtain an XOR value, and generate an XOR value sequence; encode the XOR values ​​in the XOR value sequence, and generate a compressed bit stream as a compressed time series.

[0014] Preferably, in step S2, the error bound type includes an absolute error bound and a relative error bound;

[0015] When the error bound type is an absolute error bound, a quantization-based or XOR-based streaming error bounded floating point compression method is selected;

[0016] When the error bound type is relative error bound, a streaming error bounded floating point compression method based on XOR operation is selected.

[0017] Preferably, in step S3, the original time series TS=<v1,v2,...> The floating point value v in i Quantized to integer quantization value q i , generate a sequence of quantized values<q1,q2,...> ;

[0018] The specific steps are as follows:

[0019] S301: For floating point value v i Make a prediction and generate a predicted value p i ;

[0020] S302: According to the floating point value v i The predicted value p i Combined with the absolute error bound ∈ a Calculate the quantized value q i ;

[0021] The formula is:

[0022]

[0023] Where: [*] indicates the operation of rounding the floating point value to an integer.

[0024] Preferably, in step S3, the quantized value sequence is encoded by Zigzag coding.<q1,q2,...> The quantized value q in iConvert to a positive integer, and then encode the result of Zigzag encoding through Elias Gamma encoding to get the encoding value code i , generating a compressed bitstream<code1,code2,...> ;

[0025] The formula is:

[0026] code i = Elias GammaEn(ZigzagEn(q i )+1);

[0027] Wherein: ZigzagEn represents Zigzag encoding; Elias GammaEn represents Elias Gamma encoding.

[0028] Preferably, in step S3, the bit stream is decoded by Elias Gamma decoding.<code1,code2,...> The coded value in is decoded, and then Zigzag decoding is used to decode the result of Elias Gamma decoding to obtain the quantized value q i , decoding generates a sequence of quantized values<q1,q2,...> ;

[0029] The formula is:

[0030] q i =ZigzagDe(EliasGammaDe(code i )-1);

[0031] Where: Elias GammaDe represents Elias Gamma decoding; ZigzagDe represents Zigzag decoding;

[0032] The dequantizer of the Serf_Qt decompressor performs a sequence of quantized values<q1,q2,...> The quantized value q in i Dequantize to recover the floating point value v′ i , generate decompressed time series TS′=<v1′,v2′,...> ;

[0033] The formula is:

[0034] v′ i =2×∈ a ×q i +p i ;

[0035] Where: ∈ a represents the absolute error limit; p i Represents a floating point value v i The predicted value of .

[0036] Preferably, in step S4, the original time series TS=<v1,v2,...> The floating point value v in i Add the integer offset λ to get the displacement value s i =v i +λ, generate a sequence of displacement values<s1,s2,...> ;

[0037] in

[0038] Where: max and min represent the original time series TS=<v1,v2,...> The maximum and minimum values ​​in; u is a positive integer; Indicates rounding up; Indicates rounding down.

[0039] Preferably, in step S4, the displacement value sequence is<s1,s2,...> Median displacement value s i Find the approximate value a i ∈[s i -∈,s i +∈], so that a i and a i-1 Share the most suffix bits to generate a sequence of approximate values<a1,a2,...> ;

[0040] Where: ∈ = ∈ a or ∈=∈ r ×|v i |∈ a represents the absolute error limit; ∈ r Represents the relative error limit.

[0041] Preferably, in step S4, the displacement value s is obtained by the following steps: i Find the approximate value a i :

[0042] S401: Obtain error limit ∈, previous displacement value s i-1 and i-1 The corresponding approximate value a i-1 ;

[0043] S402: Calculate low=s respectively i -∈,up=s i +∈,

[0044] S403: Constructing a splicing combination Among them, pre bn-j Indicates that bn-j prefix bits are selected from low, suf j Indicates from a i-1Select j suffix bits from , bn represents the number of floating point values ​​in the original time series; Represents a splicing operation;

[0045] S404: Select j in order from bn-1 to 0 until c1 or c2∈[s i -∈,s i +∈];

[0046] S405: Use c1 or c2 that meets the condition as the displacement value s i The approximate value of a i .

[0047] Preferably, in step S4, the approximate value sequence<a1,a2,...> Each approximation a in i Its previous approximation a i-1 Perform XOR operation to get XOR value xor i , generates a sequence of XOR values<xor1,xor2,...> ;

[0048] The SElf* encoding strategy is combined with the adaptive label allocation method to encode the XOR value sequence.<xor1,xor2,...> XOR value in i Encode to get the encoding value code i , generating a compressed bitstream<code1,code2,...> ;

[0049] The encoding process steps are as follows:

[0050] S411: Obtain an existing XOR value sequence of an existing time series of floating point type;

[0051] S412: Count the leading zero count and trailing zero count of the XOR value in the existing XOR value sequence to define the approximate rule of leading zero and trailing zero lead and rule trail ;

[0052] S413: Adaptive logo adjustment: through SElf * The encoding strategy encodes the existing XOR value sequence and counts the frequency of each case in the encoding result; the flag bit corresponding to the case with the highest frequency is set to 1, and the flag bits corresponding to the second and third highest frequency cases are set to 00 or 01 respectively;

[0053] S414: For XOR value sequence<xor1,xor2,...> Each XOR value in xor i , perform the following steps:

[0054] S4141: Determine the XOR value xor iLeading zero count, trailing zero count, and center bit;

[0055] S4142: According to rule lead and rule trail Obtain approximate leading zero count, trailing zero count, and center bit;

[0056] S4143: Calculate and

[0057] S4144: Define condition C as: lead i ≥lead i-1 、trail i ≥trail i-1 and (lead i - lead i-1 )+(trail i - trail i-1 ) < 1 + ln + tn; where lead i 、trail i are respectively the approximate leading zero count and trailing zero count of xor i ;

[0058] S4145: Determine whether the first case is satisfied: xor i = 0: If so, code i is encoded as the flag bit corresponding to the first case; otherwise, execute step S4126;

[0059] S4146: Determine whether the second case is satisfied: xor i ≠ 0 and condition C holds: If so, code i is encoded as: the flag bit corresponding to the second case followed by the approximate center bit; if not, then determine that the third case is satisfied, and code i is encoded as: the flag bit corresponding to the third case followed by ln bits of lead i 、tn bits of trail i and the approximate center bit.

[0060] Preferably, in step S4, the encoded values in the compressed bit stream <code1, code2,...> are decoded to obtain the xor value xor i , and a sequence of xor values <xor1, xor2,...> is decoded; the xor values xor i in the sequence of xor values <xor1, xor2,...> are xor - operated to obtain the approximate value a i , and a sequence of approximate values <a1, a2,...> is decoded; each approximate value a iConvert to restore floating point value v′ i , generate decompressed time series TS′=<v1′,v2′,...> .

[0061] Compared with the prior art, the streaming error-bounded floating-point data compression method of the present invention has the following beneficial effects:

[0062] The present invention can realize streaming floating point compression through a streaming error bounded floating point compression method (Serf-Qt) based on quantization. First, Serf-Qt maps continuous floating point values ​​to integer sets through a quantization process, and ensures that data distortion in the compression process is strictly controlled within a preset error bound; at the same time, by adjusting the error bound, the best balance point can be found between the compression rate and the error, ensuring that the compressed data is within an acceptable error range. Secondly, compared with the original floating point values, the quantized integer sequence is more compact in representation, because the integer can be directly represented by binary bits, and there is no need to consider the exponent and mantissa of the floating point number. This representation method significantly reduces the demand for data storage space and improves compression efficiency. Finally, the streaming time series data generated by devices such as the Internet of Things has the characteristics of continuity and strong time correlation. Serf-Qt can effectively utilize these characteristics, and through reasonable quantization strategies and encoding methods, it can realize rapid compression and decompression of a large amount of streaming data, reducing the cost of data transmission and storage; at the same time, because the error is controllable, Serf-Qt can meet the requirements of real-time and accuracy in fields such as the Internet of Things while maintaining data quality.

[0063] The present invention can realize streaming floating point compression through a streaming error bounded floating point compression method (Serf-XOR) based on XOR operation. First, Serf-XOR converts the floating point value into a displacement value and further converts it into an approximate value. Although errors are introduced in the conversion process, the error accumulation between continuous data points can be controlled by designing approximate logic and subsequent XOR operation, ensuring that the overall error is within an acceptable range. In particular, when the change between data points is small, the XOR operation can more effectively reduce redundant information while maintaining the relative relationship between data and controlling errors. Secondly, the XOR operation utilizes the correlation between data, and reduces the amount of information to be stored by comparing the changes between adjacent data points. When the difference between data points is small, the XOR result tends to a smaller value, which is more efficient when encoding, thereby improving the compression efficiency. Further, Serf-XOR combines multiple technologies such as displacement, approximation and XOR, and can more deeply explore the redundancy between data while maintaining data characteristics; at the same time, since the XOR operation and encoding process are relatively simple, the overall compression and decompression process is efficient, which is suitable for application scenarios with high real-time requirements. Finally, the streaming time series data generated by devices such as the Internet of Things has a high degree of time correlation and continuity. Serf-XOR can make full use of these characteristics and achieve efficient compression by comparing the changes in adjacent data points. At the same time, Serf-XOR maintains the relative relationship between data during the compression process, so that the decompressed data can better restore the characteristics of the original data, meeting the requirements of data quality and real-time performance in fields such as the Internet of Things.

[0064] In the present invention, Serf-Qt or Serf-XOR can be selected according to user needs and error bound types to achieve streaming floating point compression, thereby improving the flexibility of streaming floating point compression. Among them, Serf-Qt works better when the error bound is relatively large, but it only supports absolute error bounds, and may perform poorly when the error bound is small; Serf-XOR supports both absolute error bounds and relative error bounds, and usually has better compression ratio performance than Serf-Qt when the error bound is small. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0066] Figure 1 It is an application scenario of streaming error bounded floating point compression;

[0067] Figure 2 This is the Serf-Qt framework diagram;

[0068] Figure 3 This is the Serf-XOR framework diagram;

[0069] Figure 4 is a diagram of a shifter;

[0070] Figure 5 For example, displacement;

[0071] Figure 6 is the basic approximator (∈=∈ a =0.01);

[0072] Figure 7 To optimize the approximator (∈=∈ a =0.01);

[0073] Figure 8 Search example for approximate value (assuming the number of digits bn in value is 5);

[0074] Fig. 9 Logic block diagram for SElf* coding;

[0075] Fig.10 Description of data transmission (the first 4 bits in "byte 1" indicate the number of bytes required for v after compression). DETAILED DESCRIPTION

[0076] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0077] The following is a further detailed description through specific implementation methods:

[0078] Example:

[0079] This embodiment discloses a streaming error-bounded floating-point data compression method.

[0080] A streaming error-bounded floating-point data compression method, including:

[0081] S1: Get the original time series of floating point type to be compressed;

[0082] S2: Select the corresponding streaming error-bounded floating-point compression method according to the set error-bounded type: if the quantization-based streaming error-bounded floating-point compression method (Serf-Qt) is selected, execute step S3; if the exclusive-OR-operation-based streaming error-bounded floating-point compression method (Serf-XOR) is selected, execute step S4;

[0083] In this embodiment, Serf-XOR performs better than Serf-Qt in terms of applicable scenarios (supporting both relative and absolute error bounds), compression rate, and decompression speed, while Serf-Qt performs better in terms of compression rate. Users can choose the appropriate compression method according to their needs and preferences.

[0084] S3: Figure 2 As shown, the floating point values ​​in the original time series are quantized into integer quantized values ​​to generate a quantized value sequence; the quantized values ​​in the quantized value sequence are encoded to generate a compressed bit stream as a compressed time series;

[0085] S4: Figure 3 As shown, the floating-point values ​​in the original time series are shifted to obtain shift values, generating a shift value sequence; the shift values ​​in the shift value sequence are converted into approximate values ​​to obtain an approximate value sequence; each approximate value in the approximate value sequence is XORed with its previous approximate value to obtain an XOR value, generating an XOR value sequence; the XOR values ​​in the XOR value sequence are encoded to generate a compressed bit stream as a compressed time series.

[0086] The present invention can realize streaming floating point compression through a streaming error bounded floating point compression method (Serf-Qt) based on quantization. First, Serf-Qt maps continuous floating point values ​​to integer sets through a quantization process, and ensures that data distortion in the compression process is strictly controlled within a preset error bound; at the same time, by adjusting the error bound, the best balance point can be found between the compression rate and the error, ensuring that the compressed data is within an acceptable error range. Secondly, compared with the original floating point values, the quantized integer sequence is more compact in representation, because the integer can be directly represented by binary bits, and there is no need to consider the exponent and mantissa of the floating point number. This representation method significantly reduces the demand for data storage space and improves compression efficiency. Finally, the streaming time series data generated by devices such as the Internet of Things has the characteristics of continuity and strong time correlation. Serf-Qt can effectively utilize these characteristics, and through reasonable quantization strategies and encoding methods, it can realize rapid compression and decompression of a large amount of streaming data, reducing the cost of data transmission and storage; at the same time, because the error is controllable, Serf-Qt can meet the requirements of real-time and accuracy in fields such as the Internet of Things while maintaining data quality.

[0087] The present invention can realize streaming floating point compression through a streaming error bounded floating point compression method (Serf-XOR) based on XOR operation. First, Serf-XOR converts the floating point value into a displacement value and further converts it into an approximate value. Although errors are introduced in the conversion process, the error accumulation between continuous data points can be controlled by designing approximate logic and subsequent XOR operation, ensuring that the overall error is within an acceptable range. In particular, when the change between data points is small, the XOR operation can more effectively reduce redundant information while maintaining the relative relationship between data and controlling errors. Secondly, the XOR operation utilizes the correlation between data, and reduces the amount of information to be stored by comparing the changes between adjacent data points. When the difference between data points is small, the XOR result tends to a smaller value, which is more efficient when encoding, thereby improving the compression efficiency. Further, Serf-XOR combines multiple technologies such as displacement, approximation and XOR, and can more deeply explore the redundancy between data while maintaining data characteristics; at the same time, since the XOR operation and encoding process are relatively simple, the overall compression and decompression process is efficient, which is suitable for application scenarios with high real-time requirements. Finally, the streaming time series data generated by devices such as the Internet of Things has a high degree of time correlation and continuity. Serf-XOR can make full use of these characteristics and achieve efficient compression by comparing the changes in adjacent data points. At the same time, Serf-XOR maintains the relative relationship between data during the compression process, so that the decompressed data can better restore the characteristics of the original data, meeting the requirements of data quality and real-time performance in fields such as the Internet of Things.

[0088] In the present invention, Serf-Qt or Serf-XOR can be selected according to user needs and error bound types to achieve streaming floating point compression, thereby improving the flexibility of streaming floating point compression. Among them, Serf-Qt works better when the error bound is relatively large, but it only supports absolute error bounds, and may perform poorly when the error bound is small; Serf-XOR supports both absolute error bounds and relative error bounds, and usually has better compression ratio performance than Serf-Qt when the error bound is small.

[0089] In order to better introduce the technical solution of the present invention, this embodiment is described through the following parts.

[0090] 1. Compression method selection

[0091] In this embodiment, the error bound types include absolute error bounds and relative error bounds.

[0092] Specifically, the streaming error-bounded floating-point compression method (Serf) includes a quantization-based streaming error-bounded floating-point compression method (Serf_Qt) and an XOR-based streaming error-bounded floating-point compression method (Serf_XOR). Serf_Qt works better when the error bound is relatively large, but it only supports absolute error bounds and may not perform well when the error bound is small. Serf_XOR supports both absolute and relative error bounds, and usually has better compression ratio performance than Serf-Qt when the error bound is small.

[0093] Therefore, when the error bound type is an absolute error limit, a streaming error bounded floating-point compression method based on quantization or XOR operation is selected (one of them is selected according to user needs); when the error bound type is a relative error limit, a streaming error bounded floating-point compression method based on XOR operation is selected.

[0094] 2. Prior Knowledge

[0095] In this embodiment, some definitions are given first.

[0096] Definition 1: Time series. Time series TS =<v1,v2,...> is a sequence of values ​​arranged in time order, where v i-1 Called v i precedent (i>1).

[0097] Definition 2: Lossy compression with bounded absolute error and bounded relative error. Assume that the original time series is TS =<v1,v2,...> , the decompressed time series is TS' =<v'1,v'2,...> . Lossy compression with bounded absolute error satisfies |v i -v' i |≤∈ a , where ∈ a >0 is a user-specified parameter. Relative error bounded lossy compression satisfies If v i = 0, then v' i =0; if v i ≠0, then |(v i -v' i ) / v i |≤∈ r , where ∈ r >0 is specified by the user.

[0098] IEEE754 floating point data structure. Double type data is stored in 64 bits, of which 1 bit is the flag bit s and 11 bits represent the exponent 52 bits represent the mantissa A standard floating point data type satisfies:

[0099]

[0100] 3. Serf_Qt

[0101] The main idea of ​​Serf_Qt is to convert floating point values ​​to integers, because integers are easier to compress than floating point values. It contains two main parts: Serf-Qt compressor and Serf-Qt decompressor.

[0102] 1. Serf_Qt compressor

[0103] like Figure 2 As shown on the left, the Serf_Qt compressor consists of two parts: the quantizer and the encoder.<v1,v2,v3,...> First, they are quantized into integers one by one<q1,q2,q3,...> Afterwards, the quantized values ​​are sequentially encoded into a bit stream.

[0104] 1.1 Quantizer

[0105] The quantizer of the Serf_Qt compressor converts the original time series TS =<v1,v2,...> The floating point value v in i Quantized to integer quantization value q i , generate a sequence of quantized values<q1,q2,...> ;

[0106] The specific steps are as follows:

[0107] S301: For floating point value v i Make a prediction and generate a predicted value p i ;

[0108] In this embodiment, v i The predicted value p i Defined as its antecedent v i-1 .

[0109] S302: According to the floating point value v i The predicted value p i Combined with the absolute error bound ∈ a Calculate the quantized value q i ;

[0110] The formula is:

[0111]

[0112] Where: [*] indicates the operation of rounding the floating point value to an integer.

[0113] Given a quantized value qi , we can recover the value v′ by the following equation i :v′ i =2×∈ a ×q i +p i This method can ensure that the recovered value strictly satisfies the error bound constraint, as shown below:

[0114]

[0115] As far as the forecasting strategy is concerned, given that two consecutive values ​​in a time series usually change very little, we can simply let v i The predicted value p i For its precedent i-1 In addition, using this simple strategy can also improve the compression / decompression efficiency. However, due to the i During the decompression process, we cannot accurately know v i-1 , so we should set p i is the last restored value v′ i-1 , that is, p i =v′ i-1 (Note that we can compute v′ during both compression and decompression i-1 ). In particular, we set v′0=v0=0.

[0116] 1.2 Encoder

[0117] The encoder needs to encode the quantized values ​​sequentially into a bit stream. Due to the requirement of streaming, batch Huffman coding strategy and finite state entropy (FSE) coding strategy are not applicable. Therefore, we design a stream encoder for the quantized values.

[0118] Since the difference between the original floating-point value vi and its predicted value pi is small, the quantized value qi tends to be distributed around 0. In view of this, we use Elias Gamma coding to encode the quantized value. Elias Gamma coding is most commonly used to encode integers whose upper limits cannot be determined in advance, and is a self-interpreting stream coding method. However, it can only encode positive integers. Therefore, we first convert the quantized value into a positive integer based on Zigzag coding, and then encode the result using Elias Gamma coding. Among them, Elias Gamma coding (Eliasγ code / Elias Gamma code) is a general variable-length coding method for positive integers, which is characterized by optimizing storage space by reducing the occupancy of invalid bits. Zigzag coding is an encoding method for integers that achieves data compression by mapping signed integers to unsigned integers.

[0119] Therefore, the encoder of the Serf_Qt compressor converts the quantized value sequence into<q1,q2,...> The quantized value q in i Convert to a positive integer, and then encode the result of Zigzag encoding through Elias Gamma encoding to get the encoding value code i , generating a compressed bitstream<code1,code2,...> ;

[0120] The formula is:

[0121] code i = Elias GammaEn(ZigzagEn(q i )+1);

[0122] in

[0123] Wherein: ZigzagEn represents Zigzag encoding; Elias GammaEn represents Elias Gamma encoding; ⊕ represents bitwise exclusive OR operation.

[0124] In this embodiment, when q i = 0, the Zigzag encoding will return 0, so an extra 1 is needed to ensure that the input of the EliasGamma encoding is positive. Assume N i = ZigzagEn(q i )+1, Elias Gamma encoding will be written Bit 0, write N after it i For example, if q i =5, then N i = ZigzagEn(q i )+1=11, where N i The binary representation of is (1011)2, so the code i = Elias GammaEn(N i )=(0001011)2.

[0125] This embodiment describes the Serf-Qt compressor through Algorithm 1 in the following table, which takes the current value v i , the last restored value v′ i-1 , absolute error bound ∈ a and stream writer out as input. Its time complexity and space complexity are both O(1).

[0126]

[0127] 2. Serf_Qt decompressor

[0128] The Serf-Qt decompressor performs the inverse operation of the Serf-Qt compressor, which mainly consists of a decoder and a dequantizer. Figure 2 As shown on the right, we first decode the bit stream to obtain a series of quantized values<q1,q2,q3,...> Subsequently, the quantized values ​​are dequantized into a time series<v′1,v′2,v′3,...> , where the error bounds for each value are within the user-specified constraints, i.e., |v i -v′ i |≤∈ a .

[0129] 2.1 Decoder

[0130] The Serf_Qt decompressor decoder decodes the bitstream via Elias Gamma decoding<code1,code2,...> The coded value in is decoded, and then Zigzag decoding is used to decode the result of Elias Gamma decoding to obtain the quantized value q i , decoding generates a sequence of quantized values<q1,q2,...> ;

[0131] The formula is:

[0132] q i =ZigzagDe(EliasGammaDe(code i )-1);

[0133] Where: Elias GammaDe represents Elias Gamma decoding; ZigzagDe represents Zigzag decoding;

[0134] In this embodiment, Elias Gamma decoding first starts from code i Read and count the occurrences of 0 in the binary representation until the first 1 appears (assuming the number of 0s is N i ), and then read the remaining N i The first 1 read and the following N i The number of bits constitutes the decoding result. For example, if code i =(000 1011)2, then the number of zeros is N i is 3, so in addition to the first bit 1 that has been read, we need to read 3 more bits to form N i = Elias GammaDe(code i)=(1011)2=11, we subtract 1 to offset the 1 added during compression. For Zigzag decoding, assuming x is the input number of Zigzag decoding, then ZigzagDe(x)=(x>>>1)⊕(-(x&1)).

[0135] 2.2 Dequantizer

[0136] The dequantizer of the Serf_Qt decompressor performs a sequence of quantized values<q1,q2,...> The quantized value q in i Dequantize to recover the floating point value v′ i , generate decompressed time series TS′=<v1′,v2′,...> ;

[0137] The formula is:

[0138] v′ i =2×∈ a ×q i +p i ;

[0139] Where: ∈ a represents the absolute error limit; p i Represents a floating point value v i The predicted value of . i =v′ i-1 (v′0=v0=0).

[0140] This embodiment shows the pseudo code of the Serf-Qt decompressor through Algorithm 2 in the following table. In line 2, the loop is executed at most bn times, where bn is the number of bits required for each floating point value. Therefore, its time complexity is O(bn), and its space complexity is O(1).

[0141]

[0142] Serf_XOR

[0143] Serf-Qt works better when the error bound is relatively large, but it does not support lossy compression with relative error bounds. a If is very small, the performance will be very poor, and the quantized value may overflow (i.e., exceed the range that can be represented by an integer). Therefore, this embodiment further proposes a method based on XOR operation, called Serf-XOR (such as Figure 3 In order to improve the compression rate of Serf-XOR, this embodiment proposes a novel data offset technology to increase the number of leading zeros of the XOR value, a novel approximation technology to search for the best approximation within the error bound, an adaptive coding strategy to compactly encode the XOR value, and a pruning strategy to speed up the approximation search process.

[0144] Serf-XOR, which not only solves the above problems, but also achieves better compression performance. Serf-XOR mainly consists of two parts: Serf-XOR compressor and Serf-XOR decompressor.

[0145] 1. Serf_XOR compressor

[0146] like Figure 3 As shown on the left, the Serf-XOR compressor consists of a shifter, an approximator, and an encoder.<v1,v2,v3...> Sequentially flow into the shifter to obtain the shift value sequence<s1,s2,s3,…> This ensures that they have the same sign and the same exponent. The approximator then converts the shifted values ​​one by one into a series of approximate values<a1,a2,a3,...> Each approximate value ai (except the first value a1) is related to its previous value a i-1 Perform an XOR operation to generate a series of XOR values<xor1,xor2,xor3,...> , contains many leading and trailing zeros, which can be compactly compressed using an encoder.

[0147] 1.1 Displacer

[0148] The shifter of the Serf_XOR compressor is the original time series TS =<v1,v2,...> The floating point value v in i Add the integer offset λ to get the displacement value s i =v i +λ, generate a sequence of displacement values<s1,s2,...> ;

[0149] in

[0150] Where: max and min represent the original time series TS=<v1,v2,...> The maximum and minimum values ​​in; u is a positive integer; Indicates rounding up; Indicates rounding down. The maximum and minimum values ​​in the original time series can be obtained through domain knowledge (for example, the latitude of GPS points is between -180 and 180, and sensors also usually have a measurement range). They can also be set as the historical maximum and minimum values ​​of the time series, respectively, and adjusted over time in the streaming scenario. The maximum and minimum values ​​of the time series can be recorded in the metadata.

[0151] In this embodiment, the shifter adds an integer offset λ to each v i Thus, a shift value s is obtained that has the same sign (i.e., positive number) and the same exponent as the other shift values. i =v i+λ. Therefore, any two consecutive shift values ​​can produce an XOR value with many leading zeros.

[0152] How to determine the appropriate integer offset λ: On the one hand, λ should be large enough so that all s i are positive and have the same exponent. On the other hand, Figure 4 As shown in , adding an offset may cause the mantissa to shift right, resulting in some information loss. Larger offsets generally mean more information loss. In the extreme case, if λ is too large, s i may overflow. To this end, we let λ be such that all s i Falling in [2 u ,2 u+1 ), where u is a positive integer. For the convenience of description, we give the following definition.

[0153] Definition 3: Truncation. Given a floating point value v, the truncation operation is to truncate the integer part of v, denoted by ||v||. For example, ||3.14||=3, ||-3.14||=-3.

[0154] If v ≥ 0, then If v<0, then Furthermore, for any floating-point value v and any integer λ, we have In order to find the values ​​of u and λ, we proposed Theorem 1.

[0155] Theorem 1: Assume that min and max are time series TS=<v1,v2,...> The minimum and maximum possible values ​​in . If and Then it can be guaranteed that all shift values ​​s i =v i +λ in [2 u ,2 u+1 ) range, and λ is also minimum.

[0156] Figure 5 Here is an example. Suppose the minimum and maximum values ​​in the time series are -1.0 and 2.5 respectively. Then we have The original two values ​​-0.96 and 2.02 have different signs and exponents, resulting in no leading zeros in their XOR results. After adding λ=5, the two shifted values ​​4.04 and 7.02 can share the same sign and exponent bits, resulting in up to 12 leading zeros in the XOR result.

[0157] 1.2 Approximator

[0158] Approximator of the Serf_XOR compressor for a sequence of displacement values<s1,s2,...> Median displacement value s i Find the approximate value ai ∈[s i -∈,s i +∈], so that a i and a i-1 Share the most suffix bits to generate a sequence of approximate values<a1,a2,...> ;

[0159] Where: ∈ = ∈ a or ∈=∈ r ×|v i |∈ a represents the absolute error limit; ∈ r Represents the relative error limit.

[0160] In this embodiment, in the approximator module, for each shift value s i , we try to find an approximate value a i ∈[s i -∈,s i +∈], so that a i and a i-1 can share the most suffix bits (i.e., their XOR results have the most trailing zero bits). Here ∈=∈ a (i.e. based on the absolute error bound) or ∈=∈ r ×|v i |(i.e. based on relative error bounds).

[0161] Figure 6 The basic idea of ​​the approximator is shown, where a i-1 =2.535,s i =2.81,∈=∈ a = 0.01. In order to find an approximate value a i ∈[si-∈,si+∈], we can simply replace s i The last 45 bits are set to a i-1 , and get a i =2.800625 (i.e. a i By s i The 19 prefix digits and a i-1 The 45 suffix bits of Therefore, we call it s i is the prefix anchor, called a i-1 is the suffix anchor). i and a i-1 By XORing, we can get an XOR value with many leading and trailing zeros. When decompressing, due to a i ∈[s i -∈,s i +∈], we have v′ i =a i-λ∈[s i -λ-∈,s i -λ+∈], that is, v′ i ∈[v i -∈,v i +∈] satisfies the error bound requirement.

[0162] How to efficiently find a suitable i : A simple solution is to iteratively check Does the error bound requirement apply, where j traverses from bn to 0, and bn is the number of bits per floating point number (e.g., for double precision values, bn=64). If so, stop the verification process and return a i However, there may be two problems. Problem 1: The a obtained in this way i may not be optimal (i.e., it may not be consistent with a i-1 share the most suffix bits). For example, there is another Figure 7 The suitable value is 2.81625∈[2.80,2.82], which is consistent with a i-1 Shared suffix bits ratio Figure 6 The 2.800625 shown is more. Question 2: Enumerate all suffixes suf from the beginning i It may take a lot of time, which will affect the compression efficiency.

[0163] To this end, the present invention proposes a new method that can find the theoretically best approximate value a in a more efficient way. i To solve problem 1, we proposed Theorem 2 and made two optimizations. First, we let the prefix anchor be low = s i -∈ instead of s i . Secondly, in addition to checking Whether it is outside [si-∈,si+∈], we also check Figure 8 Three examples of splicing combinations are shown. Figure 8 In (c), we find Satisfies the error limit requirement, so we finally set a i =c2=(01100)2.

[0164] Theorem 2: Assume low = s i -∈ is the prefix anchor, a i-1 is the suffix anchor. If we iteratively check the two concatenation combinations and Among them, j traverses from bn to 0, and we can always get a matching condition with a i-1 The approximate value a that shares the most suffix bits i .

[0165] In Theorem 2, we need to iterate j bn+1 times, which is expensive. However, we find that it is actually not necessary to iterate j from the beginning, and propose Theorem 3 to solve Problem 2.

[0166] Theorem 3: Assume low = s i -∈,up=s i +∈, l = lead(low⊕up) represents the number of leading zeros in the low⊕up XOR result. If we start iteration j from bn–l instead of bn, we can still get the best a i .

[0167] According to Theorem 3, iterating j from bn-l to 0 is sufficient to find the best a i Since ∈ is usually small, the difference between low and up is usually small, i.e., the number of leading zeros l of low⊕up is large, which significantly reduces the iteration cost. For example, Figure 8 As shown, since lead(low⊕up)=2, we only need to iterate j from 5-2=3 instead of starting from 5.

[0168] In summary, this embodiment uses the following steps to calculate the displacement value s i Find the approximate value a i :

[0169] S401: Obtain error limit ∈, previous displacement value s i-1 and i-1 The corresponding approximate value a i-1 ;

[0170] S402: Calculate low=s respectively i -∈,up=s i +∈, l = lead(low⊕up);

[0171] S403: Constructing a splicing combination Among them, pre bn-j Indicates that bn-j prefix bits are selected from low, suf j Indicates from a i-1 Select j suffix bits from , bn represents the number of floating point values ​​in the original time series; Represents a concatenation operation, such as

[0172] S404: Select j in order from bn-1 to 0 until c1 or c2∈[s i -∈,s i +∈];

[0173] S405: Use c1 or c2 that meets the condition as the displacement value si The approximate value of a i . When there is a combination that meets the conditions, there is no need to check another combination.

[0174] 1.3 XOR operation

[0175] The Serf_XOR compressor converts the sequence of approximated values ​​into<a1,a2,...> Each approximation a in i Its previous approximation a i-1 Perform XOR operation to get XOR value xor i , generates a sequence of XOR values<xor1,xor2,...> .

[0176] 1.4 Encoder

[0177] The encoder of the Serf_XOR compressor is based on the SElf * Encoding strategy for XOR value sequence<xor1,xor2,...> XOR value in i Encode to get the encoding value code i , generating a compressed bitstream<code1,code2,...> .

[0178] To encode XOR values ​​with many leading and trailing zeros, we adopt the encoding strategy of SElf*. SElf* proposes the concept of approximate rules (a sorted array of integers). Fig. 9 As shown in (a), assuming that the approximate rule for leading zeros is rule lead = <0,8,12,16,18,20,22,24>, although the number of leading zeros is 13, we approximate it to 12 (because 12≤13<16). Therefore, the number of leading zeros can be expressed as bits (called representation cost), rather than bits (the number of leading zeros of the XOR result of a double-precision floating-point number can be 0 to 64, a total of 65 cases). After approximation, the 13th leading zero will be considered as part of the center bit, which will cause the center bit to occupy one more bit of space (called the approximation cost). In addition, SElf* also proposed a method based on dynamic programming to quickly find the best trade-off between representation cost and approximation cost.

[0179] Fig. 9 (c) shows the SElf* for the current XOR value xor i =a i ⊕a i-1 There are three encoding strategies. Case 1: If xor i = 0, it only writes the 2-bit flag "01". Case 2: If xor i≠0 and condition C is true, then write a 1-bit flag "1" followed by the approximate center bit. Here, C is the "lead i ≥lead i-1 , trail i ≥trail i-1 And (lead i -lead i-1 )+(trail i -trail i-1 )<1+ln+tn", where lead i and trail i They are xor i Approximate leading and trailing zero counts, ln and tn are defined as Fig. 9 (b) as shown. Case 3: If xor i ≠0 and condition C is not true, then write the two-bit flag "00", the ln bit lead i , tn trail i and the approximate center bit. Note that since SElf* finds case 2 to be the most common of the three cases for lossless compression, the flag it uses is the shortest.

[0180] However, the situation of lossy compression is different. According to the analysis on each data set, the fixed flag of SElf* may not be optimal. Therefore, this embodiment proposes to adaptively assign the shortest flag "1" of 1 bit to the most common situation. Specifically, we assign fixed flags at the beginning, then calculate the frequency of each situation during streaming compression, and dynamically adjust the flag allocation according to the statistical data.

[0181] In summary, the processing steps of the encoder in this embodiment are as follows:

[0182] The encoding process steps are as follows:

[0183] S411: Get the approximate rule for leading and trailing zeros lead and rule trail , and SElf * The flag bits corresponding to each case in the encoding strategy;

[0184] In this embodiment, Fig. 9 As shown in (a), the approximate rule lead =rule trail =<0,8,12,16,18,20,22,24>.

[0185] SElf *There are three encoding strategies. In lossless compression, the flag bit corresponding to "case one" is 01, the flag bit corresponding to "case two" is 1, and the flag bit corresponding to "case three" is 00.

[0186] S412: For XOR value sequence<xor1,xor2,...> Each XOR value in xor i , perform the following steps:

[0187] S4121: Determine the XOR value xor i Leading zero count, trailing zero count and center bit of ;

[0188] In this embodiment, leading zeros refer to all zeros before the first significant digit, which is represented in binary as all zeros before the highest bit with a value of 1. Trailing zeros refer to all zeros after the last significant digit, which is represented in binary as all zeros after the lowest bit with a value of 1.

[0189] S4122: According to rule lead and rule trail Get the approximate leading zero count, trailing zero count and center bit;

[0190] In this embodiment, Fig. 9 As shown in (a), the number of leading zeros is 13, but it is approximated to 12 according to the approximation rule (because 12≤13<16), that is, the first 12 zeros are approximated leading zeros. After approximation, the 13th leading zero will be regarded as part of the center bit (that is, the approximate center bit), which will cause the center bit to occupy one more space (called the approximation cost). Trailing zeros are handled in the same way as leading zeros.

[0191] S4123: Calculation and

[0192] S4124: Define condition C as: lead i ≥lead i-1 、trail i ≥trail i-1 And (lead i -lead i-1 )+(trail i -trail i-1 )<1+ln+tn; where lead i 、trail i They are xor i Approximate leading and trailing zero counts;

[0193] S4125: Determine whether situation 1 is satisfied: xor i =0: If yes, then codei The code is a flag bit corresponding to case 1; otherwise, execute step S4126;

[0194] S4126: Determine whether condition 2 is satisfied: xor i ≠0 and condition C is true: If so, then code i The code is: the flag bit corresponding to case 2 is followed by the approximate center bit; if not, it is judged that case 3 is satisfied, code i The encoding is: the flag bit corresponding to case three is followed by the lead bit of the ln position i , tn trail i and the approximate center position;

[0195] In this embodiment, ln and tn represent lead i and trail i The required binary bit width. i =(4) 10 =(100)2, then ln=3.

[0196] S413: Approximate rule update: based on XOR value sequence<xor1,xor2,...> Each XOR value xor i Update the approximate rule for leading and trailing zeros by using the leading and trailing zero counts of lead and rule trail ;

[0197] S414: Adaptive flag adjustment: Statistical XOR value sequence<xor1,xor2,...> The frequency of occurrence of each case in the encoding result; the flag bit corresponding to the case with the highest frequency is set to 1, and the flag bits corresponding to the second and third highest frequencies are set to 00 or 01 respectively.

[0198] In this embodiment, the encoding results of the XOR value sequence are counted and analyzed. For example, the flag bit "01" (i.e., case one) appears most frequently, so the flag bit of case one is adjusted to 1, and the flag bits of case two and case three are adjusted to 01 or 00. Since the lengths of 01 and 00 are the same, no distinction is made.

[0199] The present invention defines the approximate rules for leading zeros and trailing zeros, and SElf * The encoding strategy performs adaptive identification adjustment, which can reduce the length of compressed data as much as possible when compressing floating-point type data, thereby improving compression efficiency.

[0200] This embodiment describes the pseudo code of the Serf-XOR compressor through Algorithm 3 shown in the following table, which takes the current value v i , the previous approximation a i-1(we set a0=2), error bound ∈, approximate rule for leading / trailing zeros, flags for three cases, and output stream out as input. In line 19, we update and write rule and flags at the end of each window (e.g., 1,000 values) to obtain better approximate rules and flags in subsequent compression. We iterate j at most bn-l times, so its time complexity is O(bn-l). We need to store statistics on the frequency of occurrence of each case (for updating the allocation of flags) and leading / trailing zeros (for updating the approximate rules), both of which contain no more than bn records. Therefore, the space complexity of Algorithm 3 is O(bn).

[0201]

[0202] 2. Serf_Qt decompressor

[0203] like Figure 3 As shown on the right, the decoder of the Serf_Qt decompressor processes the compressed bitstream<code1,code2,...> Decode the encoded value in to get the XOR value xor i , decoding generates a sequence of XOR values<xor1,xor2,...> ; For XOR value sequence<xor1,xor2,...> XOR value in i Perform XOR operation to get approximate value a i , decoding generates an approximate value sequence<a1,a2,...> ; The approximate value sequence is transformed into<a1,a2,...> Each approximation a in i Converts to restore the floating point value v i ', generate decompressed time series TS' =<v1′,v2′,...> In Serf-XOR, the decompressor is not the strict inverse of the compressor.

[0204] This embodiment provides a pseudo code of the Serf-XOR decompressor through Algorithm 4 shown in the following table, which uses the previous approximation a i-1 , approximate rule rule, three flags flags and input stream in as input. Lines 1 to 15 are the relevant codes of the decoder. Since the flag code is "1", "00" or "01", we first try to read one bit. If it is "0", we read another bit to splice out the complete flag. Then, we process these three cases according to the value of the flag, and finally get the approximate value a in line 15 i In line 16, the inverse shifter computes the decompressed value v′ i =a i-λ. Similar to the Serf-XOR compressor, we need to read and update the approximation rules and flag assignments at the end of each window for subsequent decompression. Since there are no loops in Algorithm 4, its time complexity is O(1). In addition, we only keep a few states in memory (such as lead i-1 and trail i-1 ), so its space complexity is O(1).

[0205]

[0206] It should be noted that according to the IEEE 754 floating point representation format, in addition to double precision (bn = 64), floating point values ​​have other precisions, such as single precision (bn = 32) and half precision (bn = 16). With the advancement of deep learning, other floating point data types have emerged, such as Float16, Float32, and Tensor Float32. All of these data types have a similar basic layout to double precision values, but the exponent and mantissa have different bits. For example, for single precision, its exponent and mantissa occupy 8 bits and 23 bits, respectively. The methods we proposed (Serf-Qt and Serf-XOR) can be easily extended to floating point values ​​of other precisions. Specifically, for Serf-Qt, since it is independent of precision, no modification is required. For Serf-XOR, just set bn to the number of bits of the target precision in Algorithms 3 and 4.

[0207] 5. Experimental Description

[0208] To verify the performance of Serf, we conducted overall comparison experiments with baseline methods, data transmission simulation experiments, a series of parameter experiments, and ablation experiments.

[0209] 1. Dataset

[0210] We use 13 time series datasets, where records are sorted by timestamp. The details of these datasets are shown in the dataset information in Table 1.

[0211] Table 1 Dataset information

[0212]

[0213] 2. Experimental Setup

[0214] All experiments are conducted on a server equipped with Ubuntu 22.04, Intel Xeon E-2324G(4)@4.600GHz, 128GB RAM and 4T disk storage. The parameters of the comparative experiments are set to ∈ a =10 -3 ,∈r =1%, the batch size is 50, and the bandwidth used in the data transmission simulation experiment is 80, 160, 240, 400, and 800 kbit / s.

[0215] 3. Experimental methods

[0216] We compare Serf with 13 representative comparative methods, including three batch lossless methods (i.e., LZ77 (from Ziv J, Lempel AA universal algorithm for sequential data compression[J]. IEEE Transactions on information theory, 1977, 23(3): 337-343.), Zstd (from YCollet.2016.Zstd github repository from facebook.Retrieved March 19,2023fromhttps: / / github.com / facebook / zstd.) and Snappy (from Google.2023.Snappy|A fast compressor / decompressor.Retrieved March 19,2023from https: / / github.com / google / snappy.)), four batch lossy methods (i.e., SZ2 (from Liang X, Di S, Tao D, et al.Error-controlled lossy compression optimized for high compression ratios of scientific datasets[C] / / 2018IEEE International Conference on Big Data(BigData).IEEE,2018:438-447.), Machete(from Shi Y, Zou X, Liakos P, Papakonstantinopoulou K, et al. Sim-piece: highly accurate piecewise linear approximation through similar segment merging[J].Proceedings of the VLDB Endowment, 2023, 16(8): 1910 - 1922.) and Buff (from Liu C, Jiang H, Paparrizos J, et al. Decomposed bounded floats for fast compression and queries [J]. Proceedings of the VLDB Endowment, 2021, 14(11): 2586 - 2598.)) and six streaming lossless methods (i.e., Deflate (from Oswal S, Singh A, Kumari K. Deflate compression algorithm [J]. International Journal of Engineering Research and General Science, 2016, 4(1): 430 - 436.), LZ4 (from Bartík M, Ubik S, Kubalik P. LZ4 compression algorithm on FPGA [C] / / 2015 IEEE International Conference on Electronics, Circuits, and Systems (ICECS). IEEE, 2015: 179 - 182.), FPC (from Burtscher M, Ratanaworabhan P. FPC: A high - speed compressor for double - precision floating - point data [J]. IEEE transactions on computers, 2008, 58(1): 18 - 31.), Gorilla (from Pelkonen T, Franklin S, Teller J, et al. Gorilla: A fast, scalable, in - memory time series database [J]. Proceedings of the VLDB Endowment, 2015, 8(12): 1816 - 1827.), Chimp128 (from Liakos P, Papakonstantinopoulou K, Kotidis Y. Chimp: efficient lossless floating point compression for time series databases [J].Proceedings of the VLDB Endowment, 2022, 15(11): 3058-3070.) and Elf (from Li R, Li Z, Wu Y, et al. Elf: Erasing-based lossless floating-point compression [J]. Proceedings of the VLDB Endowment, 2023, 16(7): 1763-1776.)). For batch methods, we run them in small batches. Most of the source code (except Buff, which is written in Rust) is written in C++ and gcc 11.4.0, and the code and datasets are open source. .

[0217] 4. Comparison Standards

[0218] The performance of various methods is verified by three main indicators: compression ratio, compression time, and decompression time. The compression ratio in this experiment is defined as the ratio of the compressed data size to the original data size.

[0219] 5. Compression ratio

[0220] As shown in Table 2, compared with all methods, Serf always has the best compression ratio performance on all datasets. Specifically, compared with the best batch lossless method Zstd, Serf-XOR has an average relative improvement of 75.5% in compression ratio (i.e. (0.65-0.16) / 0.65≈75.5%); compared with batch lossy methods, since most of them are designed for large batches, their performance is not as good as Serf when the batch size is 50. Buff shows the best compression ratio (i.e. 0.28) in this category, but it can only support formats with 10 -k , and Serf-XOR still has an average relative improvement of 42.9% over it; among the streaming lossless methods, Elf has the best compression ratio (i.e., 0.33) because it uses erasure technology to greatly increase trailing zeros, but Serf's compression ratio performance is still significantly ahead.

[0221] In addition, experiments show that the compression ratio of Serf-XOR is slightly better than that of Serf-Qt (i.e., 0.16 VS 0.18), but when ∈ a = 0.001, their difference is small. Serf-XOR uses a series of optimization techniques, including data offset, value approximation and adaptive coding, which help it achieve better compression ratio.

[0222] Table 2 Overall comparison - compression ratio

[0223]

[0224] 6. Compress time

[0225] We also experimented with compression time for various algorithms, and the results are shown in Table 3. In each category, Snappy, Buff, Gorilla, and Serf-Qt have the shortest compression time, respectively. Here we mainly discuss the streaming method, because the batch method cannot be applied to the streaming scenario. Gorilla only needs 0.02μs, because it only performs XOR operation on each record without any complex logic. However, its compression ratio is as high as 0.75, which is about 4.69 times that of Serf-XOR, and in the IoT data transmission scenario, bandwidth is usually a bottleneck. Elf, the most competitive streaming competitor in terms of compression ratio, takes more compression time than Serf-Qt and Serf-XOR, because the calculation of the decimal effective digits in Elf is relatively time-consuming. Serf-XOR takes slightly longer compression time than Serf-Qt because Serf-XOR needs to perform more operations (data offset and approximate value search).

[0226] Table 3 Overall comparison - compression time

[0227] (Unit: μs. 0.00 means the time is too small to be expressed with only two decimal places)

[0228]

[0229] 7. Decompression time

[0230] We also experimented with the decompression time of various algorithms, and the results are shown in Table 4. LZ77, Machete, Elf, and Serf-XOR showed the best decompression efficiency in each category, respectively. We still focus on the streaming algorithms here. Elf performs the best among all streaming methods, probably for two reasons: first, it has a relatively better compression ratio than other streaming lossless methods, resulting in fewer bits to read; second, Elf has much fewer conditional branches than Serf. In contrast, Serf-XOR needs to constantly determine whether the flag and approximation rules should be modified.

[0231] Table 4 Overall comparison - decompression time (×0.1μs)

[0232]

[0233] Serf-XOR outperforms Serf-Qt in terms of decompression time for the following reasons: first, Serf-XOR has a better compression ratio than Serf-Qt, which helps reduce the number of decompressed bits; second, in many cases, the value after Serf-XOR XOR is 0. In these cases, we only read the flag bit during the decompression process, but not other bits (i.e., lines 4 to 6 in Algorithm 4). For Serf-Qt, multiple reads are still required when counting the number of 0s in front of the input stream.

[0234] In general, the Serf algorithm of the present invention demonstrates excellent performance in comparative experiments with 13 advanced comparison methods.

[0235] 8. Data transmission simulation experiment

[0236] In order to verify the overall performance of the method proposed in the present invention in the streaming data transmission scenario, we conducted a set of simulation experiments at different bandwidths ranging from 80 kbit / s to 800 kbit / s. Fig.10 (a) describes the simulation framework, where Server 1 represents a sensor that continuously generates a series of records, compresses each record once it is generated, and sends the compressed bits over the network; Server 2 is responsible for monitoring the input stream, reading the received bytes, and decoding the bytes into values. Since the network can only transmit data in bytes and we cannot distinguish the boundary between two records in the stream, we use 4 bits to indicate the number of bytes to read when decoding each value. For example, Fig.10 As shown in (a), the first 4 bits in byte 1 indicate that we should read two bytes (including the first 4 bits) to decode v. The last two bits in byte 2 are probably just padding and should be discarded.

[0237] Fig.10 (b) shows the total time (including compression time, transmission time and decompression time) taken by each method when all data is sequentially transmitted from server 1 to server 2. It can be observed that as the bandwidth increases, the total time required for all methods will decrease, but their relative performance remains unchanged, and Serf-Qt and Serf-XOR always perform best. For example, when the bandwidth is 240kbit / s, Serf-XOR only takes about 52 seconds, while the most competitive method Elf takes about twice as long (i.e., 93 seconds). Serf-XOR performs slightly better than Serf-Qt due to its relatively better compression ratio. This set of experiments demonstrates the feasibility and importance of the Serf method proposed in the present invention in streaming data transmission with limited bandwidth.

[0238] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A streaming error-bounded floating-point data compression method, characterized in that: include: S1: Get the original time series of floating point type to be compressed; S2: Select the corresponding streaming error-bounded floating-point compression method according to the set error-bounded type: if the quantization-based streaming error-bounded floating-point compression method is selected, execute step S3; If the streaming error bounded floating point compression method based on XOR operation is selected, step S4 is executed; S3: quantize the floating point values ​​in the original time series into integer quantized values ​​to generate a quantized value sequence; encoding the quantized values ​​in the quantized value sequence to generate a compressed bit stream as a compressed time series; S4: Shift the floating-point values ​​in the original time series to obtain shift values, and generate a shift value sequence; convert the shift values ​​in the shift value sequence into approximate values, and obtain an approximate value sequence; perform an XOR operation on each approximate value in the approximate value sequence and its previous approximate value to obtain an XOR value, and generate an XOR value sequence; encode the XOR values ​​in the XOR value sequence, and generate a compressed bit stream as a compressed time series.

2. The method for compressing floating-point data with bounded error according to claim 1, wherein: In step S2, the error bounded types include absolute error bounds and relative error bounds; When the error bound type is an absolute error bound, a quantization-based or XOR-based streaming error bounded floating point compression method is selected; When the error bound type is relative error bound, a streaming error bounded floating point compression method based on XOR operation is selected.

3. The streaming error-bounded floating-point data compression method according to claim 1, characterized in that: In step S3, the original time series TS=<v1,v2,...> The floating point value v in i Quantized to integer quantization value q i , generating a sequence of quantized values<q1,q2,...> ; The specific steps are as follows: S301: For floating point value v i Make a prediction and generate a predicted value p i ; S302: According to the floating point value v i The predicted value p i Combined with the absolute error bound ∈ a Calculate the quantized value q i ; The formula is: Where: [*] indicates the operation of rounding the floating point value to an integer.

4. The streaming error-bounded floating-point data compression method according to claim 3, characterized in that: In step S3, the quantized value sequence is converted into<q1,q2,...> The quantized value q in i Convert to a positive integer, and then encode the result of Zigzag encoding through EliasGamma encoding to get the encoding value code i , generating a compressed bitstream<code1,code2,...> ; The formula is: code i =Elias GammaEn(ZigzagEn(q i )+1); Wherein: ZigzagEn represents Zigzag encoding; Elias GammaEn represents Elias Gamma encoding.

5. The streaming error-bounded floating-point data compression method according to claim 4, characterized in that: In step S3, the bit stream is decoded by Elias Gamma<code1,code2,...> The coded value in is decoded, and then Zigzag decoding is used to decode the result of Elias Gamma decoding to obtain the quantized value q i , decoding generates a sequence of quantized values<q1,q2,...> ; The formula is: q i JZigzagOf(Elias GammaOf(code i )-1) Where: Elias GammaDe represents Elias Gamma decoding; ZigzagDe represents Zigzag decoding; The dequantizer of the Serf_Qt decompressor performs a sequence of quantized values<q1,q2,...> The quantized value q in i Dequantize to recover the floating point value v′ i , generate the decompression time series tS′=<v1′,v2′,...> ; The formula is: in i ′ =2×∈ a ×q i +p i ; Where: ∈ a represents the absolute error limit; p i Represents a floating point value v i The predicted value of .

6. The streaming error-bounded floating-point data compression method according to claim 1, characterized in that: In step S4, the original time series TS=<v1,v2,...> The floating point value v in i Add the integer offset λ to get the displacement value s i =v i +λ, generate a sequence of displacement values<s1,s2,...> ; in Where: max and min represent the original time series TS=<v1,v2,...> The maximum and minimum values ​​in; u is a positive integer; Indicates rounding up; Indicates rounding down.

7. The streaming error-bounded floating-point data compression method according to claim 6, characterized in that: In step S4, the displacement value sequence is<s1,s2,...> Median displacement value s i Find the approximate value a i ∈[s i -∈,s i +∈], so that a i and i-1 Share the most suffix bits to generate a sequence of approximate values<a1,a2,...> ; Where: ∈ = ∈ a or ∈=∈ r ×|v i |∈ a represents the absolute error limit; ∈ r Represents the relative error limit.

8. The streaming error-bounded floating-point data compression method according to claim 7, characterized in that: In step S4, the displacement value s is obtained by the following steps: i Find the approximate value a i : S401: Obtain error limit ∈, previous displacement value s i-1 and i-1 The corresponding approximate value a i-1 ; S402: Calculate low=s respectively i -∈,up=s i +∈, l = lead(low⊕up); S403: Constructing a splicing combination Among them, pre bn-j Indicates that bn-j prefix bits are selected from low, suf j Indicates from a i-1 Select j suffix bits from , bn represents the number of floating point values ​​in the original time series; Represents a splicing operation; S404: Select j in order from bn-1 to 0 until c1 or c2∈[s i -∈,s i +∈]; S405: Use c1 or c2 that meets the condition as the displacement value s i The approximate value of a i .

9. The streaming error-bounded floating-point data compression method according to claim 7, characterized in that: In step S4, the approximate value sequence<a1,a2,...> Each approximation a in i Its previous approximation a i-1 Perform XOR operation to get XOR value xor i , generates a sequence of XOR values<xor1,xor2,...> ; Through SElf based on adaptive identification adjustment * Encoding strategy for XOR value sequence<xor1,xor2,...> XOR value in i Encode to get the encoding value code i , generating a compressed bitstream<code1,code2,...> ; The encoding process steps are as follows: S411: Get the approximate rule for leading and trailing zeros lead and rule trail , and SElf * The flag bits corresponding to each case in the encoding strategy; S412: For XOR value sequence<xor1,xor2,...> Each XOR value in xor i , perform the following steps: S4121: Determine the XOR value xor i Leading zero count, trailing zero count and center bit of ; S4122: According to rule lead and rule trail Get the approximate leading zero count, trailing zero count and center bit; S4123: Calculation and S4124: Define condition C as: lead i ≥lead i-1 、trail i ≥trail i-1 And (lead i -lead i-1 )+(trail i -rtail i-1 )<1+ln+tn; where lead i 、trail i They are xor i Approximate leading and trailing zero counts; S4125: Determine whether situation 1 is satisfied: xor i =0: If yes, then code i Encoded as the flag bit corresponding to case 1; Otherwise, execute step S4126; S4126: Determine whether condition 2 is satisfied: xor i ≠0 and condition C is true: If so, then code i The code is: the flag bit corresponding to case 2 is followed by the approximate center bit; if not, it is judged that case 3 is satisfied, code i The encoding is: the flag bit corresponding to case three is followed by the lead bit of the ln position i , tn position rtail i and the approximate center position; S413: Approximate rule update: based on XOR value sequence<xor1,xor2,...> Each XOR value xor i Update the approximate rule for leading and trailing zeros by using the leading and trailing zero counts of lead and rule trail ; S414: Adaptive flag adjustment: Statistical XOR value sequence<xor1,xor2,...> The frequency of occurrence of each case in the encoding result; the flag bit corresponding to the case with the highest frequency is set to 1, and the flag bits corresponding to the second and third highest frequencies are set to 00 or 01 respectively.

10. The streaming error-bounded floating-point data compression method according to claim 9, characterized in that: In step S4, the compressed bit stream<code1,code2,...> Decode the encoded value in to get the XOR value xor i , decoding generates a sequence of XOR values<xor1,xor2,...> ; For a sequence of XOR values<xor1,xor2,...> XOR value in i Perform XOR operation to get approximate value a i , decoding generates an approximate value sequence<a1,a2,...> ; The approximate value sequence<a1,a2,...> Each approximation a in i Convert to restore floating point value v′ i , generate decompressed time series TS′=<v1′,v2′,...> .

Images