Streaming lossless floating-point number compression method based on segmented decompression strategy

Abstract

The invention discloses a streaming lossless floating-point number compression method based on a segmented decompression strategy, which relates to the technical field of time sequence data coding and mainly comprises the following steps of: for a leading zero part in an exclusive or result of a current floating-point number and a previous value thereof, continuing to use an optimization scheme based on leading zero stroke coding in a Chimp algorithm; and for the trailing zero part in the XOR result, multiplexing the stored trailing zero quantity when the third flag bit is 0, acquiring a correction parameter when the third flag bit is 1, and correcting the trailing zero part according to the mapping relationship between the correction parameter and the exponential bias term under the decimal system. Converting the decimal precision value stored under the erasure operation into an effective number behind a decimal point under a decimal system, and obtaining a trailing zero number; according to the leading zero number and the trailing zero number, reading of the floating point XOR operation result is completed. According to the method, the concept of boundary processing is introduced, so that the storage cost of the trailing zero-stroke coding is replaced by a small amount of calculation cost, and the compression storage cost of the floating point data is reduced.

Description

Technical Field

[0001] The invention relates to the technical field of time series data encoding, and in particular to a streaming lossless floating point number compression method based on a segmented decompression strategy. Background Art

[0002] With the rapid development of sensor technology and the Internet of Things (IoT), the application scenarios and scale of time series data continue to expand, leading to an explosive growth of this type of data. Typical applications such as satellite telemetry, industrial sensor networks, and financial transaction records will generate a large amount of time series data. For example, SpaceX's Starlink satellite can generate about 1TB of telemetry data per satellite per day. If such a large amount of floating-point time series data is transmitted and stored in its original format, it will significantly increase the network bandwidth usage and storage resource requirements, thereby reducing the overall operating efficiency of the system. Therefore, how to effectively compress time series data before data transmission and storage is a key issue in improving system performance.

[0003] At present, compression methods for time series data can be divided into general compression algorithms and compression algorithms optimized for specific scenarios. Although general compression algorithms such as LZ4 and Xz show good compression rates on a variety of data types, they fail to fully utilize the intrinsic characteristics of time series data. In particular, these algorithms usually run in batch mode. When processing large-scale streaming time series data, the computational overhead is high and the processing time is long, which limits their application in streaming scenarios.

[0004] For the compression of streaming floating-point time series data, the current algorithms can be divided into lossy compression and lossless compression. Lossy compression algorithms abandon part of the data precision in exchange for a higher compression rate during the compression process, but this method will cause data integrity to be damaged. Therefore, it is not suitable for key application scenarios such as scientific computing, data management, and network transmission that require high data accuracy and integrity. Lossless compression algorithms can compress without losing data integrity and are more suitable for scenarios such as scientific computing and data management. Common lossless compression algorithms such as Gorilla, Chimp, and Elf / Elf+ are usually based on floating-point XOR operations to reduce data redundancy. The Gorilla algorithm records the leading and trailing zeros in the XOR result through run length encoding (RLE), but the efficiency is not good when there are few leading and trailing zeros. On this basis, Chimp optimizes the storage method of leading zeros and introduces a dynamic trailing zero encoding strategy to improve the compression effect. Elf / Elf+ further introduces a reversible erase operation to preprocess the data into a form containing a large number of trailing zeros, thereby significantly improving the overall compression efficiency.

[0005] In summary, the existing streaming floating-point time series data compression methods in the field of lossless compression are mainly based on XOR operations, and the compression rate is optimized by run-length encoding of leading and trailing zeros. Although these algorithms have shown good performance in practical applications, their compression efficiency is often limited by the characteristics of specific data distribution and is difficult to fully adapt to various complex scenarios. Therefore, further optimizing and innovating streaming lossless compression algorithms has important technical and application value. Summary of the invention

[0006] Based on the problems existing in the existing streaming floating-point time series data compression method in lossless compression, the present invention proposes a streaming lossless floating-point number compression method based on a segmented decompression strategy, comprising the steps of: S1: Get the XOR result of the current floating point time series data point and the adjacent previous data point; S2: Read the first flag bit of the operation result. When the flag bit is 0, reuse the stored number of leading zeros and directly enter step S4. When the flag bit is 1, enter step S3. S3: Continue to read the second flag bit. When the flag bit is 0, determine that the value of the current data point is the same as the value of the adjacent previous data point. Return to step S1 and obtain the next floating point time series data. When the flag bit is 1, read the next 3 bits of data to parse the number of leading zeros. S4: Continue to read the third flag bit, reuse the stored number of trailing zeros and directly enter step S6 when the flag bit is 0, and enter step S5 when the flag bit is 1; S5: Obtain a correction parameter according to the decoding of the first 12 bits of data, and convert the decimal precision value stored in the erase operation into a valid number after the decimal point according to the mapping relationship between the correction parameter and the decimal exponent bias term, and determine the number of trailing zeros; S6: According to the obtained number of leading zeros and the number of trailing zeros, the reading of the floating-point XOR operation result is completed.

[0007] Furthermore, in the step S2, the number of leading zeros stored refers to the number of leading zeros of the previous data point, and in the step S4, the number of trailing zeros stored refers to the number of trailing zeros of the previous data point.

[0008] Furthermore, in the step S3, when the flag bit is 1, the number of leading zeros is obtained based on the Chimp leading zero optimization strategy by continuing to read the next 3 bits of data.

[0009] Furthermore, in the step S5, an inverse XOR operation is performed based on the floating-point XOR operation result and the first 12 bits of the floating-point time series data point adjacent to the previous data point to obtain a correction parameter, and the correction parameter is an exponential bias item under binary.

[0010] Furthermore, the mapping relationship between the binary exponential bias term and the decimal exponential bias term is expressed as the following formula: In the formula, is the decimal exponential bias term, is the approximate value of the exponential bias term in decimal system, and E is the correction parameter.

[0011] Furthermore, when the decimal value corresponding to the current floating-point time series data point is greater than or equal to 1, The number of valid digits to the left of the decimal point in decimal representation. When the decimal value corresponding to the current floating-point time series data point is less than 1, The opposite of the number of zeros to the right of the decimal point to the first significant digit in a decimal representation.

[0012] Furthermore, the decimal precision value stored in the erase operation is converted to the significant digits after the decimal point in the decimal system and expressed as follows: In the formula, The number of significant digits after the decimal point in decimal system. It is the approximate value of the significant digits after the decimal point in decimal system. The precision value of the current floating-point time series data in decimal representation.

[0013] Furthermore, the determination of the number of trailing zeros is expressed as the following formula: In the formula, is the approximate position of the trailing zero boundary subscript, Convert decimal numbers The number of trailing zeros when converting to binary.

[0014] Furthermore, the precision value of the current floating-point time series data in decimal representation Use 4 bits for storage.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention discloses a streaming lossless floating-point number compression method based on a segmented decompression strategy. Based on the erasure operation, the concept of boundary processing is introduced so that the storage cost of trailing zero run length encoding is replaced by a small amount of computational cost, thereby reducing the compression storage cost of floating-point data. (2) It is suitable for processing streaming time series data without losing any information after processing, thus being able to be used in real-time scientific computing and analysis scenarios; (3) Under the XOR-based differential encoding strategy, the variable-length storage mode is optimized. The traditional trailing zero run encoding is replaced by a calculation method based on erase operation, and the number of trailing zeros is dynamically determined. This significantly improves the compression performance of streaming floating-point time series data without increasing the time complexity of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A step diagram of a streaming lossless floating point number compression method based on a segmented decompression strategy; Figure 2 It is a schematic diagram of the left and right boundary processing of the XOR result; Figure 3 A schematic diagram of data for the right boundary processing of the XOR result; Figure 4 Schematic diagram of the decoding calculation of the XOR result. DETAILED DESCRIPTION

[0017] The following are specific embodiments of the present invention and the accompanying drawings to further describe the technical solution of the present invention, but the present invention is not limited to these embodiments.

[0018] In order to facilitate the understanding of the overall concept of the present invention, we first explain the erasing operation of the Elf / Elf+ algorithm in detail. Specifically, the erasing operation will be based on the number of digits after the decimal point of the floating point number in decimal representation ( ) and the true value (E) of the exponent part in the binary floating-point format in the decimal system to determine the number of bits that can be erased, and fill (erase) the tail part of the floating-point number with zeros according to specific rules to reduce the number of meaningful bits in the binary representation of the floating-point number.

[0019] In the erasure operation of the Elf / Elf+ algorithm, the "mantissa prefix search" is the core part, which is used to determine the number of bits that can be erased at the end of the floating-point number. This method can improve the compression efficiency by reducing the number of trailing zeros of the valid digits while ensuring data integrity. To help better understand the erasure operation, we first need to know a theorem: given a double-precision value and its number of decimal places and binary format For a double-precision floating point number, its binary format is Each digit after the decimal point in corresponds to the mantissa in the IEEE754 standard format. The subscript conversion between the two differs by a value of E, i.e. .in, Represents the bit subscript position of the mantissa in the IEEE75 standard format. Represents binary format The subscript position of the decimal point in . Therefore, we can finally get: (1) (2) In the formula, To convert a decimal number The number of trailing zeros when converted to binary. E is the correction parameter used to adjust the position of the trailing zero boundary in the mantissa part of the IEEE754 standard format. is the trailing zero boundary subscript position.

[0020] This formula helps us find data points quickly The best mantissa prefix position after erasure Once the position of the mantissa prefix is ​​determined, the number of significant digits in the floating point number can be reduced by removing the trailing zeros, thereby achieving compression.

[0021] Although compression methods such as the Elf / Elf+ algorithm have shown good performance in practical applications, their compression efficiency is often limited by the characteristics of specific data distribution and is difficult to fully adapt to various complex scenarios. Therefore, based on the Elf / Elf+ algorithm, the present invention further improves it, such as Figure 1 As shown, the present invention proposes a streaming lossless floating point number compression method based on a segmented decompression strategy, comprising the steps of: S1: Get the XOR result of the current floating point time series data point and the adjacent previous data point; S2: Read the first flag bit of the operation result. When the flag bit is 0, reuse the stored number of leading zeros and directly enter step S4. When the flag bit is 1, enter step S3. S3: Continue to read the second flag bit. When the flag bit is 0, determine that the value of the current data point is the same as the value of the adjacent previous data point. Return to step S1 and obtain the next floating point time series data. When the flag bit is 1, read the next 3 bits of data to parse the number of leading zeros. S4: Continue to read the third flag bit, reuse the stored number of trailing zeros and directly enter step S6 when the flag bit is 0, and enter step S5 when the flag bit is 1; S5: Obtain a correction parameter according to the decoding of the first 12 bits of data, and convert the decimal precision value stored in the erase operation into a valid number after the decimal point according to the mapping relationship between the correction parameter and the decimal exponent bias term, and determine the number of trailing zeros; S6: According to the obtained number of leading zeros and the number of trailing zeros, the reading of the floating-point XOR operation result is completed.

[0022] Before the technical points of the present invention are described in detail, the form of decimal numbers in binary representation is first introduced in depth, which can be divided into two categories: approximate representation type (Approximate Type) and exact representation type (PreciseType): Approximate representation type (Approximate Type) The decimal number of the approximate representation type cannot be accurately represented in the binary floating point representation (binary representation is infinite), and it needs to be approximated with infinite bits. In actual use, the target decimal number is rounded and truncated according to the accuracy requirements. It usually does not have a large number of trailing zeros in its binary floating point representation. Precise representation type (Precise Type) The decimal number of the exact representation type can be accurately represented by a binary floating point number (binary representation is finite), and it usually has a large number of trailing zeros in its binary floating point representation.

[0023] In the research of the present invention, it is found that if the decimal number is represented in binary floating point, a higher precision ( , The precision value of floating-point time series data in decimal representation is required to use more bits to maintain its representation precision, resulting in very limited compression space, and even possible compression negative benefits (such as Gorilla and Elf\Elf+). Therefore, the present invention proposes an XOR-based delta encoding strategy based on a variable-length storage mode. The XOR-based delta encoding strategy retains the original information of the first data point and stores subsequent data points in the same format as the previous data point ( ) is used to encode and store the result of XOR. Our scheme is based on the erase operation in the Elf / Elf+ algorithm, and introduces the concept of boundary processing so that the storage cost of the run length encoding (RLE) of trailing zeros is replaced by a small amount of computational cost.

[0024] As we understand it, the core of the erasure operation is to convert the floating-point number of the approximate representation type (Approximate Type) into the precise representation type (Precise Type) so that the data point has a large number of trailing zeros, thereby improving the compression efficiency. Specifically, the erasure operation adjusts the effective bit structure of the floating-point number so that each data point produces more trailing zeros in the binary representation, thereby indirectly reducing the number of effective bits that need to be stored in the XOR result. At the same time, after the erasure operation, the right boundary of each data point can be determined, and what we do is replace the run length encoding (RLE) of the trailing zeros with the right boundary calculation.

[0025] In the difference encoding strategy, only the information of the first data point will be retained, and the remaining stored data are the XOR results of two adjacent numbers. The processing of the XOR result will result in the processing of the left and right boundaries. Our technical solution is shown in Figure 2 , including left and right boundary processing of the XOR result.

[0026] First, we need to determine the left boundary (leading zeros): by reading the first flag bit, if the flag bit is 0 (write 0), it means that the number of leading zeros of the current data point is the same as the number of leading zeros of the previous data point ( ), reuse the stored leading zero count. If the flag is 1 (write1), it means that the number of leading zeros of the current data point is different from that of the previous data point ( ), continue to read the second flag bit. When the second flag bit is still 0, it means that the current value is the same as the previous value ( ), at this time, there is no need to continue to execute the subsequent steps, directly record the result that the current value is equal to the previous value (record the case of case=10), and return in advance to execute the processing of the next floating-point time series. When the second flag bit is 1, it means that the current value is different from the previous value ( ) and cannot be reused, so it is necessary to continue reading the next 3 bits of data (#lead(3bits)) and obtain the number of leading zeros based on the Chimp leading zero optimization strategy.

[0027] And for the right boundary of the XOR result (trailing zeros), we use Figure 3 As an example, combining several groups of numbers Figure 2 , it can be seen that the right boundary of the XOR result is often or So: When the XOR result is aligned with When the right edge of ), set the third flag of the XOR result to 0 (write 0), which means that the right boundary of the XOR result can be directly parsed to complete the decoding under variable-length storage; if the XOR result is When the right edge of ), set the third flag bit of the XOR result to 1 (write 1), and then use the calculation method we proposed to obtain the right boundary position. Among them, if the representation error between the decimal true value and the true value represented by the IEEE 754 binary floating point is 0 (Delta=0), that is, the fourth flag bit is set to 0, and the true value is the Precise Type mentioned above; if the error is not 0 (Delta≠0), set the fourth flag bit to 1, and the true value is the Approximate Type. Finally, the number of center bits after the leading zero and the trailing zero are determined (the meaningful bit data between the leading zero and the trailing zero) is obtained. By determining the number of leading zeros and the trailing zero, it is determined how many bits are read before stopping, that is, the number of center bits, thereby realizing dynamic reading under variable-length storage. The specific formula is expressed as follows: (3) (4) (5) In the formula, is the decimal exponential bias term, is the approximate value of the exponential bias term in decimal system, The number of significant digits after the decimal point in decimal system. It is the approximate value of the significant digits after the decimal point in decimal system. It is the precision value of the current floating point time series data in decimal representation. is the approximate position of the trailing zero boundary subscript, Convert decimal numbers The number of trailing zeros when converting to binary.

[0028] The principle of this group of formulas is that: since the decoding process requires and E to determine the right boundary position, and the original storage is Therefore, we propose a segmented decompression strategy that is completed in two stages: First, the first 12 bits are decoded to obtain Then, the right boundary subscript position of the current XOR result is determined by calculation to obtain the number of remaining bits to be read, and the decoding of the current XOR result is completed. The entire calculation process is shown in Figure 4 (In the figure, s is the sign bit, e1 to e 11 is the exponent bit, m1 to m 52 First, we obtain the correction parameter E, which is the exponential bias term in binary, based on the first 12 bits of data (the first bit is used to determine the positive or negative). Then, we propose The definition of and The conversion relationship is: (6) Among them, when hour, Indicates the number of significant digits to the left of the decimal point in a decimal representation; when hour, The opposite of the number of zeros to the right of the decimal point to the first significant digit in a decimal representation.

[0029] On this basis, we proposed and The approximate mapping relationship between arrive The mapping relationship is as follows: (7) This mapping relationship comes from the following derivation. According to scientific notation, the decimal value is expressed as ,in, , For binary floating point representation ,in .because and The range of n and E is different, and the change patterns of n and E are inconsistent. An approximate mapping of : (8) (9) (10) Then there is and The approximate mapping relationship between them. Through this mapping relationship, the parameters required in the decoding process can be quickly determined , so as to approximately obtain the right boundary subscript position Decoding is complete.

[0030] For further optimization, consider double-precision floating point numbers The maximum value can reach 324, and The maximum value is only about 16. Therefore, in order to save storage space, 4 bits are used to store More efficient.

[0031] In summary, the streaming lossless floating-point number compression method based on the segmented decompression strategy described in the present invention is based on the erasure operation and introduces the concept of boundary processing so that the storage cost of the trailing zero run encoding is replaced by a small amount of computing cost, thereby reducing the compression storage cost of the floating-point data.

[0032] It is suitable for processing streaming time series data without losing any information after processing, so it can be used in real-time scientific computing and analysis scenarios. Under the XOR-based difference encoding strategy, the variable-length storage mode is optimized, and the traditional trailing zero run encoding is replaced with a calculation method based on erase operation, and the number of trailing zeros is dynamically determined, thereby significantly improving the compression performance of streaming floating-point time series data without increasing the time complexity of the algorithm.

[0033] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0034] In addition, in the present invention, descriptions such as "first", "second", "one", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0035] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

Claims

1. A streaming lossless floating point number compression method based on a segmented decompression strategy, characterized in that: Includes steps: S1: Get the XOR result of the current floating point time series data point and the adjacent previous data point; S2: Read the first flag bit of the operation result. When the flag bit is 0, reuse the stored number of leading zeros and directly enter step S4. When the flag bit is 1, enter step S3. S3: Continue to read the second flag bit. When the flag bit is 0, determine that the value of the current data point is the same as the value of the adjacent previous data point. Return to step S1 and obtain the next floating point time series data. When the flag bit is 1, read the next 3 bits of data to parse the number of leading zeros. S4: Continue to read the third flag bit, reuse the stored number of trailing zeros and directly enter step S6 when the flag bit is 0, and enter step S5 when the flag bit is 1; S5: Obtain a correction parameter according to the decoding of the first 12 bits of data, and convert the decimal precision value stored in the erase operation into a valid number after the decimal point according to the mapping relationship between the correction parameter and the decimal exponent bias term, and determine the number of trailing zeros; S6: According to the obtained number of leading zeros and the number of trailing zeros, the reading of the floating-point XOR operation result is completed.

2. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 1, characterized in that: In the step S2, the number of leading zeros stored refers to the number of leading zeros of the previous data point, and in the step S4, the number of trailing zeros stored refers to the number of trailing zeros of the previous data point.

3. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 1, characterized in that: In the step S3, when the flag bit is 1, the number of leading zeros is obtained based on the Chimp leading zero optimization strategy by continuing to read the next 3 bits of data.

4. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 1, characterized in that: In the step S5, an XOR inverse operation is performed based on the floating-point XOR operation result and the first 12 bits of the floating-point time series data point adjacent to the previous data point to obtain a correction parameter, wherein the correction parameter is an exponential bias item in binary.

5. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 4, characterized in that: The mapping relationship between the binary exponential bias term and the decimal exponential bias term is expressed as the following formula: In the formula, is the decimal exponential bias term, is the approximate value of the exponential bias term in decimal system, and E is the correction parameter.

6. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 5, characterized in that: When the decimal value corresponding to the current floating-point time series data point is greater than or equal to 1, The number of valid digits to the left of the decimal point in decimal representation. When the decimal value corresponding to the current floating-point time series data point is less than 1, The opposite of the number of zeros to the right of the decimal point to the first significant digit in the decimal representation.

7. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 5, characterized in that: The decimal precision value stored in the erase operation is converted to the significant number after the decimal point in the decimal system as shown in the following formula: In the formula, The number of significant digits after the decimal point in decimal system. It is the approximate value of the significant digits after the decimal point in decimal system. The precision value of the current floating-point time series data in decimal representation.

8. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 7, characterized in that: The determination of the number of trailing zeros is expressed as the following formula: In the formula, is the approximate position of the trailing zero boundary subscript, Convert decimal numbers The number of trailing zeros when converted to binary.

9. A streaming lossless floating point number compression method based on a segmented decompression strategy as claimed in claim 7, characterized in that: The precision value of the current floating-point time series data in decimal representation Use 4 bits for storage.

Images