Lossy compression method, decompression method and device for time-series floating-point data

Abstract

The embodiment of the application discloses a lossy compression method, a decompression method and equipment for time sequence floating point data, which are used for improving the data compression rate and the decompression speed of the time sequence floating point data. The method comprises the following steps: acquiring a floating point array, the floating point array comprising a plurality of floating point data arranged in time sequence; sequentially determining other floating point data in the floating point array as target floating point data, and determining a previous floating point data of the target floating point data from the floating point array; determining a difference between the target floating point data and the previous floating point data as a difference data of the target floating point data; performing quantization processing on the difference data based on a preset error boundary to obtain quantized difference data corresponding to the target floating point data, wherein a difference between lossy data corresponding to the target floating point data and the target floating point data is not greater than the preset error boundary, and the lossy data corresponding to the target floating point data is obtained by performing inverse quantization processing on the quantized difference data corresponding to the target floating point data; and performing compression processing on each quantized difference data.

Description

Technical Field

[0001] This application relates to the field of data compression, and more particularly to lossy compression methods, decompression methods, and devices for time-series floating-point data. Background Technology

[0002] With the increasingly widespread application of time-series data, its volume is growing at an astonishing rate, making the design of compressors for time-series data a hot research topic. Since floating-point numbers are one of the most important data types in time-series data, numerous compressors for floating-point time-series data have been proposed, many of which employ lossy compression. Simultaneously, the need for efficient processing of time-series data has spurred the development of time-series databases, making them the primary means of storing time-series data. Therefore, time-series data compressors used in time-series databases can be utilized to the fullest extent, profoundly impacting the storage, transmission, and processing of time-series data. However, time-series database scenarios place additional requirements on compressors, making most time-series data compressors unsuitable for database environments.

[0003] In the existing technical solutions, there is a compressor for multi-dimensional data. Its workflow is roughly as follows: (1) make predictions based on adjacent data in all dimensions; (2) quantify the difference between the predicted value and the true value; and (3) encode this difference using Huffman coding technology.

[0004] The aforementioned technical solutions require prediction via a predictor during both compression and decompression, significantly reducing the decompression speed. Furthermore, the compression ratios offered by existing solutions are quite limited. Summary of the Invention

[0005] This application provides a lossy compression method, decompression method, and apparatus for time-series floating-point data, which can improve the data compression ratio and decompression speed of time-series floating-point data.

[0006] The first aspect of this application provides a lossy compression method for time-series floating-point data, including:

[0007] Obtain a floating-point array, wherein the floating-point array comprises multiple floating-point data arranged in time sequence;

[0008] The floating-point data in the floating-point array, except for the first floating-point data, are sequentially determined as the target floating-point data, and the previous floating-point data of the target floating-point data is determined from the floating-point array;

[0009] The difference between the target floating-point data and the previous floating-point data is determined as the difference data of the target floating-point data;

[0010] The difference data is quantized based on a preset error bound to obtain the quantized difference data corresponding to the target floating-point data. The difference between the lossy data corresponding to the target floating-point data and the target floating-point data is not greater than the preset error bound. The lossy data corresponding to the target floating-point data is obtained by performing inverse quantization processing on the quantized difference data corresponding to the target floating-point data.

[0011] Each of the quantization difference data is compressed.

[0012] In one specific implementation, the step of quantizing the difference data based on a preset error bound to obtain the quantized difference data corresponding to the target floating-point data includes:

[0013] The quotient of the difference data and the preset error boundary is rounded to obtain the initial quantization difference data corresponding to the target floating-point data;

[0014] If the initial quantization difference data is greater than a preset bit number threshold, then the largest data that is not greater than the bit number threshold is determined as the quantization difference data corresponding to the target floating-point data, and the initial quantization difference data is stored at the tail of the outlier queue.

[0015] If the initial quantization difference data is not greater than the number of bits threshold, then the initial quantization difference data is determined as the quantization difference data corresponding to the target floating-point data.

[0016] In one specific implementation, the compression processing of each of the quantization difference data includes:

[0017] Construct a quantization difference array containing each quantization difference data according to the sorting of the corresponding floating-point data in the floating-point array;

[0018] Construct the Huffman tree corresponding to the quantization difference array;

[0019] A coding length table corresponding to the Huffman tree is determined. The coding length table includes a symbol column and a code length column. The symbol column arranges each symbol in the Huffman tree in sequence. The order of the symbols in the symbol column is consistent with the lexicographical order of the Huffman codes corresponding to each symbol in the Huffman tree. The code length column includes the coding length corresponding to each symbol in the Huffman tree in sequence. The coding length of each symbol is the tree height of each symbol in the Huffman tree.

[0020] The code length column and the symbol column are compressed and stored respectively.

[0021] In one specific implementation, constructing the Huffman tree corresponding to the quantization difference array includes:

[0022] A first frequency table is constructed based on the quantization difference array. The first frequency table contains each symbol in the quantization difference array and the frequency of each symbol in the quantization difference array.

[0023] Construct a rare symbol list containing rare symbols, wherein the order of each rare symbol in the rare symbol list is consistent with the order of occurrence of each rare symbol in the quantization difference array, and the rare symbol is a symbol whose frequency in the first frequency table is lower than a preset frequency threshold;

[0024] Each rare symbol in the quantization difference array is replaced with a preset symbol to obtain a new quantization difference array, wherein the preset symbol is different from any symbol in the quantization difference array;

[0025] A second frequency table is constructed based on the new quantization difference array. The second frequency table contains each symbol in the new quantization difference array and the frequency of each symbol in the new quantization difference array.

[0026] The Huffman tree is constructed based on the second frequency table.

[0027] In one specific implementation, compressing and storing the symbol column includes:

[0028] Based on a preset variable-length integer group classification formula, a variety of mapping relationships required for storing the symbol column are determined, and each mapping relationship includes a flag bit and the data bit length corresponding to the flag bit;

[0029] Determine the number of bits required for each symbol in the symbol column, and determine the mapping relationship with the minimum data bit length (not less than) from the multiple mapping relationships, which is the target mapping relationship for each symbol;

[0030] Based on the target mapping relationship corresponding to each symbol, compressed data corresponding to each symbol is constructed. The compressed data includes the flag bit of the target mapping relationship corresponding to each symbol and the data bit of each symbol. The bit length of the data bit of each symbol is equal to the bit length of the data bit of the target mapping relationship corresponding to each symbol.

[0031] Construct a symbol compression array containing each compressed data according to the order of the corresponding symbols in the symbol column.

[0032] In one specific implementation, each symbol in the symbol column is arranged in ascending order according to its corresponding code length, and the compression and storage of the code length column includes:

[0033] The code length column is compressed using run-length encoding, and the compressed code length column is stored.

[0034] A second aspect of this application provides a decompression method for time-series floating-point data, including:

[0035] Obtain compressed data, wherein the compressed data is compressed according to the method described in any one of claims 1-6;

[0036] The compressed data is decompressed using Huffman decoding technology to obtain a quantization difference array;

[0037] Based on a preset error boundary, each quantization difference data in the quantization difference array is sequentially dequantized to obtain the lossy difference data corresponding to each quantization difference data.

[0038] Based on the sorting of the corresponding quantization difference data in the quantization difference array, construct a lossy difference array containing each lossy difference data;

[0039] Based on the first floating-point data in the floating-point array and the loss difference before each loss difference in the loss difference array, the lossy data corresponding to each loss difference is calculated, and the difference between each lossy data and the floating-point data corresponding to each lossy data is not greater than the preset error boundary.

[0040] A third aspect of this application provides a computer device, including:

[0041] An acquisition unit is used to acquire a floating-point array, wherein the floating-point array includes multiple floating-point data arranged in a time sequence;

[0042] The determining unit is used to sequentially determine the other floating-point data in the floating-point array except for the first floating-point data as the target floating-point data, and to determine the previous floating-point data of the target floating-point data from the floating-point array;

[0043] The determining unit is further configured to determine the difference between the target floating-point data and the previous floating-point data as the difference data of the target floating-point data;

[0044] The quantization unit is used to quantize the difference data based on a preset error boundary to obtain the quantized difference data corresponding to the target floating-point data. The difference between the lossy data corresponding to the target floating-point data and the target floating-point data is not greater than the preset error boundary. The lossy data corresponding to the target floating-point data is obtained by performing inverse quantization processing on the quantized difference data corresponding to the target floating-point data.

[0045] A compression unit is used to compress each of the quantization difference data.

[0046] In one specific implementation, the quantization unit is used to round down the quotient of the difference data and the preset error boundary to obtain the initial quantized difference data corresponding to the target floating-point data.

[0047] If the initial quantization difference data is greater than a preset bit number threshold, then the largest data that is not greater than the bit number threshold is determined as the quantization difference data corresponding to the target floating-point data, and the initial quantization difference data is stored at the tail of the outlier queue.

[0048] If the initial quantization difference data is not greater than the number of bits threshold, then the initial quantization difference data is determined as the quantization difference data corresponding to the target floating-point data.

[0049] In one specific implementation, the compression unit is specifically used to construct a quantization difference array containing each quantization difference data according to the sorting of the corresponding floating-point data in the floating-point array;

[0050] Construct the Huffman tree corresponding to the quantization difference array;

[0051] A coding length table corresponding to the Huffman tree is determined. The coding length table includes a symbol column and a code length column. The symbol column arranges each symbol in the Huffman tree in sequence. The order of the symbols in the symbol column is consistent with the lexicographical order of the Huffman codes corresponding to each symbol in the Huffman tree. The code length column includes the coding length corresponding to each symbol in the Huffman tree in sequence. The coding length of each symbol is the tree height of each symbol in the Huffman tree.

[0052] The code length column and the symbol column are compressed and stored respectively.

[0053] In one specific implementation, the compression unit is specifically used to construct a first frequency table based on the quantization difference array. The first frequency table includes each symbol in the quantization difference array and the frequency of each symbol appearing in the quantization difference array.

[0054] Construct a rare symbol list containing rare symbols, wherein the order of each rare symbol in the rare symbol list is consistent with the order of occurrence of each rare symbol in the quantization difference array, and the rare symbol is a symbol whose frequency in the first frequency table is lower than a preset frequency threshold;

[0055] Each rare symbol in the quantization difference array is replaced with a preset symbol to obtain a new quantization difference array, wherein the preset symbol is different from any symbol in the quantization difference array;

[0056] A second frequency table is constructed based on the new quantization difference array. The second frequency table contains each symbol in the new quantization difference array and the frequency of each symbol in the new quantization difference array.

[0057] The Huffman tree is constructed based on the second frequency table.

[0058] In one specific implementation, the compression unit is specifically used to determine multiple mapping relationships required for storing the symbol column based on a preset variable-length integer group classification formula, wherein each mapping relationship includes a flag bit and the data bit length corresponding to the flag bit;

[0059] Determine the number of bits required for each symbol in the symbol column, and determine the mapping relationship with the minimum data bit length (not less than) from the multiple mapping relationships, which is the target mapping relationship for each symbol;

[0060] Based on the target mapping relationship corresponding to each symbol, compressed data corresponding to each symbol is constructed. The compressed data includes the flag bit of the target mapping relationship corresponding to each symbol and the data bit of each symbol. The bit length of the data bit of each symbol is equal to the bit length of the data bit of the target mapping relationship corresponding to each symbol.

[0061] Construct a symbol compression array containing each compressed data according to the order of the corresponding symbols in the symbol column.

[0062] In one specific implementation, each symbol in the symbol column is arranged in ascending order according to its corresponding code length. The compression unit is specifically used to compress the code length column using run-length encoding and to store the compressed code length column.

[0063] A fourth aspect of this application provides a computer device, including:

[0064] A decompression unit is used to decompress compressed data to obtain a quantization difference array, wherein the compressed data is compressed according to the method described in any of the preceding first aspects;

[0065] The quantization unit performs dequantization processing on each quantization difference data in the quantization difference array sequentially based on a preset error boundary to obtain the lossy difference data corresponding to each quantization difference data.

[0066] The construction unit constructs a lossy difference array containing each lossy difference data according to the sorting of the corresponding quantization difference data in the quantization difference array;

[0067] The calculation unit is used to calculate the lossy data corresponding to each lossy difference based on the first floating-point data in the floating-point array and the lossy difference before each lossy difference in the lossy difference array, wherein the difference between each lossy data and the floating-point data corresponding to each lossy data is not greater than the preset error boundary.

[0068] A fifth aspect of this application provides a computer device, including:

[0069] Central processing unit, memory, and input / output interfaces;

[0070] The memory is either a short-term storage memory or a persistent storage memory;

[0071] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the method described in the first aspect.

[0072] A sixth aspect of this application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the method described in the first aspect.

[0073] A seventh aspect of this application provides a computer storage medium storing instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect.

[0074] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: A floating-point array is obtained, comprising multiple floating-point data arranged in time sequence; the other floating-point data in the floating-point array, excluding the first floating-point data, are sequentially determined as target floating-point data, and the preceding floating-point data of the target floating-point data is determined from the floating-point array; the difference between the target floating-point data and the preceding floating-point data is determined as the difference data of the target floating-point data; the difference data is quantized based on a preset error bound to obtain the quantized difference data corresponding to the target floating-point data, wherein the difference between the lossy data corresponding to the target floating-point data and the target floating-point data is not greater than the preset error bound; each quantized difference data is compressed. Based on the above, it can be seen that the method of calculating the difference in this application is to directly subtract the current data and the migrated data, without the need for a predictor, reducing the computational cost in the data compression process and improving the data decompression speed. Attached Figure Description

[0075] Figure 1 This is a schematic flowchart of a lossy compression method disclosed in an embodiment of this application;

[0076] Figure 2 This is a schematic diagram of the workflow of the lossy encoder disclosed in the embodiments of this application;

[0077] Figure 3 This is a schematic diagram of a process for pruning Huffman trees as disclosed in an embodiment of this application.

[0078] Figure 4 This is an example diagram of a variable-length integer group disclosed in an embodiment of this application;

[0079] Figure 5 This is a schematic diagram of the structure of a computer device disclosed in an embodiment of this application;

[0080] Figure 6 This is another structural schematic diagram of the computer device disclosed in the embodiments of this application;

[0081] Figure 7 This is another schematic diagram of the computer device disclosed in the embodiments of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0083] Existing compression schemes for time-series data suffer from the following technical problems: 1. Low compression ratio at high precision. Time-series databases require high compression precision, and current lossy compressors achieve low compression ratios to meet these precision requirements, while lossless compressors often have even lower compression ratios. 2. Slow decompression speed. Many time-series data compressors do not consider the needs of databases, focusing only on compression ratios while neglecting decompression speeds, which are crucial for databases. 3. Unsuitable for compressing small amounts of data. Retrieving partial data from compressed data requires decompressing all data. Therefore, to avoid decompressing too much unnecessary data when responding to queries, time-series databases divide the data into small segments and compress them separately. However, many compressors are designed to compress large amounts of data at once, resulting in a significant decrease in compression ratios and compression / decompression speeds when compressing small amounts of data.

[0084] To address the aforementioned issues, embodiments of this application provide a lossy compression method, decompression method, and apparatus for time-series floating-point data, which improve the data compression ratio and decompression speed of time-series floating-point data.

[0085] Please see Figure 1 This application provides a lossy compression method for time-series floating-point data, comprising the following steps:

[0086] 101. Obtain a floating-point array, which contains multiple floating-point data arranged in time sequence.

[0087] Generally, time-series databases are the primary means of storing time-series data. Therefore, a floating-point array can be a set of floating-point data with a time-series relationship obtained from a time-series database, or a set of floating-point data with a time-series relationship obtained through other methods.

[0088] 102. Sequentially determine the other floating-point data in the floating-point array except for the first floating-point data as the target floating-point data, and determine the floating-point data preceding the target floating-point data from the floating-point array.

[0089] Through testing and analysis of time series data, it was found that the difference between two adjacent time series data is more suitable for differential processing. Therefore, the target floating-point data and the previous floating-point data of the target floating-point data are determined in turn from the floating-point array.

[0090] 103. The difference between the target floating-point data and the previous floating-point data is determined as the difference data of the target floating-point data.

[0091] After obtaining adjacent data in step 102, the difference between the two adjacent data is directly calculated, and the difference result is used as the difference data for the later-sequenced floating-point data (i.e., the target floating-point data). Generally, the difference data of the target floating-point data (corresponding to the target floating-point data) will be smaller than the target floating-point data itself. Therefore, the floating-point data is initially compressed by using the difference method.

[0092] 104. Based on the preset error bound, the difference data is quantized to obtain the quantized difference data corresponding to the target floating-point data. The difference between the lossy data corresponding to the target floating-point data and the target floating-point data is not greater than the preset error bound. The lossy data corresponding to the target floating-point data is obtained by performing inverse quantization processing on the quantized difference data corresponding to the target floating-point data.

[0093] Next, the differential data is quantized within the preset error range to obtain the corresponding quantized differential data. Furthermore, by adjusting the quantization formula, the error between the target floating-point data and the corresponding lossy data can be controlled within the preset error range. In other words, the difference between the target floating-point data and the lossy data recovered from the differential quantized data based on the target floating-point data is controlled within the preset error range. The preset error range can be configured as needed.

[0094] In some specific implementations, the quantization formula can be: Where [] represents rounding, δ n Let x be the nth floating-point data in the floating-point array. n The corresponding quantization difference data (integers), x n-1 and x n x represents the temporally adjacent floating-point data in the floating-point array. n -x n-1 For x nThe corresponding difference data, where Δ is the preset error bound. Additionally, a can be any number in [1, 2]. If a is 1, the error between the target floating-point data and the corresponding lossy data is within ±0.5Δ; if a is 2, the error between the target floating-point data and the corresponding lossy data is within ±Δ.

[0095] It should be noted that, although this is rare, the difference between two adjacent floating-point data may exceed the original number of bits. That is, the difference between two 32-bit floating-point data may result in a value greater than 32 bits. Therefore, to ensure consistent operation of subsequent storage processes and maintain compression efficiency, the abnormal difference data needs to be processed as follows: If the initial quantized difference data (i.e., the data obtained after quantizing the difference data divided by a preset error boundary) is greater than a preset bit count threshold, then the largest data not exceeding the bit count threshold is determined as the quantized difference data corresponding to the target floating-point data, and the initial quantized difference data is stored at the tail of the outlier queue; if the initial quantized difference data is not greater than the bit count threshold, then the initial quantized difference data is determined as the quantized difference data corresponding to the target floating-point data.

[0096] Specifically, if the initial quantization difference data is greater than the corresponding floating-point data, the largest data within the corresponding number of bits (e.g., the largest data that can be identified by 32 bits) is used to replace this quantization difference, and this initial quantization difference data is stored in the outlier queue for accurate recovery during decoding. For ease of decoding, each piece of data added to the outlier queue is added from the tail of the queue, and the data obtained during decoding is extracted from the front of the outlier queue. If the initial quantization difference data is not greater than the corresponding floating-point data, it indicates that the corresponding floating-point data has already been compressed to some extent; therefore, the initial quantization difference data can be directly identified as the quantization difference data corresponding to the floating-point data.

[0097] 105. Compress each quantization difference data point.

[0098] The aforementioned steps 101 to 104 have already achieved lossy compression of the floating-point data in the floating-point array, ensuring that the error between the initially compressed data (i.e., the quantization difference data) and the corresponding floating-point data is within a preset error range. Therefore, further compression processing of the quantization difference data can improve the compression ratio.

[0099] It should be noted that since the first floating-point data in the floating-point array is not compressed, the compression process in this step can be performed by only compressing the differential quantization data (in this scheme, the first floating-point data in the floating-point array is not compressed), or by concatenating the first floating-point data in the floating-point array with the quantization differential data and compressing them together. This embodiment does not limit this.

[0100] In this embodiment, the difference is calculated by directly subtracting the current data from the migrated data, without the need for a predictor. This reduces the computational cost during data compression and improves the data decompression speed.

[0101] In some specific implementations, to improve the compression ratio, this application embodiment further employs Huffman coding technology to perform further compression processing on the differential quantization data in step 105. Details are as follows: Based on the sorting of the corresponding floating-point data in the floating-point array, construct a quantization differential array containing each quantization differential data; construct a Huffman tree corresponding to the quantization differential array; determine the encoding length table corresponding to the Huffman tree, the encoding length table including a symbol column and a code length column, the symbol column sequentially arranging each symbol in the Huffman tree, the symbol sorting in the symbol column being consistent with the lexicographical order of the Huffman code corresponding to each symbol in the Huffman tree, the code length column sequentially including the encoding length corresponding to each symbol in the Huffman tree, the encoding length of each symbol being the tree height of each symbol in the Huffman tree; compress and store the code length column and the symbol column respectively.

[0102] First, the purpose of integrating the quantization difference data into a quantization difference array is to facilitate the construction of the Huffman tree in subsequent steps. The following example illustrates this: if δ... n It is floating-point data x n Given the corresponding differential quantization data and floating-point array {x1, x2, x3}, the following quantization differential array {δ1, δ2, δ3} can be constructed. Furthermore, by sampling any Huffman coding scheme to construct the Huffman tree corresponding to the quantization differential array, and using existing methods to construct the encoding length table corresponding to this Huffman tree, the encoding length table is ultimately stored instead of the tree structure, saving a significant amount of storage space without increasing decoding time.

[0103] It should be noted that the above storage method can be either to construct the Huffman tree corresponding to the quantization difference array and store the entire Huffman tree, or to construct the Huffman tree corresponding to the quantization difference array, then construct the encoding length table corresponding to the Huffman tree, and then store the encoding length table instead of storing the Huffman tree. There is no limitation here.

[0104] In a quantized difference array, many values ​​will appear repeatedly. Values ​​that appear less frequently or only once are called rare values. The conclusion that low-level compression and high-precision compression increase the proportion of rare values ​​is statistically derived. The more rare values ​​there are, the fewer the average number of repetitions per value, which means more distinct values, more leaves in the Huffman tree, a larger Huffman tree, and greater construction and storage overhead. A similar situation exists for the decoding table.

[0105] To address the aforementioned impacts, this application further proposes the following method for constructing a Huffman tree to reduce storage overhead. Specifically, the Huffman tree can be constructed as follows: A first frequency table is constructed based on the quantization difference array. The first frequency table contains each symbol in the quantization difference array and the frequency of each symbol appearing in the quantization difference array. A rare symbol list is constructed, containing rare symbols. The order of each rare symbol in the rare symbol list is consistent with the order of appearance of each rare symbol in the quantization difference array. Rare symbols are symbols whose corresponding frequencies in the first frequency table are lower than a preset frequency threshold. Each rare symbol in the quantization difference array is replaced with a preset symbol to obtain a new quantization difference array. The preset symbol is different from any symbol in the quantization difference array. A second frequency table is constructed based on the new quantization difference array. The second frequency table contains each symbol in the new quantization difference array and the frequency of each symbol appearing in the new quantization difference array. A Huffman tree is constructed based on the second frequency table.

[0106] In simple terms, this involves using pre-defined symbols that are infrequent (i.e., symbols that won't appear in the quantization difference array) to uniformly replace the rare symbols in the quantization difference array. Then, a frequency table (i.e., a second frequency table) is constructed based on the new quantization difference array after the replacement, and the final Huffman tree is built based on the second frequency table. By reducing the height of the Huffman tree using this method, the compression ratio can be effectively improved without affecting the decompression speed.

[0107] Furthermore, since the code length column and symbol column are compressed and stored separately in this embodiment, the most suitable algorithm can be used to compress the code length column and symbol column separately to maximize the compression ratio. The compression storage of the symbol column can be performed using the following steps: Based on a preset variable-length integer group classification formula, determine multiple mapping relationships required for storing the symbol column. Each mapping relationship includes a flag bit and the corresponding data bit length. Determine the number of bits required for each symbol in the symbol column, and determine the mapping relationship with the minimum data bit length (not less than) the number of bits required for each symbol from the multiple mapping relationships, as the target mapping relationship for each symbol. Based on the target mapping relationship for each symbol, construct compressed data for each symbol. The compressed data includes the flag bit of the target mapping relationship for each symbol and the data bits for each symbol. The bit length of the data bits for each symbol is equal to the data bit length of the target mapping relationship for each symbol. Construct a symbol compressed array containing each compressed data according to the order of the corresponding symbols in the symbol column.

[0108] For details, please refer to Figure 4 This application's embodiments construct a variable-length integer group based on the concept of variable-length integers, used to compress the symbol column in the table. The variable-length integer group configures the corresponding data bit length for each flag bit through a mapping relationship; that is, the number of bits of data immediately following each flag bit is determined according to the mapping relationship. In other words, the length of the data bits corresponding to each flag bit value is no longer fixed and can be arbitrarily set. This data structure decouples flag bits from bytes, allowing the mapping relationship between flag bits and data bit lengths to be freely adjusted, making it more flexible. For binary representations with insufficient data length, leading zeros can be added to make up the difference.

[0109] Furthermore, variable-length integer arrays can also eliminate the need to fix the flag bits and data bits at specific locations within each byte (e.g., Figure 4 In the diagram, each symbol's flag bit is immediately followed by its corresponding data bit (the flag bit of each symbol is connected end-to-end with the data bit of the previous symbol). Instead of being scattered across individual bytes, the flag bits of the variable-length integer are grouped together to form a unique prefix that identifies the data length. For example, if storing a symbol column {symbol 1, symbol 2, symbol 3}, it could be done using either {flag bit 1 + flag bit 2 + flag bit 3 + data bit 1 + data bit 2 + data bit 3} or {flag bit 1 + data bit 1 + flag bit 2 + data bit 2 + flag bit 3 + data bit 3}. This embodiment does not impose a limitation. Here, flag bit n represents the flag bit corresponding to symbol n, and data bit n represents the data bit corresponding to symbol n.

[0110] Additionally, it should be noted that if a list of rare symbols exists due to the aforementioned removal of rare symbols, this list will be concatenated to the symbol column in the code length table and compressed together using the compression method described above.

[0111] Building upon the aforementioned embodiments, in some specific implementations, to ensure that shorter lexicographical codes are also smaller, the code length table is sorted from smallest to largest by the code length column without violating the lexicographical order. That is, each symbol in the symbol column is arranged from smallest to largest according to its corresponding code length, and the sorted code length column is compressed using run-length encoding to further improve the compression ratio.

[0112] Similarly, for the aforementioned various compression method embodiments, the corresponding embodiments of this application also provide the following decompression method, including the following steps: decompressing the compressed data to obtain a quantization difference array, wherein the compressed data is compressed according to any of the compression methods in the aforementioned embodiments; performing inverse quantization processing on each quantization difference data in the quantization difference array based on a preset error boundary to obtain lossy difference data corresponding to each quantization difference data; constructing a lossy difference array containing each lossy difference data according to the sorting of the corresponding quantization difference data in the quantization difference array; calculating the lossy data corresponding to each lossy difference based on the first floating-point data in the floating-point array and the lossy difference before each lossy difference in the lossy difference array, wherein the difference between each lossy data and the floating-point data corresponding to each lossy data is not greater than the preset error boundary.

[0113] Specifically, similar to other lossy compression and decompression schemes, it uses a process that reverses the compression steps to achieve decompression. However, based on the quantization difference data corresponding to the same floating-point data and the aforementioned quantization difference formula, only the lossy difference corresponding to that floating-point data can be obtained, not the corresponding difference data. Therefore, it cannot decompress the floating-point data itself, but only the corresponding lossy data.

[0114] Based on the lossy compression and decompression methods for time-series floating-point data provided in the foregoing embodiments, this invention designs a lossy encoder for floating-point numbers in time-series database scenarios: Niu Dao. Niu Dao employs absolute error control and uses a simple quantization framework to achieve high-speed decompression. Figure 2 The general workflow of Niu Dao (a compression tool) is demonstrated: First, differential quantization converts floating-point numbers in the input array into integer quantized differences. Then, Huffman coding is used to further compress the quantized differences. Decompression involves first decoding the quantized differences and then calculating and restoring the decompressed data accordingly. Furthermore, to improve Niu Dao's compression ratio in scenarios with small amounts of data and high precision, techniques such as pruning, optimal variable-length integer sets, and sorting are proposed to address the bottleneck of the Huffman tree. Notably, while symbols are typically single bytes, Niu Dao uses 32-bit integers for symbol compression in its Huffman coding. A detailed introduction to Niu Dao follows.

[0115] I. High-efficiency quantitative framework

[0116] NiuDao compression generally uses a simple quantization framework for lossy compression, supporting absolute error while providing high-speed decompression. Note that quantizing the difference between temporally adjacent data is more suitable for time-series data (because it offers a higher compression ratio) than direct quantization. Therefore, the overall NiuDao compression process is: quantize the data difference → quantize the difference → perform lossless encoding on the quantized difference. The calculation formulas for the difference and quantization can be summarized as follows:

[0117]

[0118] Where, x n-1 and x n For floating-point data that are sequentially adjacent in the floating-point array, Δ is the user-defined preset error bound, and δ is the floating-point data. n This refers to the quantization difference. The specific formula for restoring data using the quantization difference during decompression is:

[0119] y n =x n-1 +2Δδ n #(Formula 2)

[0120] Among them, y n For x n The corresponding data after the loss. Substituting Equation 1 into Equation 2, we can prove that y n ∈[x n -Δ,x n +Δ], which means it meets the absolute error requirement. However, during the decompression process, x n-1 Since it is unknown, y cannot be obtained through Equation 2. n Therefore, it is necessary to change x in equation two. n-1 Replace with y n-1 (Equation 3), and directly record y0 = x0 in the compression result. Meanwhile, to ensure error accuracy, x in Equation 1... n-1 It should also be replaced with y. n-1 (Equation 4) This also means that the decompression value of the data needs to be calculated during compression.

[0121] y n =y n-1 +2Δδ n #(Form 3)

[0122]

[0123] After differential quantization converts the timing floating-point number into a 32-bit integer, Niu Dao then uses Huffman coding to compress the quantization difference. The decoding process involves first decoding the quantization difference using Huffman coding, and then restoring the data according to Equation 3. The overall framework is simple and efficient, achieving both absolute error control and fast execution speed.

[0124] It should be noted that Equation 4 is the differential quantization formula, while Equation 3 is the formula for reconstructing data using quantization differentials. Although x in Equation 4... n y n-1 Both δ and Δ are floating-point numbers, but the [] rounding ensures that δ_n is an integer. However, it does not guarantee that δ_n falls within the range of 32-bit integers. When δ is found n When the error exceeds the specified range, special handling will be performed, as detailed below: 1. The actual Δ used in the calculation will be slightly smaller than the user-set upper limit of the preset error. This is because floating-point calculations are not perfectly precise (e.g., in Python, 10 ÷ 3 = 3.3333333333333335). Using a smaller error for quantization during compression ensures that the calculation error will not cause the compression error to exceed the upper limit of the user-set preset error. 2. Although the probability is extremely small, δ n It is indeed possible that the value exceeds the representation range of a 32-bit integer. In this case, a special value (in the specific implementation, the maximum value that a 32-bit integer can represent) will be used as δ. n and x n The data is stored directly in an outlier queue without any processing. This outlier queue is also stored in the compression results. When the data restorer encounters a quantization difference with a special value, it no longer uses Equation 3 to restore the data, but instead retrieves the data directly from the outlier queue.

[0125] II. Optimizations Related to High-Precision Small-Scale Data Compression

[0126] The main impact of low-volume compression and high-precision compression on the Niu Dao framework is the increased proportion of rare values ​​in the quantization difference, leading to increased overhead related to the Huffman tree and decoding table. To address this, the following techniques are proposed: (1) Huffman tree pruning; (2) optimal variable-length integer sets; and (3) Huffman tree sorting. The optimal variable-length integer sets and Huffman tree sorting together form the Huffman tree compressor. These techniques significantly reduce the storage overhead of the Huffman tree without reducing the decompression speed, thus improving the compression ratio. A brief introduction to the uncompressed storage method of the Huffman tree will be given first, followed by an introduction to the relevant techniques.

[0127] When storing a Huffman tree, the tree structure is not preserved; instead, only a code length table (hereinafter referred to as the code length table) is stored. Each entry in the table records only one symbol (the quantization difference is a 32-bit integer, requiring four bytes of storage without compression, 4 bytes for the quantization difference) and the length of the Huffman code assigned to it (2 bytes). The entries are sorted lexicographically according to the Huffman codes of the recorded symbols (although the codes themselves are not in the table). This sorting is necessary for constructing the decoder table during decoding. Huffman tree pruning reduces the number of entries in the code length table, the optimal variable-length integer set is used to compress the symbol column in the table, and the Huffman tree sort is used to compress the code length column in the table.

[0128] 1) Please refer to Figure 3 Huffman tree pruning refers to removing some of the tallest nodes from a Huffman tree, thereby reducing the construction time and storage space required, and also shortening the maximum code length. Although the method is called Huffman tree pruning, in practice this technique is performed before constructing the Huffman tree. After counting the frequencies of the input symbols, Huffman pruning will remove symbols from the frequency table that are not greater than a certain threshold (a preset frequency threshold). Figure 2 The threshold value of 2) is recorded as a rare symbol. These rare symbols are recorded using an additional rare symbol list in the order they appear in the input, and a uniform special symbol (i.e., a preset symbol, such as...) is used. Figure 2 Replace the $) in the input and finally update the frequency table. The Huffman tree constructed using the new frequency table is then the pruned Huffman tree. During decompression, first perform the standard Huffman decompression process, then replace the special symbols in the decompression result with symbols from the rare symbol list.

[0129] 2) Optimal variable-length integer encoding is used to compress the sign column in the table (the rare sign array generated by Huffman pruning is concatenated after the sign column for compression). Traditional variable-length integers are encoded for single integers, so the encoded result of each integer is a whole byte. In variable-length integers, the flag bit is fixed in the first bit of each byte. This makes the mapping relationship between the flag bit and the data length fixed as follows: flag bit 0 => data length 7 bits; flag bit 10 => data length 14 bits; flag bit 110 => data length 21 bits... This mapping cannot be changed.

[0130] However, in Niu Dao (a Chinese encoding platform), variable-length integers are used to encode an array rather than a single number. In this case, it is only necessary to ensure that the encoded result of the array is an integer, while the encoded result of a single data point does not need to be an integer. Therefore, variable-length integer encoding is extended to a more flexible variable-length integer group encoding (such as...). Figure 3 See details. Figure 4Variable-length integers use one bit per byte to indicate whether the current byte is the last byte to represent the data length. Arrays of variable-length integers put these flag bits together to form a unique prefix that represents the data length.

[0131] It should be noted that in actual use, in order to store more floating-point data with the least amount of space, different variable-length integer array types can be determined in the following ways.

[0132] Construct a prefix-length map that maximizes the compression ratio based on the input array, and use this map to encode the data. Constructing the map requires first determining a set of lengths. Then construct the mapping: 0→b1, 10→b2, 110→b3, ..., 11...10→b n-1 ,11…11→b n The encoded bit length obtained by using this mapping to encode the data can be calculated using Equation 5:

[0133]

[0134] Where CNT(b) i+1 ,b i ) indicates that the required number of bits is greater than b i Not greater than b i+1 The number of data points (if x∈[-2) a -2 a-1 )∪[2 a-1 ,2 a Then the number of bits required to represent x is called a+1).

[0135] The length set S required to construct the optimal mapping l OUTPUT(S) needs to be enabled. l Minimize ). This problem can be solved using dynamic programming. The core of using dynamic programming is to define a subproblem with optimal substructure. Here, the subproblem SP(b) is defined as the optimal length set search problem when it is assumed that all data requiring more than b bits to represent can be represented and need to be represented by b bits. Let SOL(b) be a solution to SP(b), and Note SOL o (b) is the optimal solution for SP(b).

[0136] Given the above definitions, proving the optimal substructure is equivalent to proving that if SOL o (b α ) includes SP(b β The solution SOL′(b) β )={x∈SOL o (b α )|x α ​}, then SOL′(b β It must be SP(b) β The optimal solution, i.e. stSOL o (b β )={x∈SOL o (b α )|x β The main proof idea is as follows: Assume SP(b) α The optimal solution SOL o (α) includes SP(b) β The solution SOL′(b) β ), and SOL′(b β If ) is not the optimal solution, then another SP(b) can be constructed. α The solution SOL′(b) α ) = SOL o (b β )∪(SOL o (b α )-SOL′(b β Combining Equation 3, we can see that SOL′(b) α ) is superior to SOL o (b α The solution is the same as that of SOL. o (b α The optimal solution is contradictory, therefore SOL′(b) is derived. β This is also the optimal solution. Based on the optimal substructure, we can obtain the recursive expression, equation six:

[0137]

[0138] And based on this, the optimal length set SOL is obtained. o (b) and the corresponding prefix-length mapping.

[0139] 3) Huffman tree sorting is used to encode code length columns. Huffman tree sorting refers to starting from level 1 of the Huffman tree and progressively moving leaf nodes to the left of non-leaf nodes, resulting in a Huffman tree where the height of a left leaf node is no greater than that of any leaf node to its right (default left leaf node is 0, right leaf node is 1). Huffman tree sorting does not change the code length of each symbol, therefore it does not affect the compression ratio. Encoding with the sorted Huffman tree results in a smaller lexicographical order for shorter codes, allowing the code length table to be sorted from smallest to largest without violating the lexicographical order. Then, running-length encoding (or any other arbitrary encoding) is applied to the sorted code length column to achieve a higher compression ratio.

[0140] ​Although the code length table does not store the codes, because the symbols in it are sorted according to the lexicographical order of the codes and the code lengths are recorded, a unique Huffman tree can be reconstructed during decoding (if b is first in the code length table, it means that its lexicographical order is the smallest, each bit in the code is 0, and the code length is 3, so the code of b is determined to be 000. If e is after b in the code length table and the code length is 4, its lexicographical order should also be as small as possible (otherwise there will be a situation where there are not enough codes later), so the code is 0010, and the remaining symbols are deduced in the same way).

[0141] After Huffman sorting, the height of the leaf nodes in the tree increases sequentially from left to right, hence the term "sorting." Constructing a code length table from the sorted Huffman tree reveals that, while maintaining lexicographical order, the code lengths are also arranged in ascending order.

[0142] Since it's agreed that left child nodes are assigned code 0 and right child nodes are assigned code 1, the lexicographical order of the codes is actually the left-to-right order on the Huffman tree. And since the code length is equal to the height of the Huffman tree, as the height of the leaf nodes increases from left to right, the code length columns in the code length table are naturally sorted from smallest to largest. The Huffman tree sorting affects the specific encoding of each symbol, therefore it occurs before generating the data encoding in the workflow. In implementation, Huffman tree sorting doesn't require manipulating the Huffman tree; it simply involves storing the leaf nodes in a priority queue sorted by code length, and then assigning codes sequentially in lexicographical order.

[0143] To expand the application scenarios of lossy compression in time-series databases, this paper proposes that lossy compression should use high-precision absolute error. A simple and efficient quantization framework is designed to meet the performance requirements of databases as much as possible, and a lossy compressor, "Niu Dao," is designed based on this framework. Furthermore, considering the requirements of high-precision compression and small-scale data compression in time-series databases, a detailed analysis of data characteristics was conducted. It was found that an excessive number of rare symbols and an excessively large Huffman tree are bottlenecks to improving the compression ratio. Techniques such as Huffman tree pruning, Huffman tree sorting, and optimal variable-length integer sets, which have almost no impact on decompression speed, are proposed and applied to Niu Dao, achieving a high compression ratio.

[0144] Testing NiuDao against state-of-the-art lossy compressors such as LFZip, SZ3, and ZFP on five real-world datasets revealed that in high-precision, low-data scenarios, NiuDao achieved a compression ratio 24%-134% higher and a decompression speed 60%-360% faster than the second-best. NiuDao was also tested on the open-source time-series database InfluxDB, showing savings of 62%-83% in storage space. Furthermore, due to its sufficiently fast decoding speed, the bottleneck for query time was disk reads, resulting in a 58%-414% improvement in query throughput from the saved storage.

[0145] Please see Figure 5This application provides a computer device, including:

[0146] Acquisition unit 501 is used to acquire a floating-point array, which includes multiple floating-point data arranged in time sequence;

[0147] The determining unit 502 is used to sequentially determine the other floating-point data in the floating-point array except for the first floating-point data as the target floating-point data, and determine the previous floating-point data of the target floating-point data from the floating-point array;

[0148] The determining unit 502 is also used to determine the difference between the target floating-point data and the previous floating-point data as the difference data of the target floating-point data;

[0149] The quantization unit 503 is used to quantize the difference data based on a preset error boundary to obtain the quantized difference data corresponding to the target floating-point data. The difference between the lossy data corresponding to the target floating-point data and the target floating-point data is no greater than the preset error boundary. The lossy data corresponding to the target floating-point data is obtained by performing inverse quantization processing on the quantized difference data corresponding to the target floating-point data.

[0150] Compression unit 504 is used to compress each quantization difference data.

[0151] In one specific implementation, the quantization unit 503 is used to round down the quotient of the difference data and the preset error boundary to obtain the initial quantized difference data corresponding to the target floating-point data.

[0152] If the initial quantization difference data is greater than the preset number of bits threshold, the largest data that is not greater than the number of bits threshold will be determined as the quantization difference data corresponding to the target floating-point data, and the initial quantization difference data will be stored at the tail of the outlier queue.

[0153] If the initial quantization difference data is not greater than the number of bits threshold, then the initial quantization difference data is determined as the quantization difference data corresponding to the target floating-point data.

[0154] In one specific implementation, the compression unit 504 is specifically used to construct a quantization difference array containing each quantization difference data according to the sorting of the corresponding floating-point data in the floating-point array;

[0155] Construct the Huffman tree corresponding to the quantization difference array;

[0156] Determine the encoding length table corresponding to the Huffman tree. The encoding length table includes a symbol column and a code length column. The symbol column arranges each symbol in the Huffman tree in sequence. The order of the symbols in the symbol column is consistent with the lexicographical order of the Huffman code corresponding to each symbol in the Huffman tree. The code length column includes the encoding length corresponding to each symbol in the Huffman tree in sequence. The encoding length of each symbol is the tree height of each symbol in the Huffman tree.

[0157] Compress and store the code length column and the symbol column separately.

[0158] In one specific implementation, the compression unit 504 is specifically used to construct a first frequency table based on the quantization difference array. The first frequency table contains each symbol in the quantization difference array and the frequency of each symbol in the quantization difference array.

[0159] Construct a list of rare symbols containing rare symbols. The order of each rare symbol in the list is consistent with the order of occurrence of each rare symbol in the quantization difference array. Rare symbols are symbols whose frequencies in the first frequency table are lower than a preset frequency threshold.

[0160] Replace each rare symbol in the quantization difference array with a preset symbol to obtain a new quantization difference array. The preset symbol is different from any symbol in the quantization difference array.

[0161] A second frequency table is constructed based on the new quantization difference array. The second frequency table contains each symbol in the new quantization difference array, as well as the frequency of each symbol in the new quantization difference array in the quantization difference array.

[0162] Construct a Huffman tree based on the second frequency table.

[0163] In one specific implementation, the compression unit 504 is specifically used to determine multiple mapping relationships required for storing the symbol column based on a preset variable-length integer group classification formula. Each mapping relationship includes a flag bit and the data bit length corresponding to the flag bit.

[0164] Determine the number of bits required for each symbol in the symbol column, and determine the mapping relationship with the minimum data bit length and not less than the number of bits required for each symbol from multiple mapping relationships, and use it as the target mapping relationship for each symbol;

[0165] Based on the target mapping relationship corresponding to each symbol, compressed data corresponding to each symbol is constructed. The compressed data includes the flag bit of the target mapping relationship corresponding to each symbol and the data bits of each symbol. The bit length of the data bits of each symbol is equal to the bit length of the data bit of the target mapping relationship corresponding to each symbol.

[0166] Construct a symbol compression array containing each compressed data according to the order of the corresponding symbols in the symbol column.

[0167] In one specific implementation, each symbol in the symbol column is arranged in ascending order according to its corresponding encoding length. The compression unit 504 is specifically used to compress the code length column using run-length encoding and to store the compressed code length column.

[0168] Please see Figure 6This application provides a computer device, comprising:

[0169] The decompression unit 601 is used to decompress the compressed data to obtain a quantization difference array, wherein the compressed data is compressed according to any of the methods described in the first aspect above;

[0170] The quantization unit 602 performs inverse quantization processing on each quantization difference data in the quantization difference array sequentially based on a preset error boundary to obtain the lossy difference data corresponding to each quantization difference data.

[0171] Construction unit 603 constructs a lossy difference array containing each lossy difference data according to the sorting of the corresponding quantization difference data in the quantization difference array;

[0172] The calculation unit 604 is used to calculate the lossy data corresponding to each lossy difference based on the first floating-point data in the floating-point array and the lossy difference before each lossy difference in the lossy difference array. The difference between each lossy data and the floating-point data corresponding to each lossy data is not greater than a preset error boundary.

[0173] Figure 7 This is a schematic diagram of a computer device structure provided in an embodiment of this application. The computer device 700 may include one or more central processing units (CPUs) 701 and a memory 705, in which one or more application programs or data are stored.

[0174] The memory 705 can be volatile or persistent storage. The program stored in the memory 705 can include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the central processing unit 701 can be configured to communicate with the memory 705 and execute the series of instruction operations stored in the memory 705 on the computer device 700.

[0175] The computer device 700 may also include one or more power supplies 702, one or more wired or wireless network interfaces 703, one or more input / output interfaces 704, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0176] The central processing unit 701 can perform the aforementioned... Figures 1 to 6 The specific operations performed by the computer device in the illustrated embodiment will not be described in detail here.

[0177] It should be noted that although the steps in the flowcharts of the various embodiments are drawn sequentially according to the arrows, unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the various embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0179] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0180] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0181] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0182] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0183] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the lossy compression method and decompression method for time-series floating-point data as described above.

Claims

1. A lossy compression method for time-series floating-point data, characterized in that, include: Obtain a floating-point array, wherein the floating-point array comprises multiple floating-point data arranged in time sequence; The floating-point data in the floating-point array, except for the first floating-point data, are sequentially determined as the target floating-point data, and the previous floating-point data of the target floating-point data is determined from the floating-point array; The difference between the target floating-point data and the previous floating-point data is determined as the difference data of the target floating-point data; The differential data is quantized based on a preset error bound to obtain quantized differential data corresponding to the target floating-point data. This includes: rounding the quotient of the differential data and the preset error bound to obtain initial quantized differential data corresponding to the target floating-point data; if the initial quantized differential data is greater than a preset bit count threshold, the largest data not greater than the bit count threshold is determined as the quantized differential data corresponding to the target floating-point data, and the initial quantized differential data is stored at the tail of the outlier queue; if the initial quantized differential data is not greater than the bit count threshold, the initial quantized differential data is determined as the quantized differential data corresponding to the target floating-point data; wherein the difference between the lossy data corresponding to the target floating-point data and the target floating-point data is not greater than the preset error bound, and the lossy data corresponding to the target floating-point data is obtained by performing inverse quantization processing on the quantized differential data corresponding to the target floating-point data. Each of the quantization difference data is compressed.

2. The method according to claim 1, characterized in that, The compression process for each of the quantization difference data includes: Construct a quantization difference array containing each quantization difference data according to the sorting of the corresponding floating-point data in the floating-point array; Construct the Huffman tree corresponding to the quantization difference array; A coding length table corresponding to the Huffman tree is determined. The coding length table includes a symbol column and a code length column. The symbol column arranges each symbol in the Huffman tree in sequence. The order of the symbols in the symbol column is consistent with the lexicographical order of the Huffman codes corresponding to each symbol in the Huffman tree. The code length column includes the coding length corresponding to each symbol in the Huffman tree in sequence. The coding length of each symbol is the tree height of each symbol in the Huffman tree. The code length column and the symbol column are compressed and stored respectively.

3. The method according to claim 2, characterized in that, The construction of the Huffman tree corresponding to the quantization difference array includes: A first frequency table is constructed based on the quantization difference array. The first frequency table contains each symbol in the quantization difference array and the frequency of each symbol in the quantization difference array. Construct a rare symbol list containing rare symbols, wherein the order of each rare symbol in the rare symbol list is consistent with the order of occurrence of each rare symbol in the quantization difference array, and the rare symbol is a symbol whose frequency in the first frequency table is lower than a preset frequency threshold; Each rare symbol in the quantization difference array is replaced with a preset symbol to obtain a new quantization difference array, wherein the preset symbol is different from any symbol in the quantization difference array; A second frequency table is constructed based on the new quantization difference array. The second frequency table contains each symbol in the new quantization difference array and the frequency of each symbol in the new quantization difference array. The Huffman tree is constructed based on the second frequency table.

4. The method according to claim 2, characterized in that, The compression and storage of the symbol column includes: Based on a preset variable-length integer group classification formula, a variety of mapping relationships required for storing the symbol column are determined, and each mapping relationship includes a flag bit and the data bit length corresponding to the flag bit; Determine the number of bits required for each symbol in the symbol column, and determine the mapping relationship with the minimum data bit length (not less than) from the multiple mapping relationships, which is the target mapping relationship for each symbol; Based on the target mapping relationship corresponding to each symbol, compressed data corresponding to each symbol is constructed. The compressed data includes the flag bit of the target mapping relationship corresponding to each symbol and the data bit of each symbol. The bit length of the data bit of each symbol is equal to the bit length of the data bit of the target mapping relationship corresponding to each symbol. Construct a symbol compression array containing each compressed data according to the order of the corresponding symbols in the symbol column.

5. The method according to claim 2, characterized in that, Each symbol in the symbol column is arranged in ascending order according to its corresponding code length. The compression and storage of the code length column includes: The code length column is compressed using run-length encoding, and the compressed code length column is stored.

6. A decompression method for time-series floating-point data, characterized in that, include: The compressed data is decompressed to obtain a quantization difference array, wherein the compressed data is compressed according to the method described in any one of claims 1-5. Based on a preset error boundary, each quantization difference data in the quantization difference array is sequentially dequantized to obtain the lossy difference data corresponding to each quantization difference data. Based on the sorting of the corresponding quantization difference data in the quantization difference array, construct a lossy difference array containing each lossy difference data; Based on the first floating-point data in the floating-point array and the loss difference before each loss difference in the loss difference array, the lossy data corresponding to each loss difference is calculated, and the difference between each lossy data and the floating-point data corresponding to each lossy data is not greater than the preset error boundary.

7. A computer device, characterized in that, include: An acquisition unit is used to acquire a floating-point array, wherein the floating-point array includes multiple floating-point data arranged in a time sequence; The determining unit is used to sequentially determine the other floating-point data in the floating-point array except for the first floating-point data as the target floating-point data, and to determine the previous floating-point data of the target floating-point data from the floating-point array; The calculation unit is used to determine the difference between the target floating-point data and the previous floating-point data as the difference data of the target floating-point data; A quantization unit is used to quantize the difference data based on a preset error boundary to obtain quantized difference data corresponding to the target floating-point data. This includes: rounding the quotient of the difference data and the preset error boundary to obtain initial quantized difference data corresponding to the target floating-point data; if the initial quantized difference data is greater than a preset bit count threshold, then the largest data not greater than the bit count threshold is determined as the quantized difference data corresponding to the target floating-point data, and the initial quantized difference data is stored at the tail of an outlier queue; if the initial quantized difference data is not greater than the bit count threshold, then the initial quantized difference data is determined as the quantized difference data corresponding to the target floating-point data; wherein the difference between the lossy data corresponding to the target floating-point data and the target floating-point data is not greater than the preset error boundary, and the lossy data corresponding to the target floating-point data is obtained by performing inverse quantization processing on the quantized difference data corresponding to the target floating-point data. A compression unit is used to compress each of the quantization difference data.

8. A computer device, characterized in that, include: Central processing unit, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the lossy compression method for time-series floating-point data as described in any one of claims 1 to 5 or the decompression method for time-series floating-point data as described in claim 6.

9. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed on the computer, cause the computer to perform a lossy compression method for time-series floating-point data as described in any one of claims 1 to 5 or a decompression method for time-series floating-point data as described in claim 6.

Images