Time sequence data compression method and device, time sequence data decompression method and device, equipment and medium
By performing the exclusive-OR operation of the timing data with its previous timing data, and compressing it according to the number of leading zeros and the effective bit length in the XOR value, the problem of low compression ratio of the timing data in the prior art is solved, and more efficient data compression and storage is achieved.
Patent Information
- Application Number
- CN202411515585.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-28
- Publication Date
- 2025-02-28
AI Technical Summary
The compression of existing time-series data compression algorithms is relatively low, and the zero characteristics in time-series data cannot be fully utilized, resulting in low storage space efficiency.
By performing the XOR operation on the timing data and its previous timing data, an XOR value is obtained, and the header data is determined based on the number of leading zeros and the effective bit length in the XOR value, and the timing data is compressed based on the header data.
The compression ratio of time series data is improved, the zero in the XOR value is fully utilized, the storage space requirement is reduced, and the data processing efficiency is improved.
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of data processing technology, and in particular, relates to a time series data compression method, decompression method, device, equipment and medium. Background Art
[0002] Data compression is the process of encoding the original data with less space. It means reducing the amount of data to reduce storage space and improve its transmission, storage and processing efficiency without losing useful information. The compression ratio refers to the ratio of the original data volume to the compressed volume. The higher the compression ratio, the lower the storage space required for the data after compression, and the easier it is to output, store and process the data. Therefore, the compression ratio can be used to measure the performance of the data compression method.
[0003] At present, when compressing time series data, the compression ratio is generally low. For example, the compression ratios of existing time series data compression algorithms such as Gorilla algorithm, Chimp algorithm, Chimp128 algorithm, Elf algorithm and ALP algorithm are not ideal. Summary of the invention
[0004] In view of this, embodiments of the present application provide a time series data compression method, decompression method, device, equipment and medium, which are used to improve the compression ratio when compressing time series data.
[0005] A first aspect of an embodiment of the present application provides a time series data compression method, including:
[0006] Acquire a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order;
[0007] Performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data;
[0008] Determine the header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine the representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data;
[0009] Based on the header data, compress each of the time series data;
[0010] The header data and the compressed data corresponding to each of the time series data are combined into a compressed data array of the time series data array.
[0011] A second aspect of an embodiment of the present application provides a time series data decompression method, including:
[0012] Acquire a compressed data array of a time series data array, wherein the compressed data array includes header data and compressed data of a plurality of time series data, wherein the header data is data representing the number of leading zeros and the effective bit length of the time series data when the time series data is compressed;
[0013] According to the header data, the compressed data is converted to obtain an exclusive OR value of the time series data and the previous time series data of the time series data;
[0014] The decompressed time series data is obtained based on the XOR value and the previous time series data of the time series data that has been decompressed.
[0015] A third aspect of an embodiment of the present application provides a time series data compression device, including:
[0016] A time series data acquisition module, used to acquire a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order;
[0017] An XOR operation module, used for performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data;
[0018] A header data determination module, used to determine the header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine the representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data;
[0019] A data compression module, used for compressing each of the time series data based on the header data;
[0020] A compressed data array determining module is used to combine the header data and the compressed data corresponding to each of the time series data into a compressed data array of the time series data array.
[0021] A fourth aspect of an embodiment of the present application provides a time series data decompression device, including:
[0022] A compressed data array acquisition module is used to acquire a compressed data array of a time series data array, wherein the compressed data array includes header data and compressed data of a plurality of time series data, wherein the header data is data representing the number of leading zeros and the effective bit length of the time series data when the time series data is compressed;
[0023] An XOR value determination module, configured to convert the compressed data according to the header data to obtain an XOR value of the time series data and the previous time series data of the time series data;
[0024] The decompression module is used to obtain the decompressed time series data based on the XOR value and the previous time series data of the time series data that has been decompressed.
[0025] A fifth aspect of an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method described in the first aspect or the second aspect above is implemented.
[0026] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect or the second aspect above is implemented.
[0027] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device executes the method described in the first aspect or the second aspect.
[0028] Compared with the prior art, the embodiments of the present application have the following advantages:
[0029] When the method in the embodiment of the present application is applied to compress the time series data, multiple continuous time series data can be used as a time series data array, and data compression is performed based on the time series data array. When compressing the time series data array, each time series data in the time series data array can be XORed with the previous time series data to obtain the XOR value of each time series data. Since the difference between the time series data and the previous time series data is generally small, the obtained XOR value generally contains multiple zeros. Based on the leading zeros in each time series data and the effective bit length in each time series data, the header data of the time series data array can be determined. The header data can be used to determine the characterization method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data; based on the determined header data, each time series data can be compressed, so that the compressed time series data can be obtained. The header data of the time series data array and the compressed data of each time series data are combined to obtain a compressed data array. The method in the embodiment of the present application compresses time series data in units of arrays, and characterizes the number of zeros in the XOR value in the entire array based on the header data of the array. The zeros in the XOR value can be fully utilized in the compression process, thereby improving the compression ratio. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art.
[0031] Figure 1 It is a schematic diagram of the steps of a time series data compression method provided in an embodiment of the present application;
[0032] Figure 2 It is a schematic diagram of a data compression process provided by an embodiment of the present application;
[0033] Figure 3 It is a schematic diagram of the steps of another time series data compression method provided by an embodiment of the present application;
[0034] Figure 4 It is a schematic diagram of the steps of a time series data decompression method provided in an embodiment of the present application;
[0035] Figure 5 It is a schematic diagram of a flow chart of time series data compression provided by an embodiment of the present application;
[0036] Figure 6 is a schematic diagram of a time series data compression device provided in an embodiment of the present application;
[0037] Figure 7 is a schematic diagram of a time series data compression device provided in an embodiment of the present application;
[0038] Figure 8 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0039] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.
[0040] Time series data has a time sequence. For example, the temperature data at each time point of each day. Since data is collected according to time, the similarity between time series data and its adjacent data is generally high. When compressing time series data, the high similarity between time series data and its adjacent data can be used to compress data based on the difference between time series data and adjacent data.
[0041] When using a time series data compression algorithm to compress and decompress time series data in a streaming scenario, it is generally expected to achieve the following goals: 1) To ensure high data throughput, the compression / decompression of the algorithm should be efficient enough. 2) All compressed data can be restored losslessly. 3) In a streaming scenario, when compressing the current reported data, there is no need to decompress the compressed historical data. 4) When the above conditions are met, the higher the compression ratio (the ratio of the original data volume to the compressed volume), the better.
[0042] For example: In the IoT scenario, the temperature and humidity sensor reports that the current outdoor temperature is 32°C. When storing the collected data, there is no need to decompress the stored historical temperature data and merge the current data before compressing it. Instead, only compress the data and keep the same compression format as the previous data so that it can be decompressed in the same way in subsequent queries. The final data is completely consistent with the original data without any accuracy errors. At the same time, the original data is stored using 64 binary bits, which may only occupy 20 bits after compression, with a compression ratio of 3.2 times, so we believe that such compression is effective.
[0043] Currently, the most widely used time series data compression technology is Gorilla, which is used or referenced by multiple time series databases, including VictoriaMetrics, Beringei, InfluxDB, Prometheus, etc. The core idea of the Gorilla algorithm is to obtain data with more leading and trailing zeros by XORing the current data with the previous data, and to express this XOR value as a range of valid binary bits (where it starts and where it ends after excluding leading and trailing zeros), and the valid binary bits corresponding to this range. In order to further save space, there is a flag bit to determine whether the current data is consistent with the previous one. If it is consistent, no information needs to be recorded; another flag bit determines whether the current valid binary bit range is included in the previous range. If it is included, the previous range information is directly used.
[0044] However, the Gorilla algorithm cannot fully utilize the existing zeros, resulting in low compression.
[0045] Based on the Gorilla algorithm, the Chimp and Chimp128 algorithms were proposed. The Chimp algorithm counts the leading zero distribution of the XOR values of some time series data sets, and uses whether the number of trailing zeros is greater than the storage cost as a sign to determine whether to compress the trailing zeros. Chimp128 focuses on improving the compression ratio, and XORs the current value with the previous 128 values to find the best compression result.
[0046] However, the compression ratio of the Chimp algorithm is still not ideal. The cost of Chimp128 is a significant increase in time overhead.
[0047] At the same time, the Elf algorithm is an algorithm for floating-point time series data. It is deduced mathematically that for floating-point numbers represented by the IEEE754 standard, when the length of the effective bit in the decimal system is known, a fixed number of bits can be erased from the mantissa to obtain more trailing zeros. At the same time, a set of solutions based on XOR is used to record the erased values.
[0048] Since the lengths of the decimal significant bits of two adjacent data are not necessarily equal, this will cause inconsistency in the erased bits. In the subsequent XOR, the trailing zeros generated will be wasted, resulting in a lower compression ratio.
[0049] The ALP algorithm is also designed to process floating-point time series data. It performs two-level uniform sampling by dividing the data into vectors (1024 values) and forming row groups, so as to determine a pair of indices for each group of vectors, representing the positive and negative powers of 10, to convert floating-point numbers into integers and eliminate the interference of rounding errors. The ALP algorithm is strictly a numerical preprocessing solution, which does not have compression capabilities by itself, but the algorithm can be used with other compression algorithms to achieve better compression ratios.
[0050] However, since floating-point numbers are converted into long integers through multiplication, overflow is very likely to occur, and when overflow occurs, the compression ratio and speed will be greatly reduced.
[0051] Based on this, an embodiment of the present application proposes a time series compression method, which can be optimized by a time series compression algorithm to improve the compression ratio.
[0052] The technical solution of the present application is described below through specific embodiments.
[0053] Reference Figure 1 , shows a schematic flow chart of a time series data compression method provided in an embodiment of the present application, which may specifically include the following steps:
[0054] S101, obtaining a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order.
[0055] The execution subject of this embodiment may be a computer device, and the computer device may be a mobile phone, a tablet computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a server or other computer device. This embodiment does not limit the specific type of the computer device.
[0056] The above-mentioned time series data array can be obtained from the data set to be compressed. The data set can be a data set that has been determined to contain multiple time series data. The data set can also be a data set that is expanded in real time. The method in this embodiment can be applied to compress a data set, and the data set includes time series data, and the time series data can be arranged in chronological order.
[0057] The method in this embodiment compresses the time series data based on the difference between the time series data and the adjacent data. Therefore, in order to accurately achieve decompression later, the first data in the data set can be determined to be the zero value, which is not compressed but serves as the anchor point data for compression and decompression of the time series data. That is, the first time series data in the data set can be written directly to the disk. Starting from the second data in the data set, data compression can be performed according to the array.
[0058] The computer device can obtain time series data from the data set on demand. The user can set the number of time series data that can be included in each array according to the data in the data set or the compression requirements. In this embodiment, the scheme is described by taking the time series data array including 256 time series data as an example.
[0059] Assuming that the time series data array can include a target number of time series arrays, the computer device can sequentially obtain the time series data from the data set, and when the number of the obtained time series data is equal to the target number, the obtained target number of time series data can be used as a time series data array. Alternatively, when the obtained number is not equal to the target number, but there is no other time series data in the data set, the obtained multiple time series data can also be used as a time series data array.
[0060] The time series data array includes a plurality of time series data, and each time series data in the time series data array is arranged in chronological order.
[0061] S102, performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data.
[0062] The difference between two adjacent time series data arrays is generally small. For example, the time series data may be temperature data collected every 1 minute. Generally, the temperature change may not be large every 1 minute. If the temperature does not change for a long time, the first temperature data can be directly recorded, and the remaining temperature data can be marked as the same as the first temperature data, thereby avoiding recording a large amount of temperature data while fully retaining the temperature data information.
[0063] Based on the above principles, the embodiment of the present application can perform data compression based on the difference between the time series data and its previous time series data. The difference between the time series data and its previous time series data can be obtained by XOR operation. XOR operation is a logical operation. When performing XOR operation, the binary bits can be added without carry, that is, the same is 0 and the different is 1. That is, both the time series data and the previous time series data can be represented by binary data. When calculating, the numerical comparison can be performed bit by bit. If the two are the same, the numerical value of the bit in the XOR value is recorded as 0. If the two are different, the numerical value of the bit in the XOR value is recorded as 0. The number of bits of the time series data is the same as the number of bits of the XOR value. The number of bits of the time series data can be determined according to the data storage method on the computer device. In this embodiment, the present application scheme is explained by taking the data stored using 64 binary bits as an example.
[0064] S103, determining the header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine the representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data.
[0065] Since the time series data is similar to the previous time series data, the number of zeros in the XOR value after the XOR operation of the time series data and the previous time series data will be relatively large. For example, when the time series data is the same as the previous time series data, the XOR value may include 64 zeros.
[0066] The XOR value can be divided into three parts: leading zeros, valid digits, and trailing zeros. When the number of leading zeros and the length of valid digits are determined, the number of trailing zeros in the XOR value is also determined. When recording the XOR value, if the number of leading zeros, the length of valid digits, and the value of valid digits are recorded, the XOR value can be determined.
[0067] Based on this, when compressing the time series data, the present embodiment can record the number of leading zeros and the effective bit length in the XOR value of the time series data. However, the number of leading zeros and the effective bit length in the XOR value are not uniform. For example, the time series data is 64 bits, the maximum number of leading zeros in the XOR value is 64, and the maximum effective bit length is also 64 bits. When recording the number of leading zeros and the effective bit length, if the same structure is used to record the number of leading zeros and the effective bit length in each compressed data, in order to ensure that the number of leading zeros and the effective bit length of each XOR value can be recorded and expressed, the number of leading zeros can only be set to 6 bits and the effective bit length can only be set to 6 bits.
[0068] If the length of the numerical value used to represent the number of leading zeros and the length of the numerical value of the effective bit length in the compressed data can be reduced, the compression ratio can be improved.
[0069] In an embodiment of the present application, the number of leading zeros and the effective bit length can be expressed by difference in the compressed data. That is, the actual number of leading zeros may not be recorded in the compressed data, but the difference with a certain value, so that the maximum value of the numerical value used to characterize the number of leading zeros can be reduced, thereby reducing the number of digits of the numerical value characterizing the number of leading zeros in the compressed data. Each time series data in the array can use the same leading zero number base value to calculate the difference, and the leading zero number base value only needs to be recorded once and can be used by the entire time series data array. Similarly, the effective bit length can also be expressed by difference in the compressed data. Each time series data in the array can use the same effective length base value to calculate the difference.
[0070] When compressing a time series data array, it is only necessary to record the leading zero number base value and the effective length base value once. The compressed data array corresponding to the time series data array may include a header data, which may record the leading zero number base value and the effective length base value.
[0071] In addition, in order to unify the structure of the compressed time series data, so that the numerical value used to characterize the number of leading zeros and the effective bit length can be quickly identified from the compressed data during decompression, the computer device also needs to determine how many digits are needed to characterize the number of leading zeros and the effective bit length in the compressed data when the time series data array is compressed. The numerical value characterizing the number of leading zeros can be the first digit, which can be determined based on the maximum difference between the real number of leading zeros and the base value of the number of leading zeros. For example, if the maximum difference between the real number of leading zeros and the base value of the number of leading zeros is 8, then 3 digits can be used to characterize the number of leading zeros, and if the maximum difference between the real number of leading zeros and the base value of the number of leading zeros is 16, then 4 digits can be used to characterize the number of leading zeros. The same is true for the effective bit length. The numerical value characterizing the effective bit length can be the second digit, which can be determined based on the maximum difference between the real effective bit length and the base value of the effective bit length.
[0072] In a possible implementation, the leading zero number base value may be the minimum number of leading zeros in each XOR value, and the effective bit length base value may be the minimum effective bit length in each XOR value. The computer device may determine the number of leading zeros in each XOR value, and determine the maximum number of leading zeros and the minimum number of leading zeros; the computer device may determine the effective bit length in each XOR value, and determine the maximum effective bit length and the minimum effective bit length.
[0073] The number of leading zeros in the compressed data can be represented by the difference between the actual number of leading zeros and the minimum number of leading zeros; the effective bit length in the compressed data can be represented by the difference between the actual effective bit length and the minimum effective bit length.
[0074] The computer device can calculate the first difference between the maximum number of leading zeros and the minimum number of leading zeros, and the first difference is the maximum value of the difference between the actual number of leading zeros and the minimum number of leading zeros in the time series data array. That is, the first difference is the maximum number of leading zeros in the compressed data. Then the computer device can determine the first digit according to the first difference, and the first digit is the number of numerical digits representing the number of leading zeros in the XOR value in the compressed data of the time series data. Specifically, the computer device can perform a logarithmic operation with base 2 on the first value, so as to obtain the first digit. For example, the maximum number of leading zeros is 16, and the minimum number of leading zeros is 8, then the maximum value of the value representing the number of leading zeros in the compressed data is 8, and 8 can be represented by 3 binary bits, so the first digit can be 3.
[0075] The same is true for the effective bit length. The computer device can calculate the second difference between the maximum effective bit length and the minimum effective bit length, and then determine the second digit according to the second difference, the second digit being the number of digits of the value representing the effective bit length in the XOR value in the compressed data of the time series data.
[0076] Afterwards, the computer device can obtain the header data based on the first digit, the second digit, the minimum number of leading zeros, and the minimum length of valid bits.
[0077] The header data may also have a structure. The structure of the header data may include the order of arrangement of the first digit, the second digit, the minimum number of leading zeros and the minimum length of valid bits, as well as the numerical length of the first digit, the numerical length of the second digit, the numerical length of the minimum number of leading zeros and the numerical length of the minimum length of valid bits. The arrangement order may be predetermined. The numerical length may be determined according to the data storage method. The computer device determines the numerical length used to characterize the first digit, the second digit, the minimum number of leading zeros and the minimum length of valid bits according to the data storage method. The numerical length of the first digit is described with the number of bits. The computer device uses 64-bit binary for data storage, so the maximum value of the first digit is 64, and 64 values can be represented using 6-bit binary. Therefore, the numerical length of the first digit in the header data may be 6. Similarly, the numerical lengths of the second digit, the minimum number of leading zeros and the minimum length of valid bits are all 6.
[0078] After determining the length of the numerical value, the computer device can convert the first digit, the second digit, the minimum number of leading zeros and the minimum value of the effective bit length into numerical values corresponding to the numerical length; then combine the converted first digit, the second digit, the minimum number of leading zeros and the minimum value of the effective bit length according to the arrangement method to obtain the header data.
[0079] S104: compress each of the time series data based on the header data.
[0080] Figure 2 is a flow chart of a data compression method provided in an embodiment of the present application. Figure 2 As shown, compressing time series data may include:
[0081] S201, determining a target number of leading zeros, a target valid bit length, and a target valid value in a target XOR value, wherein the target XOR value is an XOR value corresponding to a target timing data, and the target timing data is any timing data in the timing data array.
[0082] 000000000011111100110110101010000000000000000000000000000000000000000 can be a target XOR value, in which the target number of leading zeros is 10, the target significant bit length is 19, and the target significant value is 1111110011011010101.
[0083] S202, calculating the leading zero number difference between the target leading zero number and the minimum leading zero number.
[0084] Assuming that the minimum number of leading zeros in the header data is 7, the difference in the number of leading zeros is 3.
[0085] S203, calculating the effective bit length difference between the target effective bit length and the minimum effective bit length.
[0086] Assuming that the minimum effective digit length is 12, the effective digit length difference is 7.
[0087] S204: Convert the leading zero quantity difference into the leading zero value of the first digit.
[0088] Assuming that the first digit in the header data corresponding to the target timing array is 3, the leading zero value in the compressed data is 011.
[0089] S205: Convert the effective digit length difference into the effective digit value of the second digit.
[0090] Assuming that the second digit in the header data corresponding to the target timing array is 3, the leading zero value in the compressed data is 111.
[0091] S206, combining the leading zero value, the valid digit value and the target valid value to obtain compressed data corresponding to the target time series data.
[0092] Assuming that the compressed data is arranged in the order of leading zero value, valid bit value, and target valid value, the XOR value of the target time series data is: 01111111111110011011010101.
[0093] S105, combining the header data and the compressed data corresponding to each of the time series data into a compressed data array of the time series data array.
[0094] After each time series data in the time series data array is compressed, the header data and each compressed data may be arranged in sequence, thereby obtaining a compressed data array of the time series data array.
[0095] When compressing a data set, the first time series data can be directly recorded, and then the time series data array can be obtained starting from the second time series data. Each time series data array is compressed according to steps S101-S105 until all the data in the data set is compressed.
[0096] In this embodiment, based on the array compression, a header data can be used to record the base value of the number of leading zeros and the base value of the effective bit length, so that the number of leading zeros and the effective bit length can be represented by difference in the compressed data, reducing the number of leading zeros and the number of effective bit lengths in the compressed data, and improving the compression ratio of the time series data. This embodiment makes more efficient use of zeros and has the highest compression ratio on most data sets.
[0097] Reference Figure 3 , shows a schematic flow chart of another method for compressing time series data provided by an embodiment of the present application, which may specifically include the following steps:
[0098] S301, obtaining a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order.
[0099] S301 of this embodiment is similar to S101 of the previous embodiment, and they can be referenced to each other, and are not described in detail here.
[0100] S302, determining whether the time series data in the time series data array can be converted into integers.
[0101] Time series data can be decimals. If the time series data is a decimal, you can convert the decimal to an integer. Integer conversion can convert the time series data to an integer by multiplying the time series data by a power of 10.
[0102] However, integer conversion may result in overflow, so it is necessary to determine whether integer conversion can be performed on time series data.
[0103] S303: If any of the time series data cannot be converted into an integer, a digit is used to represent the integer conversion value.
[0104] If there is a time series data in the time series data array that cannot be converted to an integer, then the time series data array cannot be converted to an integer. In this case, a digit can be used to represent the integer conversion value. For example, a binary bit with a value of 0 can be used to indicate that the time series data array cannot be converted to an integer.
[0105] S304, if each of the time series data can be converted to an integer, the integer conversion value is represented by four digits, and the integer conversion value is the exponent value used to convert the time series data to an integer; and each of the time series data in the time series data array is converted to an integer based on the exponent value.
[0106] If each time series data in the time series data array can be converted to an integer, the exponent value used for the integer conversion can be represented by four digits. And each time series data in the time series data array is converted to an integer based on the exponent value. The integer conversion value can also be stored in the header data.
[0107] In a possible implementation, after executing step S301, the computer device may determine whether there is a floating point decimal in the time series data array. If there is no decimal in the time series data array, no integer conversion is required, and steps S305-S308 are directly executed. If there is a decimal in the time series data array, steps S302-S308 may be executed.
[0108] S305 , performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data.
[0109] S306, determining header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine a representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data;
[0110] S307: compress each of the time series data based on the header data.
[0111] S308: Combine the header data and the compressed data corresponding to each of the time series data into a compressed data array of the time series data array.
[0112] S305 - S308 of this embodiment are similar to S102 - S105 of the previous embodiment, and can be referenced to each other, and will not be described in detail here.
[0113] Based on the method in this embodiment, floating point decimals can be compressed, so that the solution of the embodiment of the present application can compress most data sets. And the complexity of integer conversion can be reduced from O(n 2 ) is reduced to O(n), making the algorithm have faster compression and decompression speeds.
[0114] Reference Figure 4 , shows a schematic flow chart of another method for decompressing time series data provided by an embodiment of the present application, which may specifically include the following steps:
[0115] S401, obtaining a compressed data array of a time series data array, wherein the compressed data array includes header data and compressed data of multiple time series data, wherein the header data is data representing the number of leading zeros and the effective bit length of the time series data when the time series data is compressed.
[0116] The execution subject of this embodiment may be the computer device described in S101, which is not limited here.
[0117] The computer device may extract header data from the compressed data, where the header data may include a first digit, a second digit, a minimum number of leading zeros, and a minimum length of valid bits.
[0118] S402: Convert the compressed data according to the header data to obtain an exclusive OR value of the time series data and the previous time series data of the time series data.
[0119] Based on the first digit, the second digit and the pre-agreed data arrangement, the computer device can determine the leading zero value and the valid bit length value in the compressed data.
[0120] Based on the leading zero value in the compressed data and the minimum number of leading zeros in the header data, the number of leading zeros in the XOR value of the time series data can be determined, where the number of leading zeros is the sum of the leading zero value in the compressed data and the minimum number of leading zeros in the header data.
[0121] Based on the effective bit length value in the compressed data and the minimum effective bit length quantity in the header data, the effective bit length quantity in the XOR value of the time series data can be determined, wherein the effective bit length quantity is the sum of the effective bit length value in the compressed data and the minimum effective bit length quantity in the header data.
[0122] After determining the number of leading zeros and the number of valid bit lengths in the compressed data, the computer device can obtain the valid bit value from the compressed data based on the number of valid bit lengths, and then determine the number of trailing zeros based on the number of leading zeros and the number of valid bit lengths, and then add zeros to the front and back parts of the valid bit value to obtain the XOR value of the time series data.
[0123] S403, obtaining the decompressed time series data based on the XOR value and the previous time series data of the time series data that has been decompressed.
[0124] In this embodiment, the computer device can decompress in sequence, that is, it can decompress from the second time series data. Since the first time series data is not compressed, after obtaining the XOR value of the second time series data, the second time series data can be determined based on the XOR value and the first time series data. Similarly, after obtaining the XOR value of the time series data, the decompressed time series data can be obtained based on the XOR value and the previous time series data.
[0125] Based on the method in this embodiment, the data compressed using the aforementioned time series data compression method can be decompressed to obtain complete data, and the decompression process will not damage the data.
[0126] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0127] The time series data compression method in the embodiment of the present application can be used with most time series data compression algorithms based on XOR. The Gorilla algorithm is used as an example for explanation. Figure 5 , Figure 5A schematic diagram of a process of time series data compression provided by an embodiment of the present application is shown.
[0128] like Figure 5 As shown in the figure, when compressing a data set, when the first value in the data set is input, no compression can be performed, and the original value of the zero value is directly stored as the zero value of the array. For the data after the zero value, the input value can be cached in the array. The length of the array can be set to 256. When there are 256 values in the array except the zero value or when the input reaches the end, the current array can be compressed.
[0129] When compressing an array, you can first determine whether each value in the array can be converted to a power. For example, for each value in the array, you can multiply it by a power of 10 to convert it into an integer. When any value in the array overflows during the power conversion, it can be determined that the array cannot be converted to a power. At this time, you can write a 1-bit record that the array cannot be converted to a power. When each value in the array does not overflow during the power conversion, it can be determined that the array can be converted to a power. At this time, you can write a 4-bit record of the exponent used when the array is converted to a power. It should be noted that when determining whether the power conversion can be performed, the zero value does not need to be considered. However, if the zero value needs to be converted, it can be converted together. Since the zero value has been written, the possible decimal part can be discarded.
[0130] After that, all values in the array can be XORed with the previous value to obtain an XOR value array, which includes the XOR value corresponding to each value. Then the number of leading zeros and trailing zeros of non-zero values in the XOR value array can be counted, and the corresponding effective bit length can be calculated. Based on the determined number, the maximum and minimum values of leading zeros and effective bit length data can be counted. Then the minimum value of the number of 6-bit leading zeros, the number of binary bits occupied by the difference between the maximum and minimum values of the number of 6-bit leading zeros, the minimum value of the 6-bit effective bit length, and the number of binary bits occupied by the difference between the maximum and minimum values of the 6-bit effective bit length can be written.
[0131] After that, the above recorded values can be used to optimize the XOR-based time series data compression algorithm, such as the Gorilla algorithm. When multiplexing fails, the number of leading zeros recorded is the actual number minus the minimum value, and 6 binary bits are used to record the number of leading zeros, and the effective bit length is the same. Then use the optimized solution to process the XOR value array. The last value of the array is used as the new zero value, the sequence number in the group is set to 1, and the second step is repeated until all input data is compressed.
[0132] In this embodiment, in the array to be compressed, the factors affecting the compression ratio include: leading zeros and the length of the binary effective digit, so that redundant zeros are not likely to appear in the compression result, the use of zeros is closer to the optimal value, and the compression effect is further improved. At the same time, for each array, the record data of the leading zero and the length of the binary effective digit only needs to be recorded once. For the data set, the storage overhead used for the record of the leading zero and the length of the binary effective digit can be evenly distributed in each array, so that the compressed data is more balanced.
[0133] Reference Figure 6 , shows a schematic diagram of a time series compression device provided in an embodiment of the present application, which may specifically include a time series data acquisition module 61, an XOR operation module 62, a header data determination module 63, a data compression module 64 and a compressed data array determination module 65, wherein:
[0134] A time series data acquisition module 61 is used to acquire a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order;
[0135] An XOR operation module 62 is used to perform an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data;
[0136] A header data determination module 63, configured to determine the header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine the representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data;
[0137] A data compression module 64, configured to compress each of the time series data based on the header data;
[0138] The compressed data array determining module 65 is used to combine the header data and the compressed data corresponding to each of the time series data into a compressed data array of the time series data array.
[0139] In a possible implementation, the header data determining module 63 includes:
[0140] A determination submodule, used to determine a maximum number of leading zeros, a minimum number of leading zeros, a maximum value of a valid bit length, and a minimum value of a valid bit length of a plurality of the XOR values;
[0141] A first difference calculation submodule, used for calculating a first difference between the maximum number of leading zeros and the minimum number of leading zeros;
[0142] A first digit determination submodule, configured to determine a first digit according to the first difference, wherein the first digit is a numerical digit representing the number of leading zeros in an XOR value in the compressed data of the time series data;
[0143] A second difference calculation submodule, used for calculating a second difference between the maximum effective bit length and the minimum effective bit length;
[0144] A second digit determination submodule, used to determine a second digit according to the second difference, where the second digit is a number of digits representing a valid bit length in an XOR value in the compressed data of the time series data;
[0145] The header data determination submodule is used to obtain the header data based on the first bit number, the second bit number, the minimum number of leading zeros and the minimum value of the effective bit length.
[0146] In a possible implementation, the header data determination submodule includes:
[0147] A length determination unit, used to determine a data storage method, and determine a numerical length used to characterize the first digit, the second digit, the minimum number of leading zeros, and the minimum length of valid bits;
[0148] A numerical value conversion unit, configured to convert the first digit, the second digit, the minimum number of leading zeros, and the minimum length of valid digits into a numerical value corresponding to the numerical value length;
[0149] A combining unit is used to combine the converted first digit number, the second digit number, the minimum number of leading zeros and the minimum value of the effective bit length to obtain the header data.
[0150] In a possible implementation, the data compression module includes:
[0151] A determination submodule, used to determine a target number of leading zeros, a target effective bit length, and a target effective value in a target XOR value, wherein the target XOR value corresponds to an XOR value of a target time series data, and the target time series data is any time series data in the time series data array;
[0152] A leading zero quantity difference calculation submodule, used for calculating the leading zero quantity difference between the target leading zero quantity and the minimum leading zero quantity;
[0153] An effective bit length difference calculation submodule, used for calculating the effective bit length difference between the target effective bit length and the effective bit length minimum value;
[0154] A leading zero value determination submodule, used for converting the leading zero quantity difference into the leading zero value of the first digit;
[0155] A significant digit value determination submodule, used for converting the significant digit length difference into a significant digit value of the second digit;
[0156] The compressed data determination submodule is used to combine the leading zero value, the valid digit value and the target valid value to obtain compressed data corresponding to the target time series data.
[0157] In a possible implementation, the header data further includes an integer conversion value, and before performing an XOR operation on each of the time series data and the previous time series data to obtain an XOR value corresponding to each of the time series data, the apparatus further includes:
[0158] A judgment module, used to determine whether the time series data in the time series data array can be converted into integers;
[0159] A first identification module, configured to represent the integer conversion value by a digit if any of the time series data cannot be converted into an integer;
[0160] The second identification module is used to represent the integer conversion value by four digits if each of the time series data can be converted to an integer, and the integer conversion value is an exponent value used to convert the time series data to an integer; and to perform integer conversion on each of the time series data in the time series data array based on the exponent value.
[0161] In a possible implementation, the above device further includes:
[0162] The zero value processing module is used to directly store the time series data if the time series data is the first time series data in the data set to be compressed.
[0163] Reference Figure 7 , shows a schematic diagram of a timing decompression device provided in an embodiment of the present application, which may specifically include a compressed data array acquisition module 71, an XOR value determination module 72 and a decompression module 73, wherein:
[0164] A compressed data array acquisition module 71 is used to acquire a compressed data array of a time series data array, wherein the compressed data array includes header data and compressed data of a plurality of time series data, wherein the header data is data representing the number of leading zeros and the effective bit length of the time series data when the time series data is compressed;
[0165] An XOR value determination module 72, configured to convert the compressed data according to the header data to obtain an XOR value of the time series data and the previous time series data of the time series data;
[0166] The decompression module 73 is used to obtain the decompressed time series data based on the XOR value and the previous time series data of the time series data that has been decompressed.
[0167] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.
[0168] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 8 As shown, the computer device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one is shown in the figure), a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, wherein the processor 80 implements the steps of any of the above-mentioned method embodiments when executing the computer program 82.
[0169] The computer device 8 may be a desktop computer, a notebook computer, a PDA, a cloud server, or other computer devices. The computer device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the computer device 8 and does not constitute a limitation on the computer device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.
[0170] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0171] In some embodiments, the memory 81 may be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 81 may also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 8. Further, the memory 81 may also include both an internal storage unit of the computer device 8 and an external storage device. The memory 81 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 81 may also be used to temporarily store data that has been output or is to be output.
[0172] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0173] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device can implement the steps in the above-mentioned method embodiments when executing the computer device.
[0174] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.
Claims
1. A time series data compression method, characterized in that: include: Acquire a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order; Performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data; Determine the header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine the representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data; Based on the header data, compress each of the time series data; The header data and the compressed data corresponding to each of the time series data are combined into a compressed data array of the time series data array.
2. The method according to claim 1, characterized in that Determining the header data of the timing data array according to the number of leading zeros and the effective bit length in each XOR value includes: Determine a maximum value, a minimum value, a maximum value, and a minimum value of the number of leading zeros, a maximum value, and a minimum value of the effective bit length of a plurality of the XOR values; Calculate a first difference between the maximum number of leading zeros and the minimum number of leading zeros; Determine a first digit according to the first difference, where the first digit is a numerical digit representing the number of leading zeros in an XOR value in the compressed data of the time series data; Calculate a second difference between the maximum effective bit length and the minimum effective bit length; Determine a second bit number according to the second difference, where the second bit number is the number of bits of a numerical value representing the effective bit length in the XOR value in the compressed data of the time series data; The header data is obtained based on the first digit number, the second digit number, the minimum value of the number of leading zeros, and the minimum value of the effective bit length.
3. The method according to claim 2, characterized in that The obtaining the header data based on the first digit, the second digit, the minimum number of leading zeros and the minimum value of the effective bit length comprises: Determine, according to a data storage method, a numerical length used to represent the first digit, the second digit, the minimum number of leading zeros, and the minimum length of valid bits; Convert the first digit, the second digit, the minimum number of leading zeros, and the minimum length of valid digits into a numerical value corresponding to the numerical length; The converted first digit number, the second digit number, the minimum value of the number of leading zeros and the minimum value of the effective bit length are combined to obtain the header data.
4. The method according to claim 2, characterized in that The compressing each of the time series data based on the header data includes: Determine a target number of leading zeros, a target effective bit length, and a target effective value in a target XOR value, wherein the target XOR value corresponds to an XOR value of target time series data, and the target time series data is any time series data in the time series data array; Calculate the leading zero number difference between the target leading zero number and the minimum leading zero number; Calculate the effective bit length difference between the target effective bit length and the minimum effective bit length; Convert the leading zero quantity difference into the leading zero value of the first digit; Converting the significant digit length difference into a significant digit value of the second digit; The leading zero value, the valid digit value and the target valid value are combined to obtain compressed data corresponding to the target time series data.
5. The method according to any one of claims 1 to 4, characterized in that: The header data also includes an integer conversion value. Before performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain the XOR value corresponding to each of the time series data, the method also includes: Determining whether the time series data in the time series data array can be converted to integers; If any of the time series data cannot be converted to an integer, the integer conversion value is represented by a digit; If each of the time series data can be converted to an integer, the integer conversion value is represented by four digits, and the integer conversion value is an exponent value used to convert the time series data to an integer; and each of the time series data in the time series data array is converted to an integer based on the exponent value.
6. The method according to claim 1, characterized in that The method further comprises: If the time series data is the first time series data in the data set to be compressed, the time series data is directly stored.
7. A time series data decompression method, characterized in that: include: Acquire a compressed data array of a time series data array, wherein the compressed data array includes header data and compressed data of a plurality of time series data, wherein the header data is data representing the number of leading zeros and the effective bit length of the time series data when the time series data is compressed; According to the header data, the compressed data is converted to obtain an exclusive OR value of the time series data and the previous time series data of the time series data; The decompressed time series data is obtained based on the XOR value and the previous time series data of the time series data that has been decompressed.
8. A time series data compression device, characterized in that: include: A time series data acquisition module, used to acquire a time series data array, wherein the time series data array includes a plurality of time series data arranged in chronological order; An XOR operation module, used for performing an XOR operation on each of the time series data and the previous time series data of the time series data to obtain an XOR value corresponding to each of the time series data; A header data determination module, used to determine the header data of the time series data array according to the number of leading zeros and the effective bit length in each of the XOR values, wherein the header data is used to determine the representation method of the number of leading zeros and the effective bit length of the time series data when compressing the time series data; A data compression module, used for compressing each of the time series data based on the header data; A compressed data array determining module is used to combine the header data and the compressed data corresponding to each of the time series data into a compressed data array of the time series data array.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.