Method and apparatus for timing data compression

Abstract

The application discloses a time series data compression method and device, and relates to the technical field of data compression, and the method comprises the following steps: creating a one-dimensional dictionary according to time series data; sorting the data in the one-dimensional dictionary to obtain a first sorting result, and creating a one-dimensional dictionary index according to the first sorting result; performing traversal XOR on the data in the one-dimensional dictionary to obtain a traversal XOR result, compressing the one-dimensional dictionary according to the traversal XOR result, and obtaining a compressed one-dimensional dictionary; determining a one-dimensional dictionary index sequence corresponding to a time sequence according to the one-dimensional dictionary index and the time sequence corresponding to the time series data, compressing the one-dimensional dictionary index sequence corresponding to the time sequence, and obtaining a compressed index sequence; and merging the compressed one-dimensional dictionary and the compressed index sequence to obtain compressed data of the compressed time series data. The application can further improve the compression ratio of data without losing accuracy, save storage space, and realize efficient storage.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data compression, in particular to a time series data compression method and device. BACKGROUND

[0002] This section is intended to provide background or context to the embodiments of the application recited in the claims. The description herein does not constitute admission that the prior publication, square, or context are prior art to the present application.

[0003] In order to realize efficient storage of time series data, it is necessary to compress the time series data. At present, for the lossless compression of integer data, there are relatively mature and efficient general compression algorithms, such as Huffman coding, arithmetic coding, dictionary coding, run-length coding, Simple8b algorithm or combination algorithm. But for the lossless compression of floating point data, there are few mature algorithms, and the relatively popular one is the XOR algorithm proposed by Facebook, which is used in the company's Gorilla time series library. The first ranked time series database Influxdb also uses this algorithm when compressing floating point data. But the compression algorithm of the time series library is to meet the storage of various time series data, that is, the time series data may contain all the representation ranges of floating point data, and the time series database can only use the general data compression algorithm according to the characteristics of time series data itself, such as time series increasing, adjacent data deviation being small. But how the time series increases, whether it is equal interval or unequal interval, and how much the data deviation is, whether it is 1.0 or 1000.0, these information cannot be known in advance, so the compression ratio is inevitably limited. SUMMARY

[0004] The embodiment of the present application provides a time series data compression method, which further improves the compression ratio of data without losing accuracy, saves storage space and realizes efficient storage. The method comprises:

[0005] Creating a one-dimensional dictionary according to the time series data;

[0006] Sorting the data in the one-dimensional dictionary to obtain a first sorting result, and creating a one-dimensional dictionary index according to the first sorting result;

[0007] Iterative XOR operation is performed on the data in the one-dimensional dictionary to obtain an iterative XOR result, and the one-dimensional dictionary is compressed according to the iterative XOR result to obtain a compressed one-dimensional dictionary;

[0008] According to the one-dimensional dictionary index and the time sequence corresponding to the time series data, the one-dimensional dictionary index sequence corresponding to the time sequence is determined, the one-dimensional dictionary index sequence corresponding to the time sequence is compressed, and a compressed index sequence is obtained;

[0009] The compressed one-dimensional dictionary and the compressed index sequence are merged to obtain compressed data of the compressed time series data.

[0010] The embodiment of the present application also provides a time series data compression device, which further improves a compression ratio of data without losing accuracy, saves storage space and realizes efficient storage, and the device comprises:

[0011] a dictionary creating module, which is used for creating a one-dimensional dictionary according to the time series data;

[0012] a dictionary index creating module, which is used for sorting data in the one-dimensional dictionary to obtain a first sorting result, and creating a one-dimensional dictionary index according to the first sorting result;

[0013] a dictionary compression module, which is used for performing traversal XOR on the data in the one-dimensional dictionary to obtain a traversal XOR result, compressing the one-dimensional dictionary according to the traversal XOR result, and obtaining a compressed one-dimensional dictionary;

[0014] an index compression module, which is used for determining a one-dimensional dictionary index sequence corresponding to a time sequence according to the one-dimensional dictionary index and the time sequence corresponding to the time series data, compressing the one-dimensional dictionary index sequence corresponding to the time sequence, and obtaining a compressed index sequence;

[0015] a merging module, which is used for merging the compressed one-dimensional dictionary and the compressed one-dimensional dictionary index sequence to obtain compressed data of the time series data.

[0016] The embodiment of the present application also provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor realizes the time series data compression method when executing the computer program.

[0017] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the time series data compression method when being executed by a processor.

[0018] The embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program realizes the time series data compression method when being executed by a processor.

[0019] In the embodiment of the present application, a one-dimensional dictionary is created according to the time series data; the data in the one-dimensional dictionary is sorted to obtain a first sorting result, and a one-dimensional dictionary index is created according to the first sorting result; the data in the one-dimensional dictionary is traversed and XORed to obtain a traversal XOR result, and the one-dimensional dictionary is compressed according to the traversal XOR result to obtain a compressed one-dimensional dictionary; the time series corresponding to the one-dimensional dictionary index sequence corresponding to the time series data is determined according to the one-dimensional dictionary index and the time series corresponding to the time series data, the one-dimensional dictionary index sequence corresponding to the time series is compressed to obtain a compressed index sequence; and the compressed one-dimensional dictionary and the compressed index sequence are merged to obtain compressed data after compression of the time series data. In this way, the compression ratio of the data can be further improved without losing accuracy, so as to save storage space and realize efficient storage. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. In the drawings:

[0021] Figure 1 A flowchart of a time series data compression method provided in an embodiment of the present application;

[0022] Figure 2 A flowchart of a method of creating a one-dimensional dictionary according to time series data provided in an embodiment of the present application;

[0023] Figure 3 An example diagram of pre-processing a time series data curve TS1 provided in an embodiment of the present application;

[0024] Figure 4 A flowchart of a method of traversing and XORing the data in a one-dimensional dictionary to obtain a traversal XOR result, compressing the one-dimensional dictionary according to the traversal XOR result to obtain a compressed one-dimensional dictionary provided in an embodiment of the present application;

[0025] Figure 5 An example diagram of a leading 0, a meaningful bit string, and a trailing 0 provided in an embodiment of the present application;

[0026] Figure 6 A flowchart of a method of adding the XOR result of each sorting data and the previous sorting data to the binary string from the second sorting data provided in an embodiment of the present application;

[0027] Figure 7 A flowchart of a method of encoding the encoding mode of a one-dimensional dictionary with a binary string provided in an embodiment of the present application;

[0028] Figure 8 A flow chart of a method for compressing an index sequence of a one-dimensional dictionary according to an embodiment of the present application is provided.

[0029] Figure 9 A schematic diagram of a time series data compression device according to an embodiment of the present application is provided.

[0030] Figure 10 A schematic diagram of a computer device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0031] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application are further described in detail below with reference to the drawings. Herein, the schematic embodiments of the present application and the descriptions thereof are used to explain the present application, but not as a limitation of the present application.

[0032] The acquisition, storage, use, processing and the like of data in the technical solutions of the present application all comply with the relevant provisions of the national laws and regulations.

[0033] The term “and / or” herein is merely used to describe an association relationship, which means that there can be three relationships, for example, A and / or B can mean that there are three cases of A alone, A and B together, and B alone. In addition, the term “at least one” herein means any one of multiple or any combination of at least two of multiple, for example, including at least one of A, B and C can mean including any one or more elements selected from the set consisting of A, B and C.

[0034] In the description of the present specification, “comprise”, “include”, “have”, “contain” and the like are all open terms, which means to include but not limited to. The description of the terms “one embodiment”, “one specific embodiment”, “some embodiments”, “for example” and the like means that the specific features, structures or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the schematic description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. The order of the steps involved in each embodiment is used to illustrate the implementation of the present application, and the order of the steps is not limited, which can be adjusted as needed.

[0035] Research has shown that time-series data compression is necessary for efficient storage. Currently, relatively mature and efficient general-purpose compression algorithms exist for lossless compression of integer data, such as Huffman coding, arithmetic coding, dictionary coding, run-length encoding, Simple8b, or combined algorithms. However, for lossless compression of floating-point data, mature algorithms are fewer. The relatively popular algorithm is the XOR algorithm proposed by Facebook, used in their Gorilla time-series database. InfluxDB, the top-ranked time-series database, also uses this algorithm when compressing floating-point data. However, the compression algorithms of time-series databases must meet the storage requirements of various time-series data types. That is, the time-series data may contain all representation ranges of floating-point data. Time-series databases can only use general data compression algorithms based on the inherent characteristics of increasing time series and small deviations between adjacent data points. However, the exact time series increments—whether at equal or unequal intervals—and the magnitude of the data deviation (1.0 or 1000.0) cannot be known in advance. Therefore, the compression ratio is inevitably limited.

[0036] Furthermore, further research revealed that industrial process time-series data, which characterizes production process data, typically has equal sampling intervals and short sampling intervals, usually at the millisecond level. Time-series data has a clear physical meaning, and the maximum change between adjacent samples can be predicted. Most importantly, due to the periodicity and determinism of the production process, time-series data exhibits obvious periodicity and fixed patterns. These characteristics provide an effective guarantee for further improving the compression ratio.

[0037] Regarding the above research, such as Figure 1 As shown, an embodiment of the present invention provides a time-series data compression method, including:

[0038] S101: Create a one-dimensional dictionary based on time-series data;

[0039] S102: Sort the data in the one-dimensional dictionary to obtain the first sorting result, and create a one-dimensional dictionary index based on the first sorting result;

[0040] S103: Perform XOR operations on the data in the one-dimensional dictionary to obtain the XOR result. Compress the one-dimensional dictionary based on the XOR result to obtain the compressed one-dimensional dictionary.

[0041] S104: Based on the one-dimensional dictionary index and the time series corresponding to the time series data, determine the one-dimensional dictionary index sequence corresponding to the time series, compress the one-dimensional dictionary index sequence corresponding to the time series, and obtain the compressed index sequence.

[0042] S105: Merge the compressed one-dimensional dictionary and the compressed index sequence to obtain the compressed data after time-series data compression.

[0043] In this embodiment of the invention, a one-dimensional dictionary is created based on time-series data; the data within the one-dimensional dictionary is sorted to obtain a first sorting result, and a one-dimensional dictionary index is created based on the first sorting result; the data within the one-dimensional dictionary is iterated and XORed to obtain an iterated XOR result, and the one-dimensional dictionary is compressed based on the iterated XOR result to obtain a compressed one-dimensional dictionary; based on the one-dimensional dictionary index and the time series corresponding to the time-series data, a one-dimensional dictionary index sequence corresponding to the time series is determined, and the one-dimensional dictionary index sequence corresponding to the time series is compressed to obtain a compressed index sequence; the compressed one-dimensional dictionary and the compressed index sequence are merged to obtain compressed data after time-series data compression. In this way, the data compression ratio can be further improved without sacrificing accuracy, thereby saving storage space and achieving efficient storage.

[0044] The above-mentioned time-series data compression method will be explained in detail below.

[0045] For the above S101, the time series data includes: one-dimensional time series data or multi-dimensional time series data.

[0046] like Figure 2 The diagram shows a flowchart of a method for creating a one-dimensional dictionary based on time-series data according to an embodiment of the present invention, including:

[0047] S201: Obtain the time series data sequence.

[0048] For example, multiple time-series data sequences TS1, TS2...TSn can be obtained, where the time-series data sequences contain identification data representing different production conditions.

[0049] Among them, the identification data is the data in the time series data. The time series data can reflect the production status. For example, different steel grade IDs mean that different types of steel are being produced, and at this time the production status changes.

[0050] In one embodiment of the present invention, the identification data includes at least one of the following: product specifications, production shifts, production setting parameters, etc.

[0051] Furthermore, time-series data sequences can originate from sources such as real-time data streams, data cache pools, data files, and databases. For example, a time-series data sequence may contain a sequence of markers indicating when dictionary creation has stopped.

[0052] S202: Determine whether the data in the time-series data sequence exists in the one-dimensional dictionary.

[0053] For example, for a sequence TS1, whose data are Dt1, Dt2, ..., Dtm, we can sequentially determine whether Dti (i = 1 to m) exists in the dictionary. For multidimensional time series data, we can sequentially determine the data of each dimension of the sequence.

[0054] S203: If it does not exist, add the data to the dictionary.

[0055] S204: If there is, do not operate, continue the next data judgment, until the stop creating dictionary condition is met, stop creating dictionary, and record the actual size of the dictionary.

[0056] The conditions include at least that the actual size of the dictionary is greater than the preset dictionary size and / or the identification data representing different production conditions changes.

[0057] For example, the sequence TS1 is a 32-bit floating point type (as shown in Figure 3 The data sequence is 0.361033, 0.361033, …, 0.360325, 0.360325, 0.360325, 0.35891, 0.35891, 0.35714, …, and the data in the dictionary is 0.361033, 0.360325, 0.35891, 0.35714, …, TS1 occurs at 90000 points, shift change occurs, that is, shift change occurs, therefore, stop creating dictionary, and at this time the actual size of the dictionary is Da=1147.

[0058] For the above S102, the data in the one-dimensional dictionary is sorted to obtain a first sorting result, and a one-dimensional dictionary index is created according to the first sorting result, for example, including: sorting the data in the one-dimensional dictionary to obtain a first sorting result; and assigning an integer index value to the data in the one-dimensional dictionary according to the first sorting result to obtain a one-dimensional dictionary index.

[0059] Here, the sorting can be any regular sorting, such as: from small to large sorting, from large to small sorting, or interval sampling sorting, etc. For example, the one-dimensional dictionary is sorted from small to large, and the data in the one-dimensional dictionary is arranged as … 0.35714 … 0.35891 … 0.360325 … 0.361033 …

[0060] In addition, according to the first sorting result, an integer index value is assigned to the data in the one-dimensional dictionary, and the order of the index can be any regular sorting, such as: from small to large index, from large to small index, or interval index, etc. The purpose is to reduce the number of different difference values after calculating the difference values of adjacent elements according to the index, so as to facilitate subsequent compression and improve the compression ratio. For example, according to the first sorting result, a small-to-large index is established for the data in the one-dimensional dictionary, which is 0, 1, 2, …, in turn.

[0061] For the above S103, as shown in Figure 4 A method flowchart provided by an embodiment of the application for traversing and XORing the data in the one-dimensional dictionary to obtain a traversal XOR result, and compressing the one-dimensional dictionary according to the traversal XOR result to obtain a compressed one-dimensional dictionary, includes:

[0062] S401: traversing XOR sorting is performed on data in the one-dimensional dictionary to obtain a second sorting result.

[0063] In an embodiment of the present application, traversing XOR sorting is performed on data in the one-dimensional dictionary to obtain a second sorting result, comprising:

[0064] Step 1: taking any data in the one-dimensional dictionary as current sorting data.

[0065] Step 2: performing bit-wise XOR operation on the current sorting data and all unsorted data in the one-dimensional dictionary to obtain an intermediate XOR result, when there is still unsorted data in the one-dimensional dictionary, performing step 3, when there is no unsorted data in the one-dimensional dictionary, jumping to step 4.

[0066] Step 3: taking data with the maximum sum of leading 0 number and trailing 0 number in the intermediate XOR result as the current sorting data, and returning to step 2.

[0067] Here, as shown in the formula (1), the leading 0 number is the number of bits that are 0 from left to right in the data represented by a binary bit string, and the trailing 0 is the number of bits that are 0 from right to left in the data represented by a binary bit string. Figure 5

[0068] Step 4: stopping execution, and obtaining the second sorting result according to each sorting data.

[0069] ​For example, the first data in the one-dimensional dictionary, 0.351446, is taken as the first sorting data. The first sorting data is XORed bitwise with all other data in the dictionary to obtain an intermediate XOR result. The data with the largest sum of leading zeros and trailing zeros in the intermediate XOR result is taken as the second sorting data. In this embodiment, 0.351446 (binary representation 0b00111110101100111111000010111101) is XORed bitwise with all other data in the one-dimensional dictionary. The intermediate XOR result of 0.351518 (binary representation 0b00111110101100111111101000100100) and 0.351446 has the largest sum of leading zeros and trailing zeros, which is 20, therefore 0. 351518 is used as the second sorting data. The second sorting data is XORed with all other data in the dictionary except for the first sorting data. The data with the largest sum of leading zeros and trailing zeros in the intermediate XOR results is used as the third sorting data. In this embodiment, the sum of the number of leading zeros and trailing zeros in the XOR results of 0.373338 (binary representation 0b00111110101111110010011000100100) and 0.351518 is the largest, which is 22. Therefore, 0.373338 is used as the third sorting data. Similarly, the i-th sorting data is XORed with the remaining unsorted data in the dictionary. The data with the largest sum of leading zeros and trailing zeros in the XOR results is used as the (i+1)-th sorting data, until all data is sorted.

[0070] S402: Based on the second sorting result, perform an XOR operation on adjacent data in the one-dimensional dictionary to obtain the adjacent XOR result.

[0071] Since the second sorting result is already sorted by the maximum sum of the number of leading zeros and trailing zeros obtained by bitwise XOR, the length of the meaningful bit string can be reduced, thus improving the compression ratio.

[0072] S403: Encode the results of adjacent XOR operations using binary strings.

[0073] In one embodiment of the present invention, the XOR result of adjacent data is encoded using a binary string, for example including: directly converting the first sorted data into a binary string; starting from the second sorted data, adding the XOR result of each sorted data with the previous sorted data into the binary string.

[0074] Specifically, such as Figure 6 The diagram shows a flowchart of a method provided by an embodiment of the present invention, which involves adding the XOR result of each sorted data item with the previous sorted data item to a binary string, starting from the second sorted data item. The method includes: starting from the second sorted data item, for each sorted data item with the XOR result of the previous sorted data item:

[0075] S601: determining a first number of bits according to an actual binary bit length of the time series data.

[0076] Specifically, the size of the first number of bits N is determined by the actual binary bit length of the time series data. If the length is L bits, 2 raised to the power of N+1 is equal to L. For example, the length of float type data is 32 bits, 2 raised to the power of 5 is equal to 32, and thus N=4. In this embodiment, TS1 is a 32-bit floating-point time series, and thus N=4.

[0077] S602: using a binary code of the first number of bits to represent the number of leading zeros or the number of trailing zeros of the XOR result.

[0078] In an embodiment of the present application, using a binary code of the first number of bits to represent the number of leading zeros or the number of trailing zeros of the XOR result includes: counting the total number of leading zeros and the total number of trailing zeros in the XOR result of each sorting data and the previous sorting data starting from the second sorting data; using a binary code of the first number of bits to represent the number of leading zeros when the total number of leading zeros is less than the total number of trailing zeros; using a binary code of the first number of bits to represent the number of trailing zeros when the total number of leading zeros is not less than the total number of trailing zeros; and incorporating the redundant zeros into the meaningful bit string if the maximum value of the binary representation of the first number of bits is less than the actual number of leading zeros or trailing zeros to be represented.

[0079] For example, count the total number of leading zeros SNL and the total number of trailing zeros SNT of all XOR results. When SNL<SNT, use N bits (i.e., the first number of bits) to represent the number of leading zeros, otherwise, represent the number of trailing zeros. According to the statistics, the number of trailing zeros is represented by N bits of binary here. If the maximum value of the N-bit binary representation is less than the actual number of leading zeros or trailing zeros to be represented, the redundant zeros are incorporated into the meaningful bit string. Assuming that N-bit binary represents the number of leading zeros, N=4, the maximum value of the N-bit binary representation is 0b1111, which is 15 in decimal, and the actual number of leading zeros to be represented is 16, so the redundant 1 leading zero is incorporated into the meaningful bit string. In this embodiment, the XOR result is 0b00000000000000000000101010011001, the number of trailing zeros is 0, and thus the N-bit binary string representing the number of trailing zeros is 0b0000.

[0080] S603: using a binary code of the second number of bits to represent the length of the meaningful bit string of the XOR result; wherein the second number of bits is equal to the first number of bits plus one.

[0081] S604: concatenating the binary code determined by the first number of bits, the binary code determined by the second number of bits, and the meaningful bit string in the XOR result excluding the leading zeros and the trailing zeros.

[0082] For example, the meaningful bit string length of 0b000000000000000000101010011001 is 12. Therefore, the M bits (i.e., the second bit, M = N + 1) of the meaningful bit string are represented as 0b01100. The binary code determined by N and M is then concatenated with the meaningful bit string. The concatenated binary string is represented as 0b000001100101010011001.

[0083] S404: Encodes the encoding method of a one-dimensional dictionary using a binary string, used to identify the encoding method.

[0084] like Figure 7 The diagram shown is a flowchart of a method for encoding a one-dimensional dictionary using binary strings, according to an embodiment of the present invention, including:

[0085] S701: The length of the binary bits of timing data is represented by two binary bits.

[0086] For example, two binary digits can encode four lengths: 0b00 represents 8 bits, 0b01 represents 16 bits, 0b10 represents 32 bits, and 0b11 represents 64 bits. In this embodiment, 0b10 is used to represent a 32-bit floating-point number for TS1.

[0087] S702: Use a single binary code to identify the meaning of the binary code of the first digit.

[0088] For example, a single binary bit can be used to represent whether N bits represent the number of leading zeros or trailing zeros. For instance, 0b0 represents the number of leading zeros, and 0b1 represents the number of trailing zeros. Here, 0b1 represents N bits representing the number of trailing zeros; therefore, the overall header information is encoded as 0b100.

[0089] S405: Combine the encoding method of the one-dimensional dictionary with the encoding of the adjacent XOR result to obtain the compressed one-dimensional dictionary.

[0090] For example, the merged binary string is 0b10000111110101100111111000010111101000001100101010011001……. Its meaning is as follows: 3-bit header information encoding, 32-bit first sorting data, 4-bit trailing zero length, 5-bit meaningful bit string length, 12-bit meaningful bit string…….

[0091] Regarding S104 above, based on the one-dimensional dictionary index and the time series corresponding to the time series data, the one-dimensional dictionary index sequence corresponding to the time series is determined. For example, this includes querying the index values ​​of the same data in the one-dimensional dictionary according to the order of the time series corresponding to the time series data to obtain the one-dimensional dictionary index sequence corresponding to the time series.

[0092] The one-dimensional dictionary index sequence contains the index values ​​corresponding to the values ​​in the time series in the one-dimensional dictionary.

[0093] For example, time series data can be mapped to indices in a one-dimensional dictionary. By querying the index values ​​of identical data in the one-dimensional dictionary corresponding to the time series data, the time series is mapped to an index sequence. In this embodiment, the time series 0.361033, 0.361033, ..., 0.360325, 0.360325, 0.360325, 0.35891, 0.35891, 0.35714, ... are mapped to the index sequences 231, 231, ..., 225, 225, 225, 148, 148, 125, ...

[0094] like Figure 8 The diagram shows a flowchart of a method for compressing an index sequence according to an embodiment of the present invention, comprising:

[0095] S801: Perform a difference operation on adjacent index values ​​in the one-dimensional dictionary index sequence corresponding to the time series.

[0096] This reduces the number of different differences, facilitating subsequent compression and improving the compression ratio. Directly performing interpolation on the original time-series data does not achieve this effect. In this embodiment, after indexing the interpolation, the number of different differences is 90, which is 7.85% of the dictionary size and 0.1% of the total original data. However, if the original time-series data is used for interpolation, the number of different differences is 1219, which is larger than the dictionary's 1147. Therefore, indexing the interpolation helps improve the compression ratio.

[0097] S802: Compress the index after the difference operation using a general lossless integer data compression algorithm.

[0098] General lossless compression algorithms for integer data include, for example, at least one or more combinations of Huffman coding, arithmetic coding, dictionary coding, run-length encoding, Simple8b algorithm, or combined algorithms.

[0099] For S105 above, the compressed one-dimensional dictionary is merged with the compressed index sequence, thus completing the compression of the time series data and obtaining the compressed data after time series data compression.

[0100] This invention also provides a time-series data compression apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the time-series data compression method, the implementation of this apparatus can be referred to the implementation of the time-series data compression method, and repeated details will not be elaborated further.

[0101] like Figure 9 The diagram shown is a schematic of a time-series data compression device provided in an embodiment of the present invention, comprising:

[0102] a dictionary creating module 901, configured to create a one-dimensional dictionary according to the time series data;

[0103] a dictionary index creating module 902, configured to sort data in the one-dimensional dictionary to obtain a first sorting result, and create a one-dimensional dictionary index according to the first sorting result;

[0104] a dictionary compressing module 903, configured to perform a traversal XOR operation on data in the one-dimensional dictionary to obtain a traversal XOR result, and compress the one-dimensional dictionary according to the traversal XOR result to obtain a compressed one-dimensional dictionary;

[0105] an index compressing module 904, configured to determine a one-dimensional dictionary index sequence corresponding to a time sequence according to the one-dimensional dictionary index and the time sequence corresponding to the time series data, and compress the one-dimensional dictionary index sequence corresponding to the time sequence to obtain a compressed index sequence;

[0106] a merging module 905, configured to merge the compressed one-dimensional dictionary and the compressed one-dimensional dictionary index sequence to obtain compressed data of the time series data.

[0107] In a possible implementation, the dictionary creating module is specifically configured to acquire a time series data sequence, wherein the time series data sequence contains identification data representing different production conditions; sequentially determine whether data in the time series data sequence exists in a one-dimensional dictionary; if not, put the data into the dictionary; if yes, do not operate, continue to determine the next data until a condition for stopping creating the dictionary is met, stop creating the dictionary, and record an actual size of the dictionary; wherein the condition at least includes that the actual size of the dictionary is greater than a preset dictionary size and / or the identification data representing different production conditions changes.

[0108] In a possible implementation, the identification data includes at least one of the following: a production product specification, a production shift, and a production setting parameter.

[0109] In a possible implementation, the dictionary index creating module is specifically configured to sort data in the one-dimensional dictionary to obtain a first sorting result, and allocate an integer index value to the data in the one-dimensional dictionary according to the first sorting result to obtain a one-dimensional dictionary index.

[0110] In a possible implementation, the dictionary compressing module is specifically configured to perform a traversal XOR sorting on data in the one-dimensional dictionary to obtain a second sorting result, perform an XOR operation on adjacent data in the one-dimensional dictionary according to the second sorting result to obtain an adjacent XOR result, encode the adjacent XOR result by using a binary string, encode an encoding mode of the one-dimensional dictionary by using the binary string to identify the encoding mode, and merge the encoding mode of the one-dimensional dictionary and the encoding of the adjacent XOR result to obtain the compressed one-dimensional dictionary.

[0111] In a possible implementation, the dictionary compression module is specifically configured to perform the following steps: step 1: taking any data in the one-dimensional dictionary as the current sorting data; step 2: performing a bitwise XOR operation on the current sorting data and all unsorted data in the one-dimensional dictionary to obtain an intermediate XOR result, and performing step 3 when there is still unsorted data in the one-dimensional dictionary, or jumping to step 4 when there is no unsorted data in the one-dimensional dictionary; step 3: taking the data with the maximum sum of the number of leading 0s and the number of trailing 0s in the intermediate XOR result as the current sorting data, and adding 1 to the sorting sequence number of the current sorting data; wherein the number of leading 0s is the number of bits that are 0 from left to right in the data represented by a binary bit string, and the number of trailing 0s is the number of bits that are 0 from right to left in the data represented by a binary bit string, and returning to step 2; and step 4: stopping execution and obtaining the second sorting result according to each sorting data.

[0112] In a possible implementation, the dictionary compression module is specifically configured to directly convert the first sorting data into a binary string, and add, starting from the second sorting data, the XOR result of each sorting data and the previous sorting data to the binary string.

[0113] In a possible implementation, the dictionary compression module is specifically configured to, starting from the second sorting data, for the XOR result of each sorting data and the previous sorting data: determine a first bit number according to the actual binary bit length of the timing data; use a binary code of the first bit number to represent the number of leading 0s or the number of trailing 0s of the XOR result; use a binary code of a second bit number to represent the length of the meaningful bit string of the XOR result; wherein the second bit number is equal to the first bit number plus 1; and concatenate the binary code determined by the first bit number, the binary code determined by the second bit number, and the meaningful bit string of the XOR result excluding the leading 0s and the trailing 0s.

[0114] In a possible implementation, the dictionary compression module is specifically configured to count the total number of leading 0s and the total number of trailing 0s in the XOR result of each sorting data and the previous sorting data starting from the second sorting data; use a binary code of the first bit number to represent the number of leading 0s when the total number of leading 0s is less than the total number of trailing 0s; use a binary code of the first bit number to represent the number of trailing 0s when the total number of leading 0s is not less than the total number of trailing 0s; and incorporate the redundant 0s into the meaningful bit string if the maximum value of the binary representation of the first bit number is less than the number of leading 0s or trailing 0s that actually needs to be represented.

[0115] In a possible implementation, the dictionary compression module is specifically configured to use a two-bit binary code to represent the binary bit length of the timing data, and use a one-bit binary code to identify the meaning of the binary code of the first bit number.

[0116] In a possible implementation, the index compression module is specifically configured to query the index values of the same data in the one-dimensional dictionary in the order of the time sequence corresponding to the time series of the time series data, to obtain a one-dimensional dictionary index sequence corresponding to the time sequence.

[0117] In a possible implementation, the index compression module is specifically configured to perform difference value operation on adjacent index values in the one-dimensional dictionary index sequence corresponding to the time sequence; and perform compression on the index after the difference value operation by using a general integer data lossless compression algorithm.

[0118] Based on the foregoing inventive concept, as shown in Figure 10 The application further provides a computer device 1000, which comprises a memory 1010, a processor 1020, and a computer program 1030 stored in the memory 1010 and capable of running on the processor 1020, and the processor 1020 implements the foregoing time series data compression method when executing the computer program 1030.

[0119] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the foregoing time series data compression method.

[0120] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the foregoing time series data compression method.

[0121] In the embodiment of the application, a one-dimensional dictionary is created according to time series data; data in the one-dimensional dictionary is sorted to obtain a first sorting result, and a one-dimensional dictionary index is created according to the first sorting result; the data in the one-dimensional dictionary is traversed and XORed to obtain a traversal XOR result, and the one-dimensional dictionary is compressed according to the traversal XOR result to obtain a compressed one-dimensional dictionary; a one-dimensional dictionary index sequence corresponding to the time sequence is determined according to the one-dimensional dictionary index and the time sequence corresponding to the time series data, the one-dimensional dictionary index sequence corresponding to the time sequence is compressed to obtain a compressed index sequence; and the compressed one-dimensional dictionary and the compressed index sequence are merged to obtain compressed data after compression of the time series data. In this way, the compression ratio of data can be further improved without loss of accuracy, so as to save storage space and realize efficient storage.

[0122] Those skilled in the art should understand that embodiments of the application can be provided as a method, a system, or a computer program product. Therefore, the application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the application can take the form of a computer program product implemented on one or more computer usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage media, etc.) containing computer usable program code.

[0123] The present application is described in reference to the flowchart and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the flowchart block or blocks.

[0124] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the flowchart block or blocks.

[0125] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart and / or block diagram block or blocks. Figure 1 one or more flowcharts and / or blocks Figure 1 means for carrying out the function specified by the flowchart block or blocks.

[0126] The specific embodiments described above are illustrative of specific embodiments of the present application and are not meant to be limiting of the scope of the application. Other embodiments of the present application will be apparent to those skilled in the art in view of this disclosure. For example, although the present application has been described in the context of a few illustrative embodiments, modifications and improvements can become apparent to those skilled in the art. Accordingly, the patent is not intended to be limited to the illustrative embodiments described herein but is to be accorded the full scope consistent with the claims, the full scope of which is set forth in the claims.

Claims

1. A method for time-series data compression, characterized in that, include: Create a one-dimensional dictionary based on time-series data; Sort the data in the one-dimensional dictionary to obtain the first sorting result, and create a one-dimensional dictionary index based on the first sorting result; The data in the one-dimensional dictionary is traversed and XORed to obtain the traversal XOR result. The one-dimensional dictionary is then compressed based on the traversal XOR result to obtain the compressed one-dimensional dictionary. This includes: traversing and XORing the data in the one-dimensional dictionary to sort it and obtain the second sorting result. Based on the one-dimensional dictionary index and the time series corresponding to the time series data, determine the one-dimensional dictionary index sequence corresponding to the time series, compress the one-dimensional dictionary index sequence corresponding to the time series, and obtain the compressed index sequence. The compressed one-dimensional dictionary and the compressed index sequence are merged to obtain the compressed data after time-series data compression. The creation of a one-dimensional dictionary based on time-series data includes: Obtain a time-series data sequence, which contains identification data representing different production conditions; Check sequentially whether each data point in the time-series data sequence exists in the one-dimensional dictionary; If it does not exist, add the data to the dictionary; If it exists, no operation is performed, and the next data judgment continues until the condition for stopping dictionary creation is met. At this point, dictionary creation is stopped, and the actual size of the dictionary is recorded. The condition includes at least the actual size of the dictionary being greater than the preset dictionary size and / or the identification data representing different production conditions changing. The process involves iterating through the data in the one-dimensional dictionary and performing an XOR sort to obtain a second sorting result, which includes: Step 1: Take any data from the one-dimensional dictionary as the current sorting data; Step 2: Perform a bitwise XOR operation between the currently sorted data and all unsorted data in the one-dimensional dictionary to obtain an intermediate XOR result. If there is still unsorted data in the one-dimensional dictionary, proceed to Step 3. If there is no unsorted data in the one-dimensional dictionary, proceed to Step 4. Step 3: Take the data with the largest sum of leading zeros and trailing zeros in the intermediate XOR results as the current sorted data, and increment the sorting index of the current sorted data by one; where the leading zeros are the number of consecutive zeros from the left in the data represented by the binary bit string, and the trailing zeros are the number of consecutive zeros from the right in the data represented by the binary bit string, and return to step 2. Step 4: Stop execution and obtain the second sorting result based on each sorted data.

2. The time-series data compression method as described in claim 1, characterized in that, The identification data includes at least one of the following: product specifications, production shifts, and production setting parameters.

3. The time-series data compression method as described in claim 1, characterized in that, Sort the data within the one-dimensional dictionary to obtain the first sorting result, and create a one-dimensional dictionary index based on the first sorting result, including: Sort the data in the one-dimensional dictionary to obtain the first sorting result; Based on the first sorting result, integer index values ​​are assigned to the data in the one-dimensional dictionary to obtain the one-dimensional dictionary index.

4. The time-series data compression method as described in claim 1, characterized in that, Perform a traversal XOR operation on the data in the one-dimensional dictionary to obtain the traversal XOR result. Compress the one-dimensional dictionary based on the traversal XOR result to obtain a compressed one-dimensional dictionary, including: Based on the second sorting result, perform an XOR operation on adjacent data in the one-dimensional dictionary to obtain the adjacent XOR result. Encode the XOR result of adjacent strings using binary strings; The encoding method of the one-dimensional dictionary is encoded using binary strings to identify the encoding method; The encoding method of the one-dimensional dictionary is combined with the encoding of the adjacent XOR result to obtain the compressed one-dimensional dictionary.

5. The time-series data compression method as described in claim 4, characterized in that, Encoding the results of adjacent XOR operations using binary strings includes: The first sorted data is directly converted into a binary string; Starting from the second sorted data, the XOR result of each sorted data with the previous sorted data is added to the binary string.

6. The time-series data compression method as described in claim 5, characterized in that, Starting with the second sorted data, the XOR result of each sorted data with the previous sorted data is added to the binary string, including: Starting from the second sorted data, for each sorted data, XOR the result with the previous sorted data: The first bit number is determined based on the actual binary bit length of the timing data; The number of leading zeros or trailing zeros in the XOR result is represented by the binary code of the first digit. The length of the meaningful bit string of the XOR result is represented by the binary code of the second bit; where the second bit is equal to the first bit plus one. The binary code determined by the first digit, the binary code determined by the second digit, and the meaningful bit string in the XOR result, excluding leading and trailing zeros, are concatenated.

7. The time-series data compression method as described in claim 6, characterized in that, The number of leading zeros or trailing zeros in the XOR result is represented by the binary code of the first digit, including: Starting from the second sorted data, count the total number of leading zeros and the total number of trailing zeros in the XOR result of each sorted data and the previous sorted data. When the total number of leading zeros is less than the total number of trailing zeros, the number of leading zeros is represented by the binary code of the first digit. When the total number of leading zeros is not less than the total number of trailing zeros, the number of trailing zeros is represented by the binary code of the first digit. If the maximum value of the binary representation of the first digit is less than the actual number of leading or trailing zeros that need to be represented, then the extra zeros are merged into the meaningful bit string.

8. The time-series data compression method as described in claim 4, characterized in that, Encoding a one-dimensional dictionary using binary strings includes: The length of binary bits for timing data is represented by two binary codes. The meaning of the binary code of the first digit is identified by a single binary code.

9. The time-series data compression method as described in claim 1, characterized in that, Based on the one-dimensional dictionary index and the time series corresponding to the time series data, determine the one-dimensional dictionary index sequence corresponding to the time series, including: By querying the index values ​​of the same data in the one-dimensional dictionary in the order of the time series corresponding to the time series data, the one-dimensional dictionary index sequence corresponding to the time series is obtained.

10. The time-series data compression method as described in claim 1, characterized in that, The one-dimensional dictionary index sequence corresponding to the compressed time series is used to obtain the compressed index sequence, which includes: Perform a difference operation on adjacent index values ​​in the one-dimensional dictionary index sequence corresponding to the time series in the one-dimensional dictionary; The index after the difference operation is compressed using a general lossless integer data compression algorithm.

11. A time-series data compression device, characterized in that, include: The dictionary creation module is used to create a one-dimensional dictionary based on time-series data; The dictionary index creation module is used to sort the data in the one-dimensional dictionary, obtain the first sorting result, and create a one-dimensional dictionary index based on the first sorting result. The dictionary compression module is used to perform XOR operations on the data in the one-dimensional dictionary to obtain the XOR result, and compress the one-dimensional dictionary based on the XOR result to obtain the compressed one-dimensional dictionary. This includes: performing XOR sorting on the data in the one-dimensional dictionary to obtain a second sorting result. The index compression module is used to determine the one-dimensional dictionary index sequence corresponding to the time series based on the one-dimensional dictionary index and the time series corresponding to the time series data, and to compress the one-dimensional dictionary index sequence corresponding to the time series to obtain the compressed index sequence. The merging module is used to merge the compressed one-dimensional dictionary with the compressed one-dimensional dictionary index sequence to obtain compressed data after time-series data compression; The dictionary creation module is specifically used to obtain time-series data sequences, which contain identification data representing different production conditions. Check sequentially whether each data point in the time-series data sequence exists in the one-dimensional dictionary; If it does not exist, add the data to the dictionary; If it exists, no operation is performed, and the next data judgment continues until the condition for stopping dictionary creation is met. At this point, dictionary creation is stopped, and the actual size of the dictionary is recorded. The condition includes at least the actual size of the dictionary being greater than the preset dictionary size and / or the identification data representing different production conditions changing. The dictionary compression module is specifically used to perform the following steps: Step 1: Take any data from the one-dimensional dictionary as the current sorting data; Step 2: Perform a bitwise XOR operation between the currently sorted data and all unsorted data in the one-dimensional dictionary to obtain an intermediate XOR result. If there is still unsorted data in the one-dimensional dictionary, proceed to Step 3. If there is no unsorted data in the one-dimensional dictionary, proceed to Step 4. Step 3: Take the data with the largest sum of leading zeros and trailing zeros in the intermediate XOR result as the current sorted data; where the number of leading zeros is the number of consecutive zeros from the left in the data represented by the binary bit string, and the number of trailing zeros is the number of consecutive zeros from the right in the data represented by the binary bit string. Return to Step 2. Step 4: Stop execution and obtain the second sorting result based on each sorted data.

12. The time-series data compression apparatus as described in claim 11, characterized in that, The dictionary index creation module is specifically used to sort the data within a one-dimensional dictionary and obtain the first sorting result; Based on the first sorting result, integer index values ​​are assigned to the data in the one-dimensional dictionary to obtain the one-dimensional dictionary index.

13. The time-series data compression apparatus as described in claim 11, characterized in that, The dictionary compression module is specifically used to perform an XOR operation on adjacent data in the one-dimensional dictionary based on the second sorting result, and obtain the adjacent XOR result. Encode the XOR result of adjacent strings using binary strings; The encoding method of the one-dimensional dictionary is encoded using binary strings to identify the encoding method; The encoding method of the one-dimensional dictionary is combined with the encoding of the adjacent XOR result to obtain the compressed one-dimensional dictionary.

14. The time-series data compression apparatus as described in claim 13, characterized in that, The dictionary compression module is specifically used to directly convert the first sorted data into a binary string. Starting from the second sorted data, the XOR result of each sorted data with the previous sorted data is added to the binary string.

15. The time-series data compression apparatus as described in claim 14, characterized in that, The dictionary compression module is specifically used to XOR the result of each sorted data with the previous sorted data, starting from the second sorted data: The first bit number is determined based on the actual binary bit length of the timing data; The number of leading zeros or trailing zeros in the XOR result is represented by the binary code of the first digit. The length of the meaningful bit string of the XOR result is represented by the binary code of the second bit; where the second bit is equal to the first bit plus one. The binary code determined by the first digit, the binary code determined by the second digit, and the meaningful bit string in the XOR result, excluding leading and trailing zeros, are concatenated.

16. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 10.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

18. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 10.

Images