A time series data compression method, electronic equipment and storage medium

Abstract

The application provides a time series data compression method, which comprises the following steps: when the storage amount of the uncompressed time series data is greater than a first set threshold, performing a first compression conversion according to categories; and when the storage amount of all the time series data is greater than a second set threshold, performing a second compression conversion, so that the maximum data compression rate can be achieved, and the storage cost can be reduced. The application also provides an electronic device and a storage medium.

Description

Technical Field

[0001] This invention relates to the field of data compression, and in particular to a method, electronic device, and storage medium for compressing time-series data. Background Technology

[0002] Currently, intelligent interconnection technologies such as the Internet of Things (IoT), the Industrial Internet, and the Internet of Vehicles (IoV) are rapidly being adopted across various industries, leading to a dramatic increase in the number of networked sensors and smart devices. With the use of these devices, a large amount of time-series data has already been generated or will be generated in the future, such as temperature and humidity data, resource consumption data (water and electricity), and industrial equipment operation data. The resulting massive amounts of time-series monitoring data present challenges in storage, transmission, and processing, placing higher demands on efficient compression and storage capabilities. While standard compression methods still offer value for the increasingly large-scale storage of IoT time-series big data, many scenarios, such as temperature tags that monitor temperature data offline for extended periods (years), present new requirements for time-series data acquisition, storage, compression / decompression, and analysis technologies. Business applications in these scenarios require the use of large amounts of historical data to analyze both current and historical time-series data. However, the large volume of historical data increases storage space requirements and thus storage costs. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] This invention provides a method for compressing time-series data, the method comprising the following steps:

[0005] S100: Acquire timing data and store it according to the set storage format;

[0006] S200, at any detection time T a If the amount of uncompressed time-series data stored in the time-series data exceeds a first preset threshold, execute S300; if the amount of stored time-series data exceeds a second preset threshold, execute S400; T a+1 -T a =△t, where △t is the set time interval and a is a natural number greater than or equal to 1; if the amount of stored time-series data is greater than or equal to the maximum storage capacity of the corresponding storage device, exit the control program; the first set threshold is less than the second set threshold, and the second set threshold is less than the maximum storage capacity;

[0007] S300, compress the uncompressed time-series data based on the first defined classification table to obtain a first compressed data table and store it, and delete the corresponding uncompressed time-series data; the i-th row of the first defined classification table includes (k1 i v1 i ), k1 iLet v1 be the i-th category in the first classification table. i For k1 i The corresponding data range, i ranges from 1 to n1, where n1 is the number of categories in the first defined classification table; the j-th row (k1) of the first compressed data table ’ j p1 j ), k1 ’ j For the category corresponding to the j-th row in the first compressed data table, k1 ’ j ∈ (k11, k12, ..., k1) i , ..., k1 n ), p1 j For uncompressed time series data belonging to k1 ’ j The number of data; j takes values ​​from 1 to m, where m is the total number of rows in the first compressed data table; set a = a + 1, and execute S200;

[0008] S400: Based on the first set classification table, compress the uncompressed time series data to obtain a first compressed data table and store it; compress the first compressed data table to obtain a second compressed data table and store it; set a=a+1, and execute S200.

[0009] The present invention has at least the following beneficial effects:

[0010] The time-series data compression method provided in this invention performs a first compression conversion according to category when the amount of uncompressed time-series data stored exceeds a first set threshold, and performs a second compression conversion according to category when the amount of all stored time-series data stored exceeds a second set threshold. This can achieve a maximum data compression rate and can effectively solve the storage space problem for application scenarios that require offline storage of data for a long period of time, such as 5 years or more, thereby reducing storage costs. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a time-series data compression method provided in an embodiment of the present invention. Detailed Implementation

[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] Figure 1 A flowchart of a time-series data compression method provided in an embodiment of the present invention.

[0015] This invention provides a method for compressing time-series data, such as... Figure 1 As shown, the method may include the following steps:

[0016] S100 acquires timing data and stores it according to the set storage format.

[0017] In this embodiment of the invention, the time-series data can be data monitored by monitoring equipment, such as temperature and humidity data, water and electricity consumption data, and industrial equipment operation data.

[0018] In this embodiment of the invention, the storage format is set to timestamp plus value. In one example, the storage format of time-series data can be shown in Table 1 below:

[0019] Table 1 Time-series data storage format

[0020]

[0021] S200, at any detection time T a If the amount of uncompressed time-series data stored in the time-series data exceeds a first preset threshold, execute S300; if the amount of stored time-series data exceeds a second preset threshold, execute S400; T a+1 -T a =△t, where △t is the set time interval and a is a natural number greater than or equal to 1; if the amount of time-series data stored is greater than or equal to the maximum storage capacity of the corresponding storage device, the control program exits, which means the data compression program exits; the first set threshold is less than the second set threshold, and the second set threshold is less than the maximum storage capacity.

[0022] In this embodiment of the invention, △t can be set based on actual needs. For example, it can be determined based on the acquisition time of the time series data, and can be set to be greater than the acquisition time of the data.

[0023] In this embodiment of the invention, the maximum storage capacity of the storage device may be the maximum number of data entries allowed to be stored, the maximum file size allowed to be stored, etc.

[0024] The first and second threshold values ​​can be set based on actual needs. In one illustrative embodiment, the first threshold value can be equal to 50% of the maximum storage capacity, and the second threshold value can be equal to 75% of the maximum storage capacity, but it is not limited to these values ​​and can be set to other values.

[0025] S300, compress the uncompressed time-series data based on the first defined classification table to obtain a first compressed data table and store it, and delete the corresponding uncompressed time-series data; the i-th row of the first defined classification table includes (k1 i v1 i ), k1 i Let v1 be the i-th category in the first classification table. i For k1 i The corresponding data range, i ranges from 1 to n1, where n1 is the number of categories in the first defined classification table; the j-th row (k1) of the first compressed data table ’ j p1 j ), k1 ’ j For the category corresponding to the j-th row in the first compressed data table, k1 ’ j ∈ (k11, k12, ..., k1) i , ..., k1 n ), p1 j For uncompressed time series data belonging to k1 ’ j The number of data; j takes values ​​from 1 to m, where m is the total number of rows in the first compressed data table; set a=a+1, and execute S200.

[0026] Time-series data originates from various sources, including IoT devices, financial centers, and the internet. The data's form and characteristics vary greatly, as do its patterns, and the required compression precision differs accordingly. Therefore, by identifying patterns in time-series data, it can be transformed from one data space to another according to certain rules, thus achieving data compression. For example, observing the temperature time-series data shown in Table 2, we can see that it can be divided into two categories based on its values: 2-8℃ and >8℃. Therefore, these can be denoted as category 0 and 1, respectively.

[0027] Table 2 Temperature Time Series Data

[0028]

[0029] Further observation reveals that consecutive data points 1-9 belong to class 0, data point 10 belongs to class 1, data points 11-15 belong to class 0, and data points 16-19 belong to class 1. Converting this to another time-series data format yields:

[0030] .

[0031] It can be observed that the converted time-series data storage reduces a significant amount of data compared to the original time-series data. Although there is some loss in accuracy, a large data compression rate can be achieved. For many IoT devices that need to store data offline for a long time (e.g., 5 years), this can effectively solve the storage space problem.

[0032] Based on the above, this application establishes a first preset classification table. In this embodiment of the invention, the data structure of the first preset classification table can consist of a key and a value. The key can be represented by numbers such as 0, 1, 2..., and the value can be a predefined numerical value. For example, taking a temperature label as an example, multiple different temperature ranges such as 2-8°C can be preset. The data structure of the first preset classification table for temperature ranges is shown in Table 3 below:

[0033] Table 3: First Setting Classification Table for Temperature Range

[0034]

[0035] Furthermore, in this embodiment of the invention, the compression of uncompressed time-series data based on a first defined classification table may include:

[0036] S301, for the first data item of uncompressed time-series data, if the value in the first data item belongs to v1 in the first defined classification table... i Then set the category of the current time series data to k1. i It also starts the category counter and increments the category counter value by 1.

[0037] S302, for any data s1 in the uncompressed time series data, if s1≤m1, execute S303; m1 is the number of uncompressed time series data; otherwise, execute S307; the value of s1 is from 2 to m1.

[0038] S303, if the category of data s1 is the same as the category of data s1-1, then increment the current category counter by 1; otherwise, execute S304.

[0039] S304, store the current classification result, which includes the category of the current time series data and the value of the corresponding category counter; and after storage, set the value of the current category counter to 0; execute S305.

[0040] S305, update the category of the current time series data to the category corresponding to data s1, and increment the value of the current category counter by 1; execute S306.

[0041] S306, set s1=s1+1; execute S302.

[0042] S307, store the current classification result and exit the control program.

[0043] The classification results obtained from S301 to S307 can be used to generate the first compressed data table.

[0044] In a specific example, taking the time series data shown in Table 4 as an example, when compressing the uncompressed time series data based on the first preset classification table, starting from the first data entry, the temperature value is found to be 6.7°C, which is within the 2-8°C range. Therefore, according to the preset rules in the first preset table, it is classified into category "0" and counted as 1. Then, the second to ninth data entries are traversed, and the temperature value of each data entry is within the 2-8°C range, so it can also be classified into category "0", and the count is 9. When traversing to the tenth data entry, the temperature value is found to be 9.2°C, which is not within the 2-8°C range. Therefore, the traversal is paused, the previous traversal results are saved, and the count is reset to zero. Traversing continues from the tenth data entry. According to the preset rules in the first preset table, the temperature value of 9.2°C is within the 8-15°C range, so it can be classified into category "1", and the count is 1. The previous traversal process can be repeated. The final stored classification result table, i.e., the first compressed data table, is shown in Table 5.

[0045] Table 4

[0046]

[0047] Table 5 Classification Results

[0048]

[0049] Furthermore, in this embodiment of the invention, the category and the number of categories in the first compressed data table are represented using two bytes. Specifically, the category (key) in the compressed data table can be represented using the high four bits of the odd-numbered bytes, and the number of categories (value) can be represented using the low four bits of the odd-numbered bytes and the even-numbered bytes.

[0050] In one illustrative embodiment, there are 300 temperature data points in the temperature range 2-8. These are represented by two bytes as 0x012C, which is 0b0000000100101100 in binary. The lower 12 bits of the binary representation can be used to store the number of data points in that category. Since zero data points don't actually need to be represented, all 12 bits of zero represent one data point, 0000 0000 0001 represent two data points, and all 1 bits represent 4096 data points. After this processing, 300 data points are represented by the binary representation 0b0000000100101011, which is 0x012B in hexadecimal. Under this mechanism, each two bytes can represent a maximum of 4096 data points. If the cumulative number of data records in that category exceeds 4096, such as 8192, then two consecutive two-byte representations can be used. If the number exceeds 8192, such as 9000, then three consecutive two-byte representations can be used. Similarly, if the number of data entries is N, then it can be represented by a continuous roundup (N / 4096) two-byte array, where roundup() means rounding up.

[0051] S400: Based on the first set classification table, compress the uncompressed time series data to obtain a first compressed data table and store it; compress the first compressed data table to obtain a second compressed data table and store it; set a=a+1, and execute S200.

[0052] The time-series data compression method provided in this embodiment of the invention performs a first compression conversion according to category when the storage amount of uncompressed time-series data exceeds a first set threshold, and performs a second compression conversion when the storage amount of all stored time-series data exceeds a second set threshold. This can achieve a maximum data compression rate and can effectively solve the storage space problem for application scenarios that require offline storage of data for a long time, such as 5 years or more, thereby reducing storage costs.

[0053] Furthermore, in one embodiment of the present invention, S400 may include:

[0054] The first compressed data table is compressed based on the second defined classification table to obtain the second compressed data table, which is then stored; a = a + 1 is set, and S200 is executed.

[0055] Among them, the r-th row of the second defined classification table includes (k2) r v2 r ), k2 r For the second category, set the r-th category in the classification table, v2 r For k2 rThe corresponding data range is r, which ranges from 1 to n2, where n2 is the number of categories in the second defined classification table; n2 < n1.

[0056] In this embodiment of the invention, the data structure of the second defined classification table is the same as that of the first defined classification table, except that the data range of the classification is larger than that of the first defined classification table. Taking the data range and classification shown in Table 3 as an example, the second defined classification table can be shown in Table 6 below:

[0057] Table 6: Second Setting Classification Table for Temperature Range

[0058]

[0059] In this embodiment of the invention, the category and the number of categories in the second compressed data table are represented using two bytes. Specifically, the category (key) in the compressed data table can be represented using the high four bits of the odd-numbered bytes, and the number of categories (value) can be represented using the low four bits of the odd-numbered bytes and the even-numbered bytes.

[0060] In one illustrative embodiment, there are 300 temperature data points in the temperature range 2-8. These are represented by two bytes as 0x012C, which is 0b0000000100101100 in binary. The lower 12 bits of the binary representation can be used to store the number of data points in that category. Since zero data points don't actually need to be represented, all 12 bits of zero represent one data point, 0000 0000 0001 represent two data points, and all 1 bits represent 4096 data points. After this processing, 300 data points are represented by the binary representation 0b0000000100101011, which is 0x012B in hexadecimal. Under this mechanism, each two bytes can represent a maximum of 4096 data points. If the cumulative number of data records in that category exceeds 4096, such as 8192, then two consecutive two-byte representations can be used. If the number exceeds 8192, such as 9000, then three consecutive two-byte representations can be used. Similarly, if the number of data entries is N, then it can be represented by a continuous roundup (N / 4096) of two bytes.

[0061] Furthermore, in this embodiment of the invention, compressing the first compressed data table based on the second defined classification table may include:

[0062] S401, For the first data entry in the first compressed data table, if the category in the first data entry matches k2... r If they are the same, then set the category of the current time series data to k2. r It also starts the category counter and increments the category counter value by 1.

[0063] S402, for any data s2 in the first compressed data table, if s2≤m2, execute S403; m2 is the number of compressed time-series data; otherwise, execute S407; the value of s2 is from 2 to m2.

[0064] S403, if the category of data s2 is the same as that of data s2-1, then increment the current category counter by 1; otherwise, execute S404.

[0065] S404, store the current classification result, which includes the category of the current time series data and the value of the corresponding category counter; and after storage, set the value of the current category counter to 0; execute S405.

[0066] S405, update the category of the current time series data to the category corresponding to data s2, and increment the value of the current category counter by 1; execute S406.

[0067] S406, set s2=s2+1; execute S402;

[0068] S407, store the current classification result and exit the control program.

[0069] Based on the classification results stored in S401 to S407, a second compressed data table can be obtained.

[0070] Furthermore, in another embodiment of the present invention, S400 may include merging the time-series data in the first compressed data table according to category to compress the first compressed data table, obtain a second compressed data table and store it; set a=a+1, and execute S200.

[0071] Furthermore, in this embodiment of the invention, merging the time-series data in the first compressed data table according to category may include:

[0072] S41, for any row i in the first set classification table, if i≤n1, start counter C, the initial value of counter C is 0, and execute S42; otherwise, exit the control program;

[0073] S42, For any data s in the first compressed data table, if s≤m, execute S43; m is the number of time-series data in the first compressed data table; otherwise, execute S45; the value of s is from 1 to m;

[0074] S43, if the category of data s is the same as k1 i Similarly, set C = C + p2 s p2 s The number of categories in data s; otherwise, execute S44;

[0075] S44, set s=s+1; execute S42;

[0076] S45, store the current result, which includes the current category k1. i and the current value of C; execute S46;

[0077] S46, set i=i+1, execute S41.

[0078] Based on the results stored in S41 to S46, a second compressed data table can be obtained.

[0079] The time-series data compression method provided in this embodiment can achieve similar technical effects to the aforementioned embodiments.

[0080] In another embodiment of the present invention, S400 may include:

[0081] S410, compress the first compressed data table based on the second defined classification table; the r-th row of the second defined classification table includes (k2 r v2 r ), k2 r For the second category, set the r-th category in the classification table, v2 r For k2 r The corresponding data range is r, which ranges from 1 to n2, where n2 is the number of categories in the second defined classification table; n2 < n1.

[0082] For specific implementation of S410, please refer to the aforementioned S401 to S407.

[0083] S420: Merge the time-series data compressed in S410 according to category to obtain the second compressed data table; set a=a+1 and execute S200.

[0084] Specifically, the S420 may include:

[0085] S421, For any row r in the second set classification table, if r≤n2, start counter C, the initial value of counter C is 0, and execute S422; otherwise, exit the control program;

[0086] S422, For any data q in the compressed time series data, if q≤h, execute S243; h is the number of compressed time series data; otherwise, execute S425; the value of q is from 1 to h;

[0087] S423, if the category of data q is the same as k2 r Similarly, set C = C + p2 q p2 q The number of categories in data q; otherwise, execute S424.

[0088] S424, set q=q+1; execute S422;

[0089] S425, store the current result, which includes the current category k2. r and the current value of C; execute S426;

[0090] S426, set r=r+1, execute S421.

[0091] Based on the results stored in S421 to S426, a second compressed data table can be obtained.

[0092] The time-series data compression method provided in this embodiment can further improve the data compression rate compared with the previous embodiments.

[0093] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0094] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0095] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0096] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of the invention is defined by the appended claims.

Claims

1. A method for compressing time-series data, characterized in that, The method includes the following steps: S100: Acquire timing data and store it according to the set storage format; S200, at any detection time T a If the amount of uncompressed time-series data stored in the time-series data exceeds a first preset threshold, execute S300; if the amount of stored time-series data exceeds a second preset threshold, execute S400; T a+1 -T a =△t, where △t is the set time interval and a is a natural number greater than or equal to 1; if the amount of stored time-series data is greater than or equal to the maximum storage capacity of the corresponding storage device, exit the control program; the first set threshold is less than the second set threshold, and the second set threshold is less than the maximum storage capacity; S300, compress the uncompressed time-series data based on the first defined classification table to obtain a first compressed data table and store it, and delete the corresponding uncompressed time-series data; the i-th row of the first defined classification table includes (k1 i v1 i ), k1 i Let v1 be the i-th category in the first classification table. i For k1 i The corresponding data range, i ranges from 1 to n1, where n1 is the number of categories in the first defined classification table; the j-th row (k1) of the first compressed data table ’ j p1 j ), k1 ’ j For the category corresponding to the j-th row in the first compressed data table, k1 ’ j ∈ (k11, k12, ..., k1) i , ..., k1 n ), p1 j For uncompressed time series data belonging to k1 ’ j The number of data; j takes values ​​from 1 to m, where m is the total number of rows in the first compressed data table; set a = a + 1, and execute S200; S400, compress the uncompressed time-series data based on the first set classification table to obtain a first compressed data table and store it, and compress the first compressed data table to obtain a second compressed data table and store it. Set a = a + 1, and execute S200.

2. The method according to claim 1, characterized in that, The compression of uncompressed time-series data based on the first defined classification table includes: S301, for the first data item of uncompressed time-series data, if the value in the first data item belongs to v1 in the first defined classification table... i Then set the category of the current time series data to k1. i And start the category counter, setting the category counter value to increment by 1; S302, For any data s1 in the uncompressed time series data, if s1≤m1, execute S303; m1 is the number of uncompressed time series data; otherwise, execute S307; the value of s1 is from 2 to m1; S303, If the category of data s1 is the same as the category of data s1-1, then increment the current category counter by 1; otherwise, execute S304. S304, Store the current classification result, which includes the category of the current time series data and the value of the corresponding category counter; and after storing, set the value of the current category counter to 0; Execute S305; S305, update the category of the current time series data to the category corresponding to data s1, and increment the value of the current category counter by 1; execute S306; S306, set s1=s1+1; execute S302; S307, store the current classification result and exit the control program.

3. The method according to claim 1, characterized in that, In S400, the first compressed data table is compressed based on the second defined classification table; the r-th row of the second defined classification table includes (k2 r v2 r ), k2 r For the second category, set the r-th category in the classification table, v2 r For k2 r The corresponding data range is r, which ranges from 1 to n2, where n2 is the number of categories in the second defined classification table; n2 < n1.

4. The method according to claim 1, characterized in that, In S400, the time-series data in the first compressed data table are merged according to category in order to compress the first compressed data table.

5. The method according to claim 1, characterized in that, The S400 further includes: S410, compress the first compressed data table based on the second defined classification table; the r-th row of the second defined classification table includes (k2 r v2 r ), k2 r For the second category, set the r-th category in the classification table, v2 r For k2 r The corresponding data range is r, which ranges from 1 to n2, where n2 is the number of categories in the second defined classification table; n2 < n1. S420: Merge the time-series data compressed in S410 according to category to obtain the second compressed data table; set a=a+1 and execute S200.

6. The method according to claim 3 or 5, characterized in that, The compression of the first compressed data table based on the second defined classification table includes: S401, for the first data point of the compressed time series data, if the category in the first data point matches k2... r If they are the same, then set the category of the current time series data to k2. r And start the category counter, setting the category counter value to increment by 1; S402, For any data s2 in the compressed time series data, if s2≤m2, execute S403; m2 is the number of compressed time series data; otherwise, execute S407; the value of s2 is from 2 to m2; S403, if the category of data s2 is the same as the category of data s2-1, then increment the current category counter by 1; otherwise, execute S404. S404, Store the current classification result, which includes the category of the current time series data and the value of the corresponding category counter; and set the value of the current category counter to 0 after storage; Execute S405; S405, update the category of the current time series data to the category corresponding to data s2, and increment the value of the current category counter by 1; execute S406; S406, set s2=s2+1; execute S402; S407, store the current classification result and exit the control program.

7. The method according to claim 4, characterized in that, The time-series data in the first compressed data table are merged according to category, including: S41, for any row i in the first set classification table, if i≤n1, start counter C, the initial value of counter C is 0, and execute S42; otherwise, exit the control program; S42, For any data s in the first compressed data table, if s≤m, execute S43; m is the number of time-series data in the first compressed data table; otherwise, execute S45; the value of s is from 1 to m; S43, if the category of data s is the same as k1 i Similarly, set C = C + p2 s p2 s The number of categories in data s; otherwise, execute S44; S44, set s=s+1; execute S42; S45, store the current result, which includes the current category k1. i and the current value of C; execute S46; S46, set i=i+1, execute S41.

8. The method according to claim 5, characterized in that, S420 further includes: S421, For any row r in the second set classification table, if r≤n2, start counter C, the initial value of counter C is 0, and execute S422; otherwise, exit the control program; S422, For any data q in the compressed time series data, if q≤h, execute S243; h is the number of compressed time series data; otherwise, execute S425; the value of q is from 1 to h; S423, if the category of data q is the same as k2 r Similarly, set C = C + p2 q p2 q The number of categories in data q; otherwise, execute S424. S424, set q=q+1; execute S422; S425, store the current result, which includes the current category k2. r and the current value of C; execute S426; S426, set r=r+1, execute S421.

9. A non-transitory computer-readable storage medium storing at least one instruction or at least one program segment, said at least one instruction or said at least one program segment being loaded and executed by a processor to implement the method of any one of claims 1-8.

10. An electronic device, characterized in that, Includes a processor and the non-transitory computer-readable storage medium as described in claim 9.

Images