Data compression method, device and system based on parameter optimization and storage medium

Abstract

The invention discloses a data compression method, device and system based on parameter optimization, and a storage medium. The method comprises the following steps: S1, obtaining time sequence data sets of different modes; and S2, according to different types of time sequence data sets, carrying out time sequence floating-point number lossless compression through a parameter optimizer. By the adoption of the technical scheme, excellent performance can be stably exerted on different data sets in a self-adaptive mode, and the compression ratio and the compression and decompression speed are further improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression, and in particular relates to a data compression method and device, a system, and a storage medium based on parameter optimization. Background Art

[0002] At present, lossless data compression based on time series floating point numbers has been developing continuously in recent years. Pfpc is a prediction-based double-precision parallel lossless compression algorithm for scientific simulation data. Pfpc uses parallel computing to improve the overall throughput. The original data is divided into blocks and distributed on multiple CPU threads, but there is a trade-off between compression ratio and throughput. Gorilla was proposed by Facebook and was first used to store time series data. Gorilla uses the XOR differential representation method of floating point numbers to compress data by recording the differences between adjacent floating point numbers instead of the actual values. The Gorilla algorithm significantly reduces the storage of duplicate data and compresses the redundant bits of floating point numbers through XOR and bit-level storage schemes. Chimp is a lossless compression algorithm based on floating point number compression in time series data. At the same time, it is based on the workflow of Gorilla. Chimp redesigns the compression strategy to improve the compression ratio when the trailing zeros of the XOR residual are less than 6.

[0003] However, the current time series floating-point lossless compression algorithms are difficult to achieve stable results on different types of data sets. Even the Chimp algorithm, which has received much attention in recent years, cannot maintain its advantage on the current data set while maintaining its advantage on other data sets. The current time series floating-point lossless compression algorithms lack sufficient adaptability and robustness. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a data compression method and device, system, and storage medium based on parameter optimization, which can adaptively and stably exert excellent performance on different data sets, so as to further improve the compression ratio, compression and decompression speed.

[0005] To achieve the above object, the present invention adopts the following technical solution:

[0006] A data compression method based on parameter optimization, comprising:

[0007] Step S1, obtaining time series data sets of different modes;

[0008] Step S2: According to different categories of time series data sets, lossless compression of time series floating point numbers is performed through a parameter optimizer.

[0009] Preferably, the parameter optimizer is constructed by a heuristic genetic algorithm.

[0010] Preferably, in the parameter optimizer, the initial population length is set to 63, and the statistics of leading 0s are used to perform a heuristic genetic algorithm on the time series data set to obtain the best 8 encoding values ​​from 0 to 64.

[0011] The present invention also provides a data compression device based on parameter optimization, comprising:

[0012] Acquisition module, used to obtain time series data sets of different modes;

[0013] The compression module is used to perform lossless compression of time series floating point numbers through parameter optimizers according to different categories of time series data sets.

[0014] Preferably, the parameter optimizer is constructed by a heuristic genetic algorithm.

[0015] Preferably, in the parameter optimizer, the initial population length is set to 63, and the statistics of leading 0s are used to perform a heuristic genetic algorithm on the time series data set to obtain the best 8 encoding values ​​from 0 to 64.

[0016] An embodiment of the present invention further provides a data compression system based on parameter optimization, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a data compression method based on parameter optimization when executed by the processor.

[0017] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes a data compression method based on parameter optimization when running.

[0018] The present invention constructs a new parameter optimizer through a heuristic genetic algorithm for lossless compression of time series floating-point numbers, which can adaptively and stably exert excellent performance on different data sets, so as to further improve the compression ratio, compression and decompression speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0020] Figure 1 A flow chart of a data compression method based on parameter optimization according to an embodiment of the present invention;

[0021] Figure 2 This is a logical schematic diagram of a heuristic genetic algorithm parameter optimizer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Embodiment 1:

[0025] like Figure 1 As shown, an embodiment of the present invention provides a data compression method based on parameter optimization, comprising:

[0026] Step S1, obtaining time series data sets of different modes;

[0027] Step S2: According to different categories of time series data sets, lossless compression of time series floating point numbers is performed through a parameter optimizer; and a parameter optimizer is constructed through a heuristic genetic algorithm.

[0028] As an implementation method of the embodiment of the present invention, Figure 2 As shown, step S2 includes:

[0029] Step 21: For the parameter optimizer, first prepare the parameter optimizer by selecting the floating point data in the initial data set and setting D = {d1, d2, ..., d n} is the set of statistical data, each element d i is a double precision floating point number. i With d i-1 Everyone gets a B i =binary(d i )=b i1 b i2 ...b i64 ,b in ∈{0,1}, and perform XOR. For each B i With B i-1 can get their XOR value, When the result is stored in X = {x1, x2, ..., x n}, we can calculate the number of '0's before the first '1' appears, that is, the leading 0, which is called L_count. It should be noted that k is x iThe first '1' position in "1(x i [j] = '0')" is an indicator function, which takes 1 when the jth bit is '0', otherwise it takes 0. i If there is no '1' in the function, the function returns the value of k-1 to count the L_count of each XOR value.

[0030] Step 22. Inside the parameter optimizer, mainly regarding the value of the code of the 8-bit leading 0, a set L_count_code{L1, L2, L3, L4, L5, L6, L7, L8} is set. The first bit is defaulted to '0', so L1=0. All subsequent work is carried out around the 7 values ​​​​except the first bit. The position of these 7 values ​​​​in the 63 is called L_Flag.

[0031] Step 23: In order to obtain a suitable L_Flag, an initial population is created and its length is set to 63. According to the normal double-precision floating point format, the length should be 64. This is because the first bit of the 64-bit encoding length cannot be discarded. Therefore, for the sake of simplicity, the first bit that should be 0 is directly removed by default, leaving 63 bits, and a genome G is set, where G = {g1, g2, ..., g 63}, each g i ∈{0,1}.

[0032] Step 24: insert L_Flag into the set initial population, and directly count the remaining 7 L_Flags. Use these 7 L_Flag values ​​as i n The coordinates in the array are inserted, and the result of the insertion is replaced by '0'. For example, suppose the 7 flags are 8, 10, 12, 14, 16, 18, and 20 respectively. Then i n [8]=0,i n

[10] =0,i n

[12] =0,i n

[14] =0,i n

[16] =0,i n

[18] =0,i n

[20] = 0, and then insert it into the corresponding position in the 63rd position. This value is inserted as the index of the position, and the value of the corresponding position in the individual is '0'. After this, the fitness needs to be calculated.

[0033] Step 25, the fitness function is equivalent to a Loss function. The reason why the value of the optimal fitness is set as small as possible is because of the particularity of the distribution of L_count itself, and this can converge faster to get the desired effect. According to the characteristic distribution of the data set, set min pThe lowest value is '0', and the max p The highest value at position '0', max s is the maximum step length between two '0's. In the subsequent selection operation, the tournament selection method is used, and the individual with the smallest fitness value is selected at the end. In the selection of the crossover process, the multi-point crossover method is used to make the internal convergence speed of the optimizer faster. After adopting other crossover methods and comparing them, the similarity between offspring is greatly reduced, which is more in line with the envisioned offspring diversity. However, there will be certain problems when the strategy is crossover. For example, the number of '0's originally set will change due to the crossover process, which will directly affect the final result. For similar problems, constraints are used to match the number of '0's. First, I={i|child[i]=1} is used to index the position of '1' in the offspring individual, and then the number of '0's is counted to calculate the difference between the actual and expected number of '0's. Finally, in the mutation stage, a random '0' is changed to '1' and '1' is changed to '0'. This operation is somewhat similar to XOR, but this is also done to ensure the diversity of offspring and avoid falling into the situation of local convergence optimality. After that, the obtained individuals are brought into the L_count parameter matrix to conform to the logical rules of the compression algorithm.

[0034] In Table 1, the algorithm “HGA” using parameter optimizer is tested on IoT and two hybrid datasets and compared with Chimp, ACTF, and ACTF. 128 Compare with the best algorithms.

[0035] Table 1

[0036]

[0037] In Table 2, the algorithm “HGA” using the parameter optimizer is tested on the server dataset and compared with Chimp, ACTF, and 128 Compare with the best algorithms.

[0038] Table 2

[0039]

[0040] In Table 3, the parameter optimizer algorithm “HGA” is tested on data sets such as city temperature, wind speed, stock market data, and air pressure, and compared with Chimp, ACTF, and ACTF. 128 Compare with the best algorithms.

[0041] Table 3

[0042]

[0043] Embodiment 2:

[0044] The embodiment of the present invention further provides a data compression device based on parameter optimization, comprising:

[0045] Acquisition module, used to obtain time series data sets of different modes;

[0046] The compression module is used to perform lossless compression of time series floating point numbers through parameter optimizers according to different categories of time series data sets.

[0047] As an implementation method of the embodiment of the present invention, a parameter optimizer is constructed by a heuristic genetic algorithm.

[0048] As an implementation of an embodiment of the present invention, in a parameter optimizer, the initial population length is set to 63, and a heuristic genetic algorithm is performed on the time series data set using statistics of leading 0s to obtain the best 8 encoding values ​​from 0-64.

[0049] Embodiment 3:

[0050] An embodiment of the present invention further provides a data compression system based on parameter optimization, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a data compression method based on parameter optimization when executed by the processor.

[0051] Embodiment 4:

[0052] An embodiment of the present invention further provides a storage medium, on which a computer program is stored, and the computer program executes a data compression method based on parameter optimization when running.

[0053] The embodiments described above are only descriptions of the preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the design spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by ordinary technicians in this field should all fall within the protection scope determined by the claims of the present invention.

Claims

1. A data compression method based on parameter optimization, characterized in that: include: Step S1, obtaining time series data sets of different modes; Step S2: According to different categories of time series data sets, lossless compression of time series floating point numbers is performed through a parameter optimizer.

2. The data compression method based on parameter optimization according to claim 1, characterized in that: Construct parameter optimizer through heuristic genetic algorithm.

3. The data compression method based on parameter optimization according to claim 2, characterized in that: In the parameter optimizer, the initial population length is set to 63, and the statistics of leading 0s are used to perform a heuristic genetic algorithm on the time series data set to obtain the best 8 encoding values ​​from 0 to 64.

4. A data compression device based on parameter optimization, characterized in that: include: Acquisition module, used to obtain time series data sets of different modes; The compression module is used to perform lossless compression of time series floating point numbers through parameter optimizers according to different categories of time series data sets.

5. The data compression device based on parameter optimization according to claim 4, characterized in that: Construct parameter optimizer through heuristic genetic algorithm.

6. The data compression device based on parameter optimization according to claim 5, characterized in that: In the parameter optimizer, the initial population length is set to 63, and the statistics of leading 0s are used to perform a heuristic genetic algorithm on the time series data set to obtain the best 8 encoding values ​​from 0 to 64.

7. A data compression system based on parameter optimization, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the data compression method based on parameter optimization as described in any one of claims 1 to 3 is executed.

8. A storage medium, characterized in that: The storage medium stores a computer program, which executes the data compression method based on parameter optimization according to any one of claims 1 to 3 when running.

Images