Data Compression Method and Device, System, and Storage Medium Based on Parameter Optimization

Abstract

The present invention discloses a data compression method, device, system, and storage medium based on parameter optimization, comprising: step S1, obtaining time series datasets of different modes; and step S2, performing lossless compression of time series floating-point numbers using a parameter optimizer based on the different types of time series datasets. The technical solution of the present invention can adaptively and stably deliver excellent performance on different datasets, further improving compression ratios and compression and decompression speeds.

Description

Technical Field

[0001] The present invention belongs to the technical field of data compression, and in particular relates to a data compression method and device, a system, and a storage medium based on parameter optimization. Background Art

[0002] Lossless data compression based on time series floating-point numbers has been developing steadily in recent years. Pfpc is a prediction-based parallel lossless compression algorithm for double-precision scientific simulation data. Pfpc leverages parallel computing to improve overall throughput. The raw data is divided into blocks and distributed across multiple CPU threads, but there is a trade-off between compression ratio and throughput. Gorilla, proposed by Facebook, was first used to store time series data. Gorilla uses the XOR differential representation of floating-point numbers, compressing data by recording the differences between adjacent floating-point numbers rather than their actual values. The Gorilla algorithm significantly reduces the storage of duplicate data, compressing redundant bits of floating-point numbers through XOR and bit-level storage. Chimp is a lossless compression algorithm for floating-point numbers in time series data. Based on Gorilla's workflow, Chimp redesigns its compression strategy to improve the compression ratio when the number of trailing zeros in the XOR residual is less than 6.

[0003] However, current lossless compression algorithms for floating-point numbers in time series struggle to achieve stable results on different types of data sets. Even the Chimp algorithm, which has received much attention in recent years, cannot maintain its advantage on both the current data set and other data sets. Current lossless compression algorithms for floating-point numbers in time series lack sufficient adaptability and robustness. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a data compression method and device, system, and storage medium based on parameter optimization, which can adaptively and stably exert excellent performance on different data sets, thereby further improving the compression ratio, compression and decompression speed.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A data compression method based on parameter optimization, comprising:

[0007] Step S1: Obtain time series data sets of different modes;

[0008] Step S2: Based on different categories of time series data sets, lossless compression of time series floating point numbers is performed using a parameter optimizer.

[0009] Preferably, the parameter optimizer is constructed by a heuristic genetic algorithm.

[0010] Preferably, in the parameter optimizer, the initial population length is set to 63, and the statistics of leading 0s are used to perform a heuristic genetic algorithm on the time series data set to obtain the best 8 encoding values from 0 to 64.

[0011] The present invention also provides a data compression device based on parameter optimization, comprising:

[0012] Acquisition module, used to obtain time series datasets of different modes;

[0013] The compression module is used to perform lossless compression of time series floating-point numbers through parameter optimizers based on different categories of time series datasets.

[0014] Preferably, the parameter optimizer is constructed by a heuristic genetic algorithm.

[0015] Preferably, in the parameter optimizer, the initial population length is set to 63, and the statistics of leading 0s are used to perform a heuristic genetic algorithm on the time series data set to obtain the best 8 encoding values from 0 to 64.

[0016] An embodiment of the present invention also provides a data compression system based on parameter optimization, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a data compression method based on parameter optimization when executed by the processor.

[0017] An embodiment of the present invention further provides a storage medium having a computer program stored thereon. The computer program executes a data compression method based on parameter optimization when running.

[0018] The present invention constructs a new parameter optimizer through a heuristic genetic algorithm for lossless compression of time series floating-point numbers, which can adaptively and stably exert excellent performance on different data sets, further improving the compression ratio, compression and decompression speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0020] Figure 1 This is a flow chart of a data compression method based on parameter optimization according to an embodiment of the present invention;

[0021] Figure 2 This is a logical diagram of a heuristic genetic algorithm parameter optimizer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0023] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] Example 1:

[0025] like Figure 1 As shown, an embodiment of the present invention provides a data compression method based on parameter optimization, comprising:

[0026] Step S1: Obtain time series data sets of different modes;

[0027] Step S2: According to different categories of time series data sets, lossless compression of time series floating point numbers is performed through a parameter optimizer; and a parameter optimizer is constructed through a heuristic genetic algorithm.

[0028] As an implementation method of the embodiment of the present invention, Figure 2 As shown, step S2 includes:

[0029] Step 21: For the parameter optimizer, first prepare the parameter optimizer by selecting the floating point data in the initial data set and setting D = {d1, d2, ..., d n} is the set of statistical data, each element d i For each d i with d i-1 Can get B i =binary(d i )=b i1 b i2 ...b i64 ,b in ∈{0,1}, and perform XOR. For each B i With B i-1 Can get its XOR value, When the result is stored in X={x1,x2,...,x n}, we can calculate the number of '0's before the first '1' appears, that is, the leading 0, which is called L_count. It should be noted that k is x iThe first '1' position in "1(x i [j]='0')" is an indicator function, which takes 1 when the jth bit is '0', otherwise it takes 0. i If there is no '1' in the function, the function returns the value of k-1 to count the L_count of each XOR value.

[0030] Step 22. Within the parameter optimizer, the main focus is on the encoding of the 8-bit leading 0. A set L_count_code {L1, L2, L3, L4, L5, L6, L7, L8} is set. The first bit is set to '0' by default, so L1 = 0. All subsequent work revolves around the 7 values ​​excluding the first bit. The position of these 7 values ​​in the 63 is called L_Flag.

[0031] Step 23: To obtain a suitable L_Flag, an initial population is created with a length of 63. According to the normal double-precision floating-point format, the length should be 64. This is because the first bit of the 64-bit encoding length that is 0 cannot be discarded. Therefore, for the sake of simplicity, the first bit that should be 0 is directly removed by default, leaving 63 bits, and a genome G is set, where G = {g1, g2, ..., g 63}, each g i ∈{0,1}.

[0032] Step 24: Insert L_Flag into the set initial population and directly count the remaining 7 L_Flags. Use these 7 L_Flag values as i n The coordinates in the array are inserted, and the result of the insertion is replaced by '0'. For example, suppose the 7 flags are 8, 10, 12, 14, 16, 18, and 20 respectively. Then i n [8]=0,i n

[10] =0,i n

[12] =0,i n

[14] =0,i n

[16] =0,i n

[18] =0,i n

[20] = 0, and then insert it into the corresponding position in bit 63. This value is inserted as the index of the position, and the value of the corresponding position in the individual is '0'. After this, the fitness needs to be calculated.

[0033] Step 25, the fitness function is equivalent to a Loss function. The reason why the value of the optimal fitness is set as small as possible is because of the particularity of the distribution of L_count itself, and this can converge faster to get the desired effect. According to the characteristic distribution of the data set, set min pThe lowest value is '0', max p The highest value at position '0', max s is the maximum step length between two zeros. In the subsequent selection process, a tournament selection method is used, and the individual with the lowest fitness value is selected at the end. The crossover process uses a multi-point crossover approach, which accelerates convergence within the optimizer. Compared to other crossover methods, the similarity between offspring is significantly reduced, more consistent with the desired offspring diversity. However, this strategy presents certain challenges during crossover. For example, the number of zeros in the original set of seven zeros will change during the crossover process, directly affecting the final result. To address this issue, a constraint is implemented to match the number of zeros. First, the position of the 1 in the offspring individual is indexed using I = {i|child[i] = 1}. The number of zeros is then counted to calculate the difference between the actual and expected number of zeros. Finally, in the mutation phase, a random 0 is converted to a 1, and a 1 to a 0. This operation is somewhat similar to XOR, but it is also implemented to ensure offspring diversity and avoid falling into a local convergence optimum. After that, the obtained individuals are brought into the L_count parameter matrix to conform to the logical rules of the compression algorithm.

[0034] In Table 1, the algorithm “HGA” using parameter optimizer is tested on IoT and two mixed datasets and compared with Chimp, ACTF and 128 Compare with the best algorithms.

[0035] Table 1

[0036]

[0037] In Table 2, the algorithm “HGA” using the parameter optimizer is tested on the server dataset and compared with Chimp, ACTF, and 128 Compare with the best algorithms.

[0038] Table 2

[0039]

[0040] In Table 3, the parameter optimizer algorithm “HGA” is tested on data sets such as city temperature, wind speed, stock market data, and air pressure, and compared with Chimp, ACTF, and ACTF. 128 Compare with the best algorithms.

[0041] Table 3

[0042]

[0043] Example 2:

[0044] An embodiment of the present invention further provides a data compression device based on parameter optimization, comprising:

[0045] Acquisition module, used to obtain time series datasets of different modes;

[0046] The compression module is used to perform lossless compression of time series floating-point numbers through parameter optimizers based on different categories of time series datasets.

[0047] As an implementation method of the embodiment of the present invention, a parameter optimizer is constructed by using a heuristic genetic algorithm.

[0048] As an implementation of an embodiment of the present invention, in a parameter optimizer, the initial population length is set to 63, and a heuristic genetic algorithm is performed on the time series data set using statistics of leading 0s to obtain the best 8 encoding values from 0 to 64.

[0049] Example 3:

[0050] An embodiment of the present invention also provides a data compression system based on parameter optimization, comprising: a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program executes a data compression method based on parameter optimization when executed by the processor.

[0051] Example 4:

[0052] An embodiment of the present invention further provides a storage medium having a computer program stored thereon. The computer program executes a data compression method based on parameter optimization when running.

[0053] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A data compression method based on parameter optimization, characterized in that: include: Step S1: Obtain time series data sets of different modes; Step S2: Based on different categories of time series data sets, lossless compression of time series floating-point numbers is performed using a parameter optimizer; a parameter optimizer is constructed using a heuristic genetic algorithm; in the parameter optimizer, the initial population length is set to 63, and a heuristic genetic algorithm is performed on the time series data set using leading 0 statistics to obtain the optimal 8 encoding values from 0 to 64; Step S2 includes: Step 21: For the parameter optimizer, select the floating point data in the initial data set, and set D = {d1, d2, ..., d n } is the set of statistical data, each element d i is a double-precision floating point number; for each d i with d i-1 Can get B i =binary(d i )=b i1 b i2 ...b i64 ,b in ∈{0,1}, and perform XOR; for each B i With B i-1 Can get its XOR value, When the result is stored in X={x1,x2,...,x n }, calculate the number of '0's before the first '1' appears, which is called L_count. Where k is x i The first '1' position in "1(x i [j]='0')" is an indicator function, which takes 1 when the jth bit is '0', otherwise it takes 0; if x i If there is no '1' in the result, the function returns the value of k-1 to count the L_count of each XOR value; Step 22: Set a set L_count_code {L1, L2, L3, L4, L5, L6, L7, L8}. The first bit is set to '0' by default, so L1 = 0. All subsequent work is carried out around the 7 values ​​excluding the first bit. The position of these 7 values ​​in the 63 is called L_Flag; Step 23: Create an initial population with a length of 63. Remove the first bit that should be 0, leaving 63 bits. Set a genome G, where G = {g1, g2, ..., g 63 }, each g i ∈{0,1}; Step 24: insert L_Flag into the set initial population and directly count the remaining 7 L_Flags; use these 7 L_Flag values as i n The coordinates in the array are inserted, and the result of the insertion is replaced by '0'; Step 25: According to the characteristic distribution of the data set, set min p The lowest value is '0', max p The highest value at position '0', max s is the maximum step length between two '0's; in the selection operation, the tournament selection method is used, and the individual with the smallest fitness value is selected at the end; in the crossover process, the multi-point crossover method is used. First, the position of the '1' in the child individual is indexed by I = {i|child[i] = 1}, and then the number of '0's is counted to calculate the difference between the actual and expected number of '0's; in the mutation stage, a random '0' is changed to a '1', and a '1' is changed to a '0'; the obtained individual is brought into the L_count parameter matrix to conform to the logical rules of the compression algorithm.

2. A data compression device based on parameter optimization for implementing the data compression method based on parameter optimization according to claim 1, characterized in that: include: Acquisition module, used to obtain time series datasets of different modes; The compression module is used to perform lossless compression of time series floating-point numbers through parameter optimizers based on different categories of time series datasets.

3. A data compression system based on parameter optimization, characterized in that: include: A memory and a processor, wherein the memory stores a computer program executed by the processor, and when the computer program is executed by the processor, the data compression method based on parameter optimization according to claim 1 is executed.

4. A storage medium, characterized in that The storage medium stores a computer program, which executes the data compression method based on parameter optimization according to claim 1 when running.

Images