Data compression method based on differential time sequence electric quantity data

Abstract

The invention discloses a data compression method based on differential time sequence electric quantity data, and the method comprises the steps: 1, collecting power utilization data, arranging the power utilization data according to a time sequence, and forming an original data queue; step 2, aligning the original data queue by using a proximity matching method, and performing linear interpolation on missing data; step 3, performing differential processing on the data queue after linear interpolation to obtain time sequence differential data; step 4, performing integer processing on the time sequence differential data; step 5, carrying out Golomb coding on the data after integer processing; step 6, compressing and packaging the data; the problems that in high-precision and high-density data collection in the prior art, in the data storage and transmission process, the data size is too large, and the storage space and the transmission bandwidth are not enough are solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data compression method based on differential time series electrical quantity data. Background Art

[0002] With the improvement of data acquisition accuracy, data capacity has increased significantly compared to low-frequency acquisition, which poses a huge challenge to data transmission and storage.

[0003] Traditional compression algorithms, such as lzma, gzip, and zip compression algorithms, cannot fully utilize the data correlation of continuously changing time series data, and are not friendly to the performance of limited resource processors such as microcontrollers.

[0004] In the existing technology of high-precision and high-density data acquisition, the data storage and transmission process has the problems of excessive data volume and insufficient storage space and transmission bandwidth. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a data compression method based on differential time series power data to solve the problems of excessive data volume, insufficient storage space and transmission bandwidth in the existing technology during high-precision and high-density data acquisition.

[0006] Technical solution of the present invention:

[0007] A data compression method based on differential time series power data, comprising:

[0008] Step 1: Collect electricity consumption data, arrange the electricity consumption data in time sequence, and form a raw data queue;

[0009] Step 2: Align the original data queue using the proximity matching method and perform linear interpolation on the missing data;

[0010] Step 3: Perform differential processing on the data queue after linear interpolation to obtain time series difference data;

[0011] Step 4: integerize the time series difference data;

[0012] Step 5: Perform Golomb coding on the integerized data.

[0013] Step 6: Compress and package the data.

[0014] The electricity usage data in step 1 includes current, voltage, power and timestamp information.

[0015] The proximity matching method described in step 2 aligns the data to the time series points based on the time range threshold and the forward matching method.

[0016] The linear interpolation is performed directly according to the average value or weighted interpolation is performed according to the offset.

[0017] Differentiation processing, obtaining time series difference data includes using difference processing to generate a difference sequence, which is achieved according to the formula:

[0018] Δ t =X t -X t-1 (t≥1).

[0019] The integer processing includes setting the magnification Δ' f =round(Δ f ×10), Δ f For accuracy.

[0020] Golomb coding involves three basic factors: the quotient and remainder of the data, and the compression parameter k. The compression parameter k has an empirical value formula of k≈0.693×|d|, where d is the coefficient:

[0021] |d| is

[0022] The encoding structure is: [unary code][0][fixed-length code][unary code][0][fixed-length code]…[unary code][0][fixed-length code];

[0023] The unary code is determined by the quotient; the length of the fixed-length code is determined by kC len =log2k; If the data is a signed number, a sign bit is fixed before the unary code, 0 represents a positive number, and 1 represents a negative number.

[0024] The compression and packaging of data includes packaging the data, the Golomb coding compression parameter K value, the data packet metadata and the integer compression ratio.

[0025] Beneficial effects of the present invention:

[0026] This invention provides a data compression method based on differential time-series power data. By utilizing a time-series algorithm, an interpolation algorithm, and Golomb coding, this method overcomes the storage space and bandwidth consumption issues associated with high-precision, high-density data sampling. This method, supplemented by proximity matching and linear interpolation, matches and interpolates data, ensuring the data processing capacity of the differential algorithm and addressing its sensitivity to missing data. Furthermore, relying on the data continuity of the time-series algorithm, this method accurately compresses data, significantly reducing processor consumption. Furthermore, it compresses the original bitstream to a ratio of 4:1 or even higher.

[0027] The present invention solves the problems of excessive data volume, insufficient storage space and transmission bandwidth in the existing technology during high-precision and high-density data acquisition and data storage and transmission. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0029] A data compression method based on differential time series power data includes the following steps:

[0030] S1. Collect electricity consumption data;

[0031] Electricity consumption data includes current, voltage, power and timestamp information; at the same time, the data is arranged in time sequence to form a raw data queue.

[0032] S2. Align the original data queue using the proximity matching method and perform linear interpolation on missing data;

[0033] S201, aligning the data to time series points using a proximity matching method, based on a time range threshold and a forward matching method;

[0034] The data queues are aligned by a proximity matching method;

[0035] Q 1:[09:00,09:01,09:03]

[0036] Q 2:[09:00,09:02,09:04]

[0037] Unified time index

[0038] t2 is the time point of the next moment, t1 is the time point of the previous moment, and Δt is the time difference.

[0039] T align=[09:00,09:01,09:02,09:03,09:04].

[0040] S202. Through linear interpolation, directly interpolate the missing points of the sequence output by S201 according to the average value. If required by the configuration file, use the time segmentation method.

[0041] Linear interpolation can use two optional paths, which can be selected according to the configuration. You can directly use the alignment data for average interpolation, or you can use weighted interpolation based on the offset to increase accuracy.

[0042]

[0043] t is the current timestamp, t prev is the timestamp of the previous point, t next is the timestamp of the next point, X t is the current time data, It is the data at the previous time point; It is the data at the next time point.

[0044] S3. Perform differential processing on the data queue output by S2 to obtain time series differential data.

[0045] Differentiating data queues uses differential processing to generate differential sequences, significantly compressing the data length. This is achieved using the following formula:

[0046] Δ t =X t -X t-1 (t≥1)

[0047] For example, based on the queue [120.5, 120.7, 121.0, 121.2, 121.1], the differential queue [+0.2, +0.3, +0.2, -0.1] is obtained. Data 120.5 is X0 of the entire differential queue. Data X0 needs to be packaged into the data packet parameters in step S6 as decoding parameters.

[0048] S4: Integerize the data output from S3 to avoid floating-point calculations and simplify the calculation process. Integerization requires the proper selection of integerization coefficients. Ensure that the extreme values ​​after integerization are not excessively long, generally not exceeding the original data length. If they exceed this, the original data is directly stored. Whenever possible, all decimals are converted to integers. For data with significant variability, the original data is still used to reduce storage size.

[0049] By converting to integer, set the magnification Δ' f =round(Δ f ×10). where Δ f For precision, if the precision is 3 decimal places, then the magnification Δ' f The more decimal places retained, the longer the required code length.

[0050] S5. Perform Golomb encoding on the S4 data to compress the data space.

[0051] The S5 Golomb code consists of three basic factors: the quotient, remainder, and compression parameter k. The compression parameter k has an empirical formula k≈0.693×|d|, where d is a coefficient.

[0052] |d| is

[0053] Columbus code has large absolute value and small absolute value according to evaluation. The small absolute value refers to X i The quotient is much smaller than the parameter k, so the quotient is small, the compressed code is shorter, and the compression rate is higher. On the contrary, a large absolute value means a lower compression rate.

[0054] For the Golomb code, it is divided into 3 parts: first, the unary code, followed by 0, and finally the fixed-length code.

[0055] The encoding structure is as follows: [unary code][0][fixed-length code][unary code][0][fixed-length code]…[unary code][0][fixed-length code].

[0056] The unary code is determined by the quotient. For example, if the differential code is 4 and the parameter k is 3, then the quotient q = 1, and the unary code is 1. If the differential code is 8, then the quotient q = 2, and the unary code is 11. The variable-length portion is 110.

[0057] The length of the fixed-length code is determined by kC len =log2k, for example, if k is 3 or 4, the fixed-length code length is 2; if k is 8, the fixed-length code length is 3.

[0058] If the data is a signed number, a sign bit must be added before the unary code, where 0 represents a positive number and 1 represents a negative number.

[0059] S6. Pack the output data of S5, the Golomb coding parameter K value, the data packet metadata (that is, the initial data x0), and the integer compression ratio to complete the subpackaging of the compressed data.

[0060] The data is packaged into a compressed data frame in a specific format, including a 1-byte header, 2-byte length data, 2-byte identification code, X04 bytes, 1-byte k value, 2-byte magnification, variable-length data, 2-byte CRC, and 1-byte tail.

[0061] The present invention provides a data compression method based on differential time series electrical quantity data, comprising the following steps:

[0062] For data with even growth or changes, the Golomb coding compression effect is better and easier to achieve the expected results. Compared with non-monotonic changing data such as voltage, the Golomb coding compression efficiency is lower.

[0063] Verification requires the following steps:

[0064] Step 1: Start collecting voltage and power parameters of the device. Continue for half an hour. First, transmit the data via LoRa or other bandwidth.

[0065] Step 2: Capture the most complete part of the data and start preparing the algorithm.

[0066] Step 3: Use the proximity matching method to align the two sequences and use the linear interpolation algorithm to fill in the missing data.

[0067] Step 4: Round the sequence to meet the needs of Columbus coding

[0068] Step 5: Evaluate the parameter k of the Golomb coding and compile the compressed data.

[0069] Step 6: Pack the data and complete the compression of a data frame.

Claims

1. A data compression method based on differential time series power data, characterized by: The method comprises: Step 1: Collect electricity consumption data, arrange the electricity consumption data in time sequence, and form a raw data queue; Step 2: Align the original data queue using the proximity matching method and perform linear interpolation on the missing data; Step 3: Perform differential processing on the data queue after linear interpolation to obtain time series differential data; Step 4: integerize the time series difference data; Step 5: Perform Golomb coding on the integerized data. Step 6: Compress and package the data.

2. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: The electricity usage data in step 1 includes current, voltage, power and timestamp information.

3. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: The proximity matching method described in step 2 aligns the data to the time series points based on the time range threshold and the forward matching method.

4. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: The linear interpolation is performed directly according to the average value or weighted interpolation is performed according to the offset.

5. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: Differentiation processing, obtaining time series difference data includes using difference processing to generate a difference sequence, which is achieved according to the formula: Δ t =X t -X t-1 (t≥1)。 6. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: The integer processing includes setting the magnification Δ' f =round(Δ f ×10), Δ f For accuracy.

7. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: Golomb coding involves three basic factors: the quotient and remainder of the data, and the compression parameter k. The compression parameter k has an empirical value formula of k≈0.693×|d|, where d is the coefficient: |d| is The encoding structure is: [unary code][0][fixed-length code][unary code][0][fixed-length code]…[unary code][0][fixed-length code]; The unary code is determined by the quotient; the length of the fixed-length code is determined by kC len =log2k; If the data is a signed number, a sign bit is fixed before the unary code, 0 represents a positive number, and 1 represents a negative number.

8. The data compression method based on differential time series electrical quantity data according to claim 1, characterized in that: The compression and packaging of data includes packaging the data, the Golomb coding compression parameter K value, the data packet metadata and the integer compression ratio.

Images