Lossy time series data compression and decompression method based on grey model

Abstract

The application discloses a lossy time series data compression and decompression method based on a grey model, and aims at solving the problem that the existing data compression method cannot accurately reflect the development and change characteristics of the compressed sequence segment, and the error between the recovered sequence and the original sequence is large. The exponential function is used to fit the to-be-compressed segment in the time series data, so that the development and change characteristics of the compressed sequence segment are better reflected; in the process of finding the termination boundary of the to-be-compressed sequence segment, the maximum fitting error of the data items in the segment is used as the evaluation index instead of the average fitting error of the segment fitting function, so that the error upper limit of the data items in the recovered sequence can be ensured, and the original sequence can be more accurately recovered.

Description

Technical Field

[0001] This invention relates to data compression and decompression methods, specifically to a lossy time series data compression and decompression method based on a grey model. Background Technology

[0002] With the increasingly widespread application of next-generation information technologies such as the Internet of Things and big data in various fields, people are collecting more and more data from social production and daily life. Time series data is an important form of this data. Time series data typically represents the observation results of a process, and its values ​​are collected from measurements taken at uniform time intervals or a given sampling frequency. X i ∈R m×n The time series is represented as follows: N is the number of observed samples, and m and n are the number of observed values ​​and the number of observed variables, respectively. When n = 1, it is called a univariate time series; when n ≥ 2, it is called a multivariate time series. During time series data acquisition, compression techniques can be used before data transmission to reduce transmission costs, transmitting only the compressed data. When the analysis of the data does not require the exact numerical values ​​of the series but focuses on its waveform shape, lossy compression methods can be used to compress and decompress the acquired time series data.

[0003] Existing lossy sequence data compression methods all use linear function fitting methods. However, linear functions cannot accurately reflect the development and change characteristics of the compressed sequence segments, and the error between the recovered sequence and the original sequence is large. Summary of the Invention

[0004] The purpose of this invention is to provide a lossy time series data compression and decompression method based on a grey model, in order to solve the technical problem that existing data compression methods cannot accurately reflect the development and change characteristics of the compressed sequence segments, and that the recovered sequence has a large error compared with the original sequence.

[0005] To achieve the above objectives, this invention provides a lossy time series data compression method based on a grey model, characterized by the following steps:

[0006] Step 1: For a univariate time series X = [x1, x2, ..., x...] m ] T Perform a translation so that each term of X is non-negative; the translated sequence is denoted as X. ’ ,but:

[0007] X ’ (k) = X(k) + offset;

[0008] Where X(k)=x kLet represent the value of the k-th term in X, 1 ≤ k ≤ m, where m represents the length of the sequence; offset is the shift amount, and:

[0009]

[0010] Where X min It is the minimum value of X;

[0011] Step 2, record X ’ Let the position of the first item in the current uncompressed sequence be k, and the position numbering starts from 1, so the initial k = 1; let the modeling length of the current GM(1,1) model be K, and let the initial value of K be h;

[0012] Step 3: If (k+K-1)≤m, take the current sequence segment to be compressed. for:

[0013]

[0014] Where, x' k For X ’ The value at time k;

[0015] If (k+K-1)>m, then proceed to step 8;

[0016] Step 4, Calculation First-order cumulative generation sequence

[0017]

[0018] in, calculate The nearest neighbor mean generation sequence

[0019]

[0020] in,

[0021] Step 5: Denote the model parameter column as Then the GM(1,1) model The least squares estimated parameters satisfy:

[0022]

[0023] calculate GM(1,1) model restored value

[0024]

[0025] in, e is the natural constant; a and b are the parameters of the GM(1,1) model, respectively;

[0026] Step 6, Calculation and residual sequence

[0027]

[0028] in,

[0029] Step 7: For each like Not greater than the threshold δ t If so, increase the modeling length by (K+1) and return to step 3;

[0030] If it exists Greater than the threshold δ t And K = h, let the current compression term be... Add it to the end of the compressed sequence Y; update the index of the first item of the current uncompressed sequence to (k+1), and return to step 3;

[0031] If it exists Greater than the threshold δ t And K > h, let the current compression term be . a prev and b prev To model the parameters of the GM(1,1) model with length K-1, the current compressed term is added to the end of the compressed sequence Y; the index of the first term of the current uncompressed sequence is updated to (k+).

[0032] K-1), set the modeling length to K=h, and return to step 3;

[0033] Step 8: Create a compressed term (1, 0, 0, x) for each remaining sequence term in turn. ’ k ), where x ’ k The remaining sequence items are processed, and the generated compressed items are added to the end of the compressed sequence Y in sequence; the lossy compression of sequence X is completed, and the compressed sequence Y is the compressed result data.

[0034] Furthermore, step 1 is preceded by:

[0035] Multivariate time series or image data (structured high-dimensional data) can be expanded and converted into univariate time series in a certain order.

[0036] Furthermore, in step 2, h ≥ 5 and is an integer.

[0037] This invention also provides a method for decompressing lossy time series data based on a grey model, characterized by the following steps:

[0038] Step 1: Record the position number of the current compression item as s = 1;

[0039] Step 2: Extract the compressed item with position number s from the compressed sequence Y;

[0040] If its modeling length K = 1, then take the 4th value of the compressed tuple and add it to the sequence. end;

[0041] Otherwise, extract the fourth value of the compressed tuple and add it to the sequence. Finally, a GM(1,1) model is established using the 2nd to 4th terms of the compressed tuple as parameters. Calculate the restored values ​​of the remaining (K-1) terms:

[0042]

[0043] And add them to in sequence. end;

[0044] Step 3: If s is the last item of the compressed sequence Y, the decompression process ends. This is the lossy restored sequence obtained by decompression; otherwise, update the position number s of the currently processed compressed item to s+1 and return to step 2.

[0045] Step 4: Calculate the lossy recovery sequence

[0046]

[0047] Furthermore, step 4 is followed by:

[0048] The lossy recovery sequence Convert the data into multivariate time series or image data (structured high-dimensional data) by reversing the unfolding process.

[0049] The beneficial effects of this invention are:

[0050] 1. This invention uses an exponential function to fit the segments to be compressed in time series data. Compared with compression algorithms that use linear function fitting, this method can better reflect the development and change characteristics of the segments of the compressed sequence. In the process of finding the termination boundary of the segments to be compressed, this invention uses the maximum fitting error of the data items in the segment as the evaluation index, rather than the average fitting error of the segment fitting function. This can ensure the upper limit of the error of the data items in the recovered sequence, and thus can more accurately recover the original sequence.

[0051] 2. This invention applies the grey model to the lossy compression and decompression calculation of time series data, and adaptively compresses as many sequence data items as possible within a specified error range, thereby achieving a better compression ratio. Attached Figure Description

[0052] Figure 1 This is a graphical representation of the example time series (Rock_Train) used in the embodiments of the present invention;

[0053] Figure 2 This is a graphical representation of the example time series (Rock_Train) used in the embodiments of the present invention and its compressed lossy recovery sequence. Detailed Implementation

[0054] To better explain and facilitate understanding of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] A lossy compression and decompression method for time series data based on a grey model is proposed for the compression and decompression calculation of univariate time series data. Let the univariate time series be X = [x1, x2, ..., x...]. m ] T Where X(k)=x k Let represent the value of the sequence at time k, and m represent the length of the sequence, i.e., the number of observations. A "compression term" represents the result of compressing a sequence segment, denoted as a quadruple. Where K is the GM(1,1) modeling length of the sequence segment, and a and b are the parameters of the GM(1,1) model, namely the development coefficient and the grey action amount, respectively. Let Y be the value of the first term of this sequence segment. Let Y be the compressed sequence of X, which is a sequence composed of a series of compressed terms, initially an empty sequence.

[0056] The GM(1,1) model, also known as the grey model, is where G stands for Grey and M stands for Model. The first '1' in parentheses indicates that a first-order differential equation is used for modeling, and the second '1' indicates that there is one variable in the differential equation.

[0057] The lossy time series data compression and decompression method based on the grey model GM(1,1) provided by this invention is now applied to the first time series of Rock_Train from the UCR dataset for compression and decompression. Let this time series be X, with a length of 2844, and graphically represented as follows: Figure 1 As shown.

[0058] Running a lossy compression method on X involves the following steps:

[0059] Step 1: Calculate X min =1.0645, because Xmin Since the offset is greater than 0, the offset is 0, meaning X does not need to be translated. ’ =X.

[0060] Step 2, record X ’ The position of the first item in the current uncompressed sequence is k=1; take h=5, and denote the modeling length of the current GM(1,1) model as K=h=5.

[0061] Step 3: When (k+K-1)≤2844, select the current sequence segment to be compressed based on the position k of the first item of the current uncompressed sequence and the modeling length K. For example, when k=1 and K=5 (unless otherwise specified, all numbers are rounded to 4 decimal places),

[0062] When k = 2841 and K = 5, (k + K - 1) = 2845 > m = 2844, proceed to step 8.

[0063] Step 4, Calculation First-order cumulative generation sequence calculate The nearest neighbor mean generation sequence For example, when k=1 and K=5,

[0064]

[0065] Step 5: Denote the model parameter column as

[0066] Then, from the GM(1,1) model, we have calculate GM(1,1) model restored value

[0067] in, For example:

[0068] When k=1 and K=5

[0069] (Retain 6 significant decimal places).

[0070] calculate The restored GM(1,1) model values ​​are:

[0071]

[0072] Step 6: Calculate the residual sequence:

[0073]

[0074] in, When k=1 and K=5

[0075] Step 7: For each restored value Take the threshold δ t for That is, the residual term does not exceed 5% of the value of the corresponding term in the compressed sequence segment.

[0076] If there is If successful, increase the modeling length to K = K + 1, and return to step 3. For example:

[0077] When k=1 and K=5, δ sub = [δ1, δ2, ..., δ K ] T =[0.6540, 0.6542,

[0078] [0.6839, 0.6726, 0.6650] T .

[0079] Due to all have If the condition is met, increase the modeling length to K=6 and return to step 3.

[0080] If it exists Greater than the threshold δ t And K = 5, let the current compression term be... Add it to the end of the compressed sequence Y; update the index of the first item of the current uncompressed sequence to (k+1), and return to step 3. For example, when k=2571 and K=5, because If there is a residual term that does not meet the threshold condition, and K = 5, then record the current compressed term as (1, 0, 0, 1.6029), add the current compressed term to the end of the compressed sequence Y; update the position of the first term of the current uncompressed sequence term to k = 2572, and move to step 3.

[0081] If it exists Greater than the threshold δ t And K > 5, let the current compression term be . a prev and b prev Set the parameters of the GM(1,1) model with a modeling length of K-1; update the position of the first item of the current uncompressed sequence to (k+K-1), set the modeling length to K=5, and return to step 3. For example, when k=1 and K=160, because There exists a residual term that does not meet the threshold condition, and since K > 5, the current compression term is denoted as (159, -0.00106821,

[0082] 13.702494, 13.0795), add the current compressed item to the end of the compressed sequence Y; update the position of the first item of the current uncompressed sequence to k=160, set the modeling length to K=5, and move to step 3.

[0083] Step 8: At this point, there are fewer than 5 uncompressed items in sequence X, so the GM(1,1) model cannot be used for compression. Therefore, a compressed item (1, 0, 0, x) is created for each remaining data item (3.4716, 3.4359, 2.5912, 1.733). ’ k ), where x ’ k The remaining sequence items are processed sequentially, and the generated compressed items are appended to the end of the compressed sequence Y. Lossy compression of sequence X is complete, and the compressed sequence Y is the compressed result data (where a is retained to 9 decimal places, b, x...). k (Rounded to 6 decimal places), its length is 63:

[0084] [(159, -0.001068212, 13.702494, 13.079500),

[0085] (244, 0.000072456, 15.727255, 15.433100),

[0086] (66, 0.002555210, 14.204234, 14.646700),

[0087] (119, 0.000388768, 13.441516, 12.750000),

[0088] (174, 0.001200470, 11.855699, 12.320000),

[0089] (529, -0.000244958, 9.657067, 10.016300),

[0090] (33, 0.008511368, 10.559402, 10.422800),

[0091] (162, 0.000846075, 7.747732, 8.150100),

[0092] (205, -0.001004968, 7.083086, 7.132500),

[0093] (219，0.000082183，8.585585，8.258200)，

[0094] (182，0.001579834，8.444298，8.170800)，

[0095] (382，0.002685510，5.890205，6.030000)，

[0096] (59，0.008428802，2.068445，2.002300)，

[0097] (25，0.004479795，1.177777，1.186700)，

[0098] (12，-0.029625268，1.064872，1.128500)，

[0099] (1，0.000000000，0.000000，1.602900)，

[0100] (1，0.000000000，0.000000，1.701100)，

[0101] (1，0.000000000，0.000000，1.811900)，

[0102] (1，0.000000000，0.000000，1.899400)，

[0103] (1，0.000000000，0.000000，2.000600)，

[0104] (1，0.000000000，0.000000，2.134900)，

[0105] (1，0.000000000，0.000000，2.270400)，

[0106] (1，0.000000000，0.000000，2.407800)，

[0107] (1，0.000000000，0.000000，2.574300)，

[0108] (1，0.000000000，0.000000，2.737200)，

[0109] (1，0.000000000，0.000000，2.943300)，

[0110] (1，0.000000000，0.000000，3.164700)，

[0111] (1，0.000000000，0.000000，3.388100)，

[0112] (1，0.000000000，0.000000，3.616600)，

[0113] (1，0.000000000，0.000000，3.845400)，

[0114] (1，0.000000000，0.000000，4.095000)，

[0115] (1，0.000000000，0.000000，4.388000)，

[0116] (10，-0.030210365，4.869424，4.675300)，

[0117] (22，-0.006893315，6.477991，6.424400)，

[0118] (20，-0.022118750，8.106231，7.991000)，

[0119] (69，0.001272806，11.787404，12.096300)，

[0120] (29，0.000422077，12.225387，11.422300)，

[0121] (22，0.002886174，10.997935，11.468800)，

[0122] (28，0.023379243，9.885462，9.726300)，

[0123] (12，0.018923555，5.283717，5.219300)，

[0124] (17，0.006415932，4.007613，4.305200)，

[0125] (7，0.013643470，3.599221，3.358600)，

[0126] (1，0.000000000，0.000000，2.897800)，

[0127] (1，0.000000000，0.000000，3.170600)，

[0128] (1，0.000000000，0.000000，3.247500)，

[0129] (1，0.000000000，0.000000，3.116400)，

[0130] (1，0.000000000，0.000000，2.821400)，

[0131] (1，0.000000000，0.000000，2.885700)，

[0132] (1，0.000000000，0.000000，3.355800)，

[0133] (1，0.000000000，0.000000，3.135500)，

[0134] (1，0.000000000，0.000000，2.787200)，

[0135] (1，0.000000000，0.000000，2.590300)，

[0136] (1，0.000000000，0.000000，2.378700)，

[0137] (1，0.000000000，0.000000，3.110600)，

[0138] (1，0.000000000，0.000000，3.411600)，

[0139] (1，0.000000000，0.000000，3.047800)，

[0140] (1，0.000000000，0.000000，3.172500)，

[0141] (1，0.000000000，0.000000，3.606000)，

[0142] (1, 0.000000000, 0.000000, 3.069400),

[0143] (1, 0.000000000, 0.000000, 3.471600),

[0144] (1, 0.000000000, 0.000000, 3.435900),

[0145] (1, 0.000000000, 0.000000, 2.591200),

[0146] (1, 0.000000000, 0.000000, 1.733000)

[0147] The compression rate is then 4 × 63 / 2844 = 8.9%.

[0148] Calculate the lossy recovery sequence from the compressed sequence Y The steps (initially an empty sequence) are as follows:

[0149] Step 1: Record the lossy recovery sequence If empty, the current position number of the compressed item being processed is s = 1.

[0150] Step 2: Extract the compressed term with position index s from the compressed sequence Y. If its modeling length K = 1 (the value of the first term in the compressed term quadruple), then extract the value of the fourth term in the compressed term quadruple and add it to the lossy restoration sequence. End. Otherwise, retrieve the 4th value of the compressed tuple and add it to the end. Finally, a GM(1,1) model is established using the second to fourth terms of the compressed tuple as parameters. Calculate the restored values ​​of the remaining (K-1) terms:

[0151]

[0152] And add them to in sequence. end.

[0153] Step 3: If s is the last item of the compressed sequence Y, the decompression process ends. This is the lossy restored sequence obtained by decompression; otherwise, update the position number s of the currently processed compressed item to s+1 and return to step 2.

[0154] Step 4: Calculate the lossy recovery sequence Since offset = 0, therefore:

[0155]

[0156] Lossy recovery sequence The graphical representation is as follows Figure 2 As shown.

[0157] This invention can be used in applications where the exact numerical values ​​of a sequence are not required during analysis, but the focus is on its waveform shape. By converting multivariate time series or image data into univariate time series data (e.g., stitching together individual univariate time series data or structured high-dimensional data in an image within a certain order), the method of this invention is also applicable to lossy compression and decompression of such data.

Claims

1. A lossy time series data compression method based on a grey model, characterized in that, Includes the following steps: Step 1: Expand the multivariate time series or image data into a univariate time series. Then, convert the univariate time series... Perform a translation, so that Each term is a non-negative value, and the shifted sequence is denoted as . ,but: ； in, express The Middle The value of the item, , Indicates the length of the sequence; Let be the translation amount, and: ； in yes The minimum value; Step 2, record The position of the first item in the current uncompressed sequence is The position numbering starts from 1, so the initial... =1; record the current value. The modeling length is ,initial Take the value h; Step 3, if Take the current sequence segment to be compressed. for: ； in, for middle The value at time; like Then proceed to step 8; Step 4, Calculation First-order cumulative generation sequence : ； in, ;calculate The nearest neighbor mean generation sequence : ； in, ; Step 5: Denote the model parameter column as , ;but Model The least squares estimated parameters satisfy: ； calculate of Model restored value : ； in, ; It is a natural constant; and They are respectively The parameters of the model; Step 6, Calculation and residual sequence : ； in, ; Step 7: For each ,like Not greater than the threshold Then increase the modeling length by Return to step 3; If it exists Greater than the threshold ,and Let the current compression item be... Add it to the compressed sequence End; Update the position of the first item in the current uncompressed sequence to... Return to step 3; If it exists Greater than the threshold ,and Let the current compression item be... , and For modeling length is hour The model's parameters will be used to add the current compression term to the compression sequence. End; Update the position of the first item in the current uncompressed sequence to... Set the modeling length to Return to step 3; Step 8: Create a compression item for each remaining sequence item in turn. ,in This refers to the remaining sequence items currently being processed, and the generated compressed items are added to the compressed sequence in sequence. the end of; sequence Lossy compression is complete, compressed sequence This is the compressed result data.

2. The lossy time series data compression method based on the grey model according to claim 1, characterized in that: In step 2, And it is an integer.

3. A method for decompressing lossy time series data based on a grey model, characterized in that, Includes the following steps: Step 1: Record the position number of the currently processed compression item. ; Step 2, from the compressed sequence The position number to be extracted is The compressed item; If its modeling length Then, retrieve the 4th value of the compressed tuple and add it to the sequence. end; Otherwise, extract the fourth value of the compressed tuple and add it to the sequence. Finally, it is constructed using the values ​​of the 2nd to 4th items of the compressed tuple as parameters. Model Calculate the rest Restoration value of the item : ； And add them to in sequence. end; Step 3, if It is a compressed sequence The last step is to complete the decompression process. That is, the lossy restored sequence obtained by decompression; otherwise, the position number of the currently processed compressed item. Updated to Return to step 2; Step 4: Calculate the lossy recovery sequence : 。 4. The method for decompressing lossy time series data based on a grey model according to claim 3, characterized in that, Step 4 is followed by: The lossy recovery sequence Convert the data into multivariate time series or image data by reversing the unfolding process.

Images