Time sequence data compression method based on full training of double autoregression models

Abstract

The invention relates to the technical field of data compression and storage, in particular to a time sequence data compression method based on full training of double autoregression models, which comprises the following steps of: establishing a prediction model; the application of the double-autoregression algorithm comprises the following steps: reducing a prediction error through two times of autoregression modeling, further modeling the autocorrelation of an error term, and optimizing a parameter set through maximum likelihood estimation; according to the dynamic residual threshold mechanism, residual changes are calculated based on total data, a dynamically-adjusted residual threshold judgment mechanism is introduced, and when the residual exceeds a dynamically-adjusted threshold, model parameter updating is triggered; storing data and parameters; recovering the decoding end; and cyclically executing the steps. According to the method, the prediction error is remarkably reduced through full data training, the accuracy of compression and decoding is improved, the prediction precision is improved, the dual-autoregression algorithm is adopted for data prediction, the balance between the precision and the algorithm complexity is obtained, meanwhile, the method is suitable for various types of time sequence data sets, and the universality is high.

Description

Technical Field

[0001] The present invention relates to the technical field of data compression and storage, and specifically to a time-series data compression method based on full training of a double autoregressive model. Background Art

[0002] The time-series data generated by the operation of network devices is the data generated by recording events changing over time, which can reflect the changes in the network state. Against the background of the rapid popularization and application of intelligent interconnection technologies such as the Internet of Things and the Industrial Internet of Things, the sharp increase in the number of intelligent devices has led to an explosive growth of time-series data. Subsequently, the problem of storing and processing a large amount of time-series data has put forward new requirements for the storage capacity of time-series databases.

[0003] In the research on time-series database storage technology, data compression is a fundamental and important issue. On the one hand, time-series data compression can not only reduce the storage requirements of the compressed data, but also reduce the resource costs required for data transmission; on the other hand, a good data compression method can simplify the data mining process, thereby improving the efficiency and reliability of data processing. Therefore, major database software such as InfluxDB, TDengine, Prometheus, and OpenTSDB have all carried out in-depth research on time-series data.

[0004] During the data writing phase in InfluxDB, data is first stored in memory. At this time, the compression is relatively lightweight and relies on the RLE (Run-Length Encoding) algorithm for compression. The RLE algorithm is not limited to specific data types but is a compression algorithm based on data content. It mainly compresses repeated data fields in a data sequence. It achieves compression by replacing consecutive occurrences of the same field with two parts: one is the value that appears, and the other is the number of consecutive occurrences of that value, thus reducing the storage space. When the data needs to be stored on disk (data persistence phase), it uses the TSM (Time-Structured Merge Tree) file format for complex data compression. The Merge Tree divides the data into multiple partitions, and each partition is an ordered data set. These partitions may be divided according to the data volume or time range. During querying, the Merge Tree can effectively merge and retrieve these partitions to achieve efficient query operations. In the TSM file format, InfluxDB uses different compression methods for different data types. For timestamps, it uses differential encoding (delta algorithm), stores the first timestamp of the time series, and then stores the difference between subsequent timestamps and the previous one. This method takes advantage of the continuity and incrementality of timestamps in time series data, and most differences can be represented with fewer bits. The compression of integer data also uses a differential encoding similar to that of timestamps, storing the difference between subsequent values and the previous one. For floating-point numbers, InfluxDB uses Facebook's Gorila compression algorithm. It first stores the complete floating-point number of the first value, and then for subsequent values, it only stores the bits that have changed compared to the previous value. Therefore, this method can effectively reduce the required storage space. For string data, InfluxDB uses a dictionary-based compression method. It first constructs a string dictionary that maps each unique string to a unique ID. Then, when storing data, it only stores these IDs instead of the complete strings. This method is particularly effective when the data set contains a large number of repeated strings. Boolean values generally have only two states, true and false. InfluxDB uses bit compression technology for them, that is, each boolean value only needs to occupy one bit to represent, where 0 represents false and 1 represents true. Generally speaking, InfluxDB utilizes multiple compression algorithms and data structures to ensure efficient compression when storing time series data and reduce the consumption of storage space.

[0005] Prometheus maintains the same compression strategy during the writing and persistence phases. Similar to InfluxDB, it uses different compression methods for different data types. Prometheus uses the Gorila compression algorithm for the timestamps, integer, and floating-point values of each time series. For string and boolean data, like InfluxDB, it uses dictionary compression and bit compression methods respectively.

[0006] In contrast to InfluxDB, TDengine uses different compression algorithms for different types of data monomers during the data writing phase and uses an overall compression method during the data persistence phase. When performing monomer compression, TDengine uses the Delta-of-Delta algorithm to compress timestamps. It is a variant of differential encoding. This algorithm first performs a differential encoding on the data to obtain the differences between adjacent data points, and then performs a second differential encoding on the differences to obtain the differences between adjacent differences. For unsigned integers and signed integers, it uses the Simple8B algorithm and Zig-Zag to compress them respectively. The Simple8B algorithm divides 64-bit words into multiple blocks, each block containing a control bit and 7 data bits. The control bit indicates the encoding method of the data stored in the block, and variable-length encoding is usually used here to achieve efficient compression of integer data. The Zig-Zag algorithm reduces the number of bits of an integer and the storage space occupancy by converting a signed integer to an unsigned integer and encoding it through bit operations. When performing overall compression, TDengine uses the LZ4 algorithm to compress the compressed data as undifferentiated binary data. The LZ4 algorithm usually divides the data into fixed-size blocks and calculates the hash value of each segment. If two segments have the same hash value, they may be the same or similar segments. The LZ4 algorithm will further verify whether the data at the current position is really the same as the data at the corresponding position in the hash table to ensure that there are no false alarms for duplicate segments. If the verification is successful, the LZ4 algorithm will record the start position and length of this duplicate segment and replace it with the corresponding pointer. Finally, only the pointer and encoding information need to be stored in the compressed data, thus achieving efficient compression of the data.

[0007] Different from the above databases, OpenTSDB itself does not implement specific compression algorithms, but relies on the compression technologies supported by its underlying database HBase, such as Snappy, LZ4, etc. The Snappy algorithm is a fast compression and decompression algorithm developed by Google. Similar to the LZ4 algorithm, the Snappy algorithm divides the input data into chunks of 64KB in size and uses a hash table to find duplicate segments. If duplicate segments are found in a chunk, the Snappy algorithm replaces these duplicate segments with corresponding pointers and stores the pointers in the compressed data. This process is called dictionary compression. In addition to dictionary compression, Snappy also applies some simple lossless compression methods, such as Huffman coding, etc. These methods help to further reduce the size of the compressed data. Finally, Snappy combines these compressed data chunks and outputs the compressed data. Snappy provides a reasonable compression ratio and excellent compression and decompression speed, and is especially suitable for scenarios that require rapid data processing, such as real-time data processing. The LZ4 algorithm has an advantage in compression speed, but the compression ratio may be slightly lower.

[0008] The present invention explores a way to compress time series data from an idea completely different from the prior art. With the development of machine learning, it has become possible to accurately predict data. Therefore, it is possible to store only the prediction model parameters and the residuals between the prediction data and the original data without storing the original data at all. However, due to the excessively large model parameters of the deep learning prediction model, not only does the size of the stored content not decrease during storage, but the storage pressure will increase instead. Therefore, the present invention uses a double autoregressive model. Since the model parameters can remain unchanged for a long time and the residual values are very small, less space can be used for storage. At the same time, the full training method is used to optimize the model parameters using the complete historical data, significantly improving the prediction accuracy, model stability, and data compression efficiency, and can further reduce the size of the residuals. During decoding, the decoding end constructs a prediction model based on the parameters to generate prediction data, and lossless recovery can be achieved by adding the residuals. Summary of the Invention

[0009] The object of the present invention is to overcome or at least partially solve the above problems, and a time-series data compression method based on full training of a double autoregressive model is proposed. It is divided into two modules: compression encoding and decoding recovery. The compression encoding module is responsible for the compression function of time-series data. It uses a prediction model to predict time-series data, calculates the residual by comparing the predicted data with the original data. Since the parameters of the prediction model are fixed, as the prediction progresses, the phenomenon that the residual increases with time will occur. Therefore, this paper introduces a method of full model training and a mechanism for judging and updating model parameters based on residual thresholds to monitor the change of residuals. When the residual exceeds a certain threshold, it triggers the update of model parameters. This mechanism can effectively cope with the situation when the data distribution or trend changes, and maintain the accuracy and stability of prediction. In order to enable the decoding end to recover the prediction model, some original data in the first window and the prediction model parameters will be stored in the database. At the decoding end, the subsequent time-series data can be predicted using these data and the stored residuals to losslessly recover the original data.

[0010] The encoding module is designed around the prediction module. There are various models for predicting time-series sequences. From complex models such as Transformer and LSTM to simple autoregressive models, all can predict data. Since we need to store model coefficients and prediction residuals, both model coefficients and residuals will affect the compression effect. Complex models have small prediction residuals but many coefficients, while simple models are the opposite. Here, we propose a double-recursive algorithm that can achieve a balance between the number of model coefficients and the size of residuals.

[0011] To achieve the above object, the present invention provides the following technical solution: A time-series data compression method based on full training of a double autoregressive model, including the following steps:

[0012] S1. Establishment of a prediction model: According to the characteristics of time-series data, use all data to train a prediction model to generate an optimized set of initial prediction model parameters, and calculate the residuals between the predicted values and the true values of all data points;

[0013] S2. Application of the double autoregressive algorithm: Reduce the prediction error through two autoregressive modellings. First, represent the data at the current moment as a linear combination of the data at the previous m moments plus an error term, and calculate and determine the initial autoregressive coefficients; further model the autocorrelation of the error term, and optimize the parameter set through maximum likelihood estimation;

[0014] S3. Dynamic residual threshold mechanism: Calculate the change of residuals based on all data, introduce a dynamically adjusted residual threshold judgment mechanism, and trigger model parameter update when the residual exceeds the dynamically adjusted threshold;

[0015] S4. Storage of data and parameters: Store some original data in the initial window, the prediction model parameters optimized by all data, and the residual values;

[0016] S5, Restoration at the decoding end: Extract the prediction model parameters and the initial raw data from the stored data, reconstruct the prediction model, calculate and generate the predicted values using the prediction model, and combine the residual data to achieve lossless restoration of the raw data;

[0017] S6, Loop and execute steps S1 - S5, sequentially process the entire time - series data set until the compression and decoding restoration of all data are completed.

[0018] In a preferred embodiment, the double autoregressive algorithm in step S2 includes the following steps:

[0019] S2.1, First autoregression: Construct a linear combination equation, where the current - moment value is expressed as a linear combination of the previous m - moment values plus an error term, that is:

[0020] a t =φ 1 a t-1 +φ 2 a t-2 +…+φ m a t-m +ε t (1)

[0021] where a t is the current value, φ 1 , φ 2 ,... φ m are autoregressive coefficients, and ε t is the prediction error;

[0022] S2.2, Calculate the initial autoregressive coefficients using the least - squares method. Form a system of equations from equation (1), use T data points in the entire time - series data set to construct the model. All T data points form the training set, and there are T - m data points to be predicted. Predict using the first m data points (past observations). The m past observations and the constant term (prediction error) required for these T - m data points to be predicted form a matrix. The size of the matrix is (T - m)×(m + 1), where each row contains m observations and a constant term. These observations and constant terms form a vector Y. The autoregressive coefficients are obtained through the solution of equation (2):

[0023] X T Xφ=X T Y (2)

[0024] where X T represents the transpose of matrix X;

[0025] S2.3, Second autoregression: Just using the autoregressive method once, the prediction error ε t is relatively large, ε t and ε tis related to the previous value, i.e., ε t has a large autocorrelation. Therefore, the second-order autoregressive method is used to reduce the prediction error. That is, the prediction error term is decomposed into a linear combination of the current error and multiple past errors. Therefore, Equation (1) is modified as follows:

[0026] a t = φ 1 a t-1 + φ 2 a t-2 +…+ φ m a t-m + θ 1 ε t-1 + θ 2 ε t-2 + θ q ε t-q + ε t (3)

[0027] where ε t-q is the error value before the q-th moment. Therefore, the next task becomes to find the optimal coefficients φ 1 ,.., φ m , θ 1 ,.., θ q such that the prediction of a t is as accurate as possible. Since the autocorrelation of ε t is basically removed in the second-order autoregression, in this case, ε t will be a white noise. The optimization of φ 1 ,.., φ m , θ 1 ,.., θ q is completed in two steps;

[0028] Initialization of φ 1 ,.., φ m :

[0029] Replace the part of θ 1 ε t-1 + θ 2 ε t-2 + θ q ε t-q + ε t in Equation (3) with a random number. The φ 1 ,.., φ m , can be estimated by the least squares method Equation (2);

[0030] Initialization of θ 1 ,.., θ q :

[0031] Rewrite Equation (3) as:

[0032] a t = μ + θ 1 ε t-1 + θ 2 ε t-2 + θ q ε t-q + ε t (4)

[0033] μ is a fixed value with respect to the random numbers ε t-1 ,.., ε t-q , ε t ,Equation (4) is similar to Equation (1), and the coefficient θ can still be obtained by the least squares method similar to Equation (2) 1 ,.., θ q ,to obtain an initial parameter set Θ = {φ 1 , φ 2 ,..., φ m , θ 1 , θ 2 ,..., θ q};

[0034] S2.4. Jointly optimize the parameter set through maximum likelihood estimation to maximize the joint probability density of the observed data. During the training phase, the dataset used is {a 1 , a 2 ,..., a T}, and the probability distribution that each a t independently follows is f(a t ). Since a t can be estimated by the parameter Θ, f(a t ) can be denoted as f(a t ; Θ). Use the method of maximum likelihood estimation to find the optimal parameter set Θ, that is, when these parameters are given, the probability of the observed data {a 1 , a 2 ,..., a T} appearing is the largest. Therefore, it is reasonable to define the likelihood function L(Θ) as the product of the joint probability density functions of all observed values:

[0035]

[0036] We want to find the optimal parameter set Θ to maximize Equation (5). Since taking the product requires a large amount of calculation, to simplify the calculation, take the logarithm of the likelihood function to obtain the log-likelihood function as:

[0037]

[0038] Taking the normal distribution as an example, assume that a t follows a normal distribution N(μ, σ 2 with a mean of μ and a variance of σ2 ), if the data does not follow a normal distribution, an appropriate probability distribution can be selected according to the actual distribution of the data to construct the likelihood function. Different probability distributions have different probability density functions. The likelihood function is defined based on these functions, and its probability density function is: Then for the data set {a 1 , a 2 ,..., a T}}, the log-likelihood function is:

[0039]

[0040] From Equation (3), we can obtain:

[0041] According to the definition, we can get:

[0042]

[0043] Substituting Equations (8), (9), and (10) into (7), we can obtain:

[0044]

[0045] Given a t , t = 1,..., m and the initial parameter set Θ, max(lnL(Θ)) is an unconstrained optimization problem. The BFGS algorithm is used to obtain the parameter Θ that maximizes the log-likelihood function k , and the steps are as follows:

[0046] Calculate the gradient: g k = ▽lnL(Θ k );

[0047] Update the parameter: Θ k+1 = Θ K - H K g k ;

[0048] Calculate the gradient change: y k = g k+1 - g k ;

[0049] Calculate the parameter change: s k = Θ k+1 - Θ k ;

[0050] Update the approximation of the Hessian matrix:

[0051] where the element H ij of the Hessian matrix H is defined as: For optimizing the parameter update direction and step size selection during the process.

[0052] Furthermore, since time series data is continuously generated, in the context of full data training, the entire dataset is used to train the prediction model. However, incremental prediction of time-varying data is also required. We divide the dataset into multiple segments (each segment contains data with a window size of m) and input them into the encoder. To capture the temporal dependencies in the data, the first T data points are used for training, and the model coefficients are determined from them to predict the value at the next time point, and these coefficients and residuals are stored.

[0053] Furthermore, as the prediction progresses, the prediction accuracy of the prediction model using the same coefficients will gradually decrease, i.e., ε i becomes larger and larger, which will lead to an increase in the storage space for ε i Therefore, we introduce the concept of the residual threshold T max By controlling the residual to be less than T max to balance the prediction residuals and the number of model coefficient updates, the space occupancy of the model parameters is reduced.

[0054] Specifically, when , directly store the prediction model parameters and the residual values. is the mean value of the residuals in a window. When , the parameters of the model will be refitted to find the optimal parameters in the current state, and the updated prediction model will be used to continue predicting the next window of data, and the model parameters and residual values will be stored.

[0055] It should be noted that during retraining, the decoding end program needs to be started to obtain the T data values required for training the parameters. A counter is used to record the number of predictions. When the mean error is greater than the threshold, the entire process will be repeated, the current value of the counter will be stored, and the counter will be reset to zero. To enable the decoding module to successfully restore the prediction model, the model parameters that need to be saved include the autoregressive term coefficients, moving average term coefficients, counter times data, and the difference between the original data and the predicted data of the prediction model.

[0056] Furthermore, during the decoding process, the prediction model and the predicted data corresponding to the data segment are restored by extracting the stored model parameters, and the original data is losslessly restored by adding the stored residuals. The specific implementation process of the decoding module: First, obtain the initial prediction model parameters from the storage, restore the prediction model, and obtain the data a 1 , a 2 , …, a m of the first sliding window from the storage and input them into the prediction model to obtain the predicted values a' m+1 , a' m+2 , …, and then extract the corresponding residual values εm+1 , ε m+2 , …, recover the true value a corresponding to the time step according to the residual value m+1 , a m+2 , …, and add the true value to the sliding window. Still with the window size of m, perform sliding prediction until the current prediction times are the same as the times recorded by the encoding end stored. It means that the prediction module coefficients are updated. At this time, the coefficients need to be re-extracted, and the above process is looped until the complete data is fully recovered.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The present invention significantly reduces the prediction error through full-scale data training, improves the accuracy of compression and decoding, and improves the prediction accuracy;

[0059] 2. The present invention uses a double autoregressive algorithm for data prediction, estimates the output of the next moment through the linear combination of the current moment value, the previous multi-point values and the error term, and achieves a balance between accuracy and algorithm complexity; 3. The present invention is applicable to various types of time series data sets and has strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is the specific framework diagram of the compression stage of the present invention;

[0061] Figure 2 is the specific framework diagram of the decompression stage of the present invention;

[0062] Figure 3 is the schematic diagram of the algorithm training process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0063] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the protection scope of the present invention.

[0064] Please refer to Figures 1-3 , the present invention provides a time series data compression method based on full-scale data training of a double autoregressive model. In this method, first, a prediction model is trained through full-scale data, and historical data is used to train the model to determine the autoregressive coefficients and prediction errors. During the prediction process, the prediction model predicts future time series data according to the trained autoregressive coefficients and calculates the residuals from the actual observed values. The key to this prediction compression process is to predict data through the autoregressive coefficients of the model, and the obtained residuals and model parameters are stored to achieve compression. The specific steps are as follows:

[0065] Full - scale data training: By using all historical data to train an autoregressive model, the optimal parameters (autoregressive coefficients) of the model are obtained.

[0066] Prediction residual calculation: After the model training is completed, use the autoregressive model to predict future time - series data, and calculate the residual between the predicted value and the actual observed value.

[0067] Compressed storage: Store the autoregressive coefficients of the prediction model and the prediction residuals together to achieve compression of time - series data.

[0068] As Figure 2 shown, the decompression process is to losslessly restore the original time - series data by extracting the stored model parameters and residuals. First, extract the model parameters (including autoregressive coefficients) and residual values from the storage, then use the model parameters for prediction calculation, and add the stored residual values to restore the original data. The specific steps of the decompression process are as follows:

[0069] Extract model parameters: Extract the prediction model parameters (such as autoregressive coefficients) and residuals saved during training from the storage.

[0070] Predict and restore data: Use the extracted model parameters for data prediction, and add the stored residuals to the predicted values to restore the original data.

[0071] Lossless restoration: By gradually restoring the values at each time step, finally restore the complete time - series data to achieve lossless compression.

[0072] As Figure 3 shown, the overall time - series data compression process of the present invention includes two stages: compression and decompression. The whole process realizes data compression through prediction model training, residual calculation, and storage, and performs restoration by extracting the stored parameters and residuals during decompression. The following are the detailed steps of the overall process:

[0073] Data training stage:

[0074] Step (1): Segment the time - series data and use full - scale data for training to calculate the autoregressive coefficients of the prediction model. Let the time - series data be a 1 , a 2 , … a T , and the prediction model can be expressed as: where φ i is the autoregressive coefficient, m is the model order, and φ i can be calculated by methods such as the least - squares method.

[0075] Step (2): Predict the data for each time period and calculate the residual between the predicted value and the actual value:

[0076] Residual εt = a t -a t ', where a t is the actual value and a t ' is the predicted value.

[0077] Step (3): Store the model parameters and the residual values together to reduce the data storage requirements.

[0078] Data decompression phase:

[0079] Step (4): Extract the parameters of the prediction model and the residual values from the storage.

[0080] Step (5): Use the extracted model parameters for data prediction and superimpose the stored residual values to restore the original data. Let the extracted model parameters be φ i , and the stored residual be ε t , then the restored data a t = a t '+ ε t , where

[0081] The compression and decompression are carried out in a loop: As the time-series data is generated and predicted, the same model training and decompression processes are used to continuously compress and decompress new data, realizing continuous data storage and restoration.

[0082] In summary, the present invention uses all the time-series data to train the prediction model. In the initial stage, historical data is utilized to calculate the optimized autoregressive coefficients by methods such as the least squares method and generate the initial set of prediction model parameters. The use of all the data can ensure that the autoregressive coefficients can fully reflect the long-term trend and short-term changes of the data, reduce the influence of outliers on the model, and enable the trained model coefficients to more accurately fit the real data distribution, thereby reducing the prediction error and the residual.

[0083] Although the embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A time series data compression method based on full training of a dual autoregressive model, characterized in that: The following steps are involved: S1. Prediction model establishment: Based on the characteristics of time series data, the prediction model is trained using the full amount of data to generate an optimized initial prediction model parameter set, and the residual between the predicted value and the true value of all data points is calculated; S2. Application of double autoregressive algorithm: reduce the prediction error through double autoregressive modeling. First, the current moment data is represented as a linear combination of the previous m moment data plus the error term, and the initial autoregressive coefficient is calculated and determined. The autocorrelation of the error term is further modeled, and the parameter set is optimized through maximum likelihood estimation. S3, Dynamic residual threshold mechanism: Calculate the residual change based on the full data, introduce a dynamically adjusted residual threshold judgment mechanism, and trigger the model parameter update when the residual exceeds the dynamically adjusted threshold; S4, data and parameter storage: storing part of the original data in the initial window, the prediction model parameters and residual values ​​after optimization with the full amount of data; S5, decoding end recovery: extract prediction model parameters and initial original data from the stored data, reconstruct the prediction model, use the prediction model to calculate and generate prediction values, and combine the residual data to achieve lossless recovery of the original data; S6. Execute steps S1-S5 in a loop, and process the entire time series data set in sequence until the compression and decoding recovery of the entire data are completed.

2. According to claim 1, a time series data compression method based on full training of a dual autoregressive model is characterized in that: The dual autoregressive algorithm of step S2 comprises the following steps: S2.

1. First autoregression: Construct a linear combination equation, where the current moment value is expressed as a linear combination of the previous m moment values ​​plus an error term, that is: a t =φ1a t-1 +φ2a t-2 +…+φ m a t-m +e t (1) where a t is the current value, φ1, φ2, ... φ m is the autoregressive coefficient, ε t is the prediction error; S2.

2. The initial autoregressive coefficient is calculated by the least square method. Formula (1) is used to form an equation system. The T data points in the entire time series data set are used to build the model. All T data points constitute the training set. There are a total of Tm test points. The first m data points (past observations) are used for prediction. The m past observations and constant terms (prediction errors) required for these Tm test points form a matrix. The size of the matrix is ​​(Tm)×(m+1), where each row contains m observations and a constant term. These observations and constant terms form a vector Y. The autoregressive coefficient is obtained by solving formula (2): X T Xφ=X T Y (2) Where X T represents the transpose of matrix X; S2.3, Second autoregression: The forecast error term is decomposed into a linear combination of the current error and the past multi-point errors, so formula (1) is modified as follows: a t =φ1a t-1 +φ2a t-2 +…+φ m a t-m +θ1ε t-1 +θ2ε t-2 +θ q e t-q +e t (3) where ε t-q is the error value before time q; S2.4, through the maximum likelihood estimation, the joint optimization parameter set is used to maximize the joint probability density of the observed data. In the training phase, the data set is {a1, a2, …, a T }, each a t The probability distribution of independent obedience is f(a t ), due to a t can be estimated using the parameter Θ, so f(a t ) can be written as f(a t ; Θ); define the likelihood function L(Θ) as the product of the joint probability density functions of all observations: In order to simplify the calculation, the logarithm of the likelihood function is taken, and the log-likelihood function is obtained as follows: Taking the normal distribution as an example, assuming a t The mean is μ and the variance is σ 2 The normal distribution N(μ,σ 2 ), its probability density function is: Then for the data set {a1,a2,…,a T }, the log-likelihood function is: From formula (3), we can get: According to the definition: Substituting equations (8), (9) and (10) into (7), we can obtain: Given a t , t=1,…,m and the initial parameter set Θ, the BFGS algorithm is used to obtain the parameter Θ that maximizes the log-likelihood function k .

3. The method for compressing time series data based on full training of a dual autoregressive model according to claim 2, characterized in that: The implementation of the dynamic residual threshold mechanism of step S3 includes: S3.

1. Calculate the mean and variance of the residuals in the current window; S3.2, when the residual mean exceeds the preset threshold or the variance exceeds the stability threshold, the model parameter retraining is triggered; S3.

3. Store the updated model parameters and corresponding residual data.

4. The method for compressing time series data based on full training of a dual autoregressive model according to claim 3, characterized in that: The data and parameter storage in step S4 includes: S4.1, the first T original data points in the initial window are used to reconstruct the prediction model at the decoding end; S4.2, storing the autoregressive coefficient set after full data training and optimization; S4.

3. For each data window, record the residual sequence of all data points in the window for restoring the original data at the decoding end; S4.

4. Use a counter to record the number of times the model parameters are updated, and record the timestamp of each update, so that the prediction model parameters can be dynamically adjusted at the decoding end according to the number of updates.

5. The method for compressing time series data based on full training of a dual autoregressive model according to claim 4, characterized in that: The decoding end recovery process in step S5 specifically includes: S5.1, load the initial window data to rebuild the prediction model; S5.2, using a sliding window mechanism to recursively generate prediction values; S5.3, superimposing the stored residuals and predicted values ​​to restore the original data; S5.

4. Dynamically adjust the prediction model parameters according to the update times recorded by the counter. When the prediction times reach the update times recorded by the counter, it indicates that the prediction module coefficients are updated. The updated model parameters are extracted from the storage again, the prediction model is updated, and the prediction and recovery operations are continued until the complete data is fully restored.

6. A time series data compression method based on full training of a dual autoregressive model according to any one of claims 2 to 5, characterized in that: The maximum likelihood estimation adopts the BFGS optimization algorithm, which includes the following steps: Compute the gradient: Update parameter: Θ k+1 =Θ K -H K g k ; Calculate the gradient change: y k =g k+1 -g k ; Calculation parameter changes: s k =Θ k+1 -Θ k ; Update the Hessian matrix approximation: The elements of the Hessian matrix H are ij Defined as:

Images