Transform-based seriously missing multivariate time sequence interpolation algorithm

Abstract

The invention discloses a Transform-based seriously missing multivariate time sequence interpolation algorithm, and relates to the technical field of data interpolation, the algorithm comprises the following steps: S1, analyzing a database taking time as a sequence to obtain a multivariate time sequence data set, and preprocessing the multivariate time sequence data set to obtain a sample data set; s2, constructing a TransSMITS model, wherein the TransSMITS model comprises a sequence interpolation module and a weighting module; and S3, inputting the sample data set into the TransSMITS model, and training the TransSMITS model to obtain a standard TransSMITS model. According to the transform-based seriously missing multivariate time sequence interpolation algorithm provided by the invention, the problem that the filling precision of an existing model on a seriously missing multivariate time sequence is relatively poor is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data interpolation, and in particular to a Transformer-based severe missing multivariate time series interpolation algorithm. Background Art

[0002] Time series data is crucial in the fields of medicine, sensor networks, meteorology, etc. However, due to equipment failure, transmission errors and other reasons, data collection is often missing. Missing data will reduce the amount of effective information and affect the effective mining of relevant laws. Therefore, it is necessary to process the missing part of the data in order to provide complete and reliable input for its downstream analysis and support subsequent analysis and decision-making.

[0003] Traditional interpolation methods such as interpolation and mean substitution are based on shallow model design and are suitable for linear and smooth time series. However, in the analysis of complex time series with nonlinear characteristics, the accuracy of such methods is significantly reduced because the correlation and dependency between different variables or data are not considered. Machine learning methods attempt to restore missing values ​​using local relationships of data, such as interpolation methods based on K-nearest neighbor algorithms and local least squares interpolation methods. However, in the analysis of nonlinear complex time series with high missing rates, the above methods show significant performance degradation due to the failure to fully consider the correlation and dependency between variables.

[0004] In recent years, deep learning-based methods have shown obvious advantages in learning complex data features, capturing deep dependencies, and processing high-dimensional nonlinear data, significantly improving the performance of missing value interpolation. For example, methods such as the GRU-D model, MRNN, and BRITS show that RNN models have advantages in capturing the correlation between the time dimension and feature dimension of multivariate time series data. However, they still face limitations in dealing with long-term dependencies, complex feature correlations, and gradient vanishing or exploding problems, which in turn affects the model's effective use of the contextual information of the missing position, limiting its performance in the case of high missing rates.

[0005] Therefore, a Transformer-based severely missing multivariate time series interpolation algorithm is urgently needed to solve the problem of poor accuracy of existing models when processing severely missing multivariate time series data in order to improve performance. Summary of the invention

[0006] To solve the above problems, this application proposes the TransSMITS model to address the limitations of existing models in capturing correlations between complex dimensions, processing data with high missing ratios, and maintaining efficient computing capabilities. This model strengthens the learning of long-term dependencies and feature correlations in time series, and achieves efficient and accurate interpolation of severely missing time series data by simulating diverse missing patterns and deepening the network structure; in response to the difficulty of RNN models in effectively utilizing the contextual information of missing positions when there is high missing rate data, this paper constructs a multi-dimensional attention residual encoder to enhance the model's ability to capture the intrinsic correlation of the sequence, effectively alleviate the gradient vanishing problem, and improve the stability and convergence speed of model training.

[0007] In this application, a Transformer-based severe missing multivariate time series interpolation algorithm is provided to effectively solve the problem of poor accuracy of existing models in filling severe missing multivariate time series, including the following steps: S1. Analyze the database with time series as the data set to obtain a multivariate time series data set, and preprocess the multivariate time series data set to obtain a sample data set; S2. constructing a TransSMITS model, wherein the TransSMITS model includes a sequence interpolation module and a weighting module; S3. Input the sample data set into the TransSMITS model and train the TransSMITS model to obtain the standard TransSMITS model.

[0008] Preferably, in S1, the database with time as the sequence is analyzed to obtain a multivariate time series data set, and the multivariate time series data set is preprocessed to obtain a sample data set. The specific contents are as follows: If the sampling of the multivariate time series data set is synchronized, the sample data is directly obtained. If the sampling is not completely synchronized, the multivariate time series data is aligned in the time domain to obtain the sample data; Divide the sample data into training and test sets according to the target ratio, and then divide the target ratio from the training set as the validation set; The target observations are randomly masked in the validation and test sets for evaluation.

[0009] Preferably, the sequence interpolation module is used to interpolate missing multivariate time series data, including a coarse interpolation module and a fine interpolation module; The coarse interpolation module horizontally splices the time series and the mask matrix, projects them through the fully connected layer, and adds them to the position code to obtain a feature matrix; Inputting the feature matrix into a multi-dimensional attention residual encoder, wherein the multi-dimensional attention residual encoder includes a multi-head self-attention mechanism with diagonal zeroing and a deep feed-forward residual network; The fine interpolation module uses the output missing item prediction value of the coarse interpolation module to replace the missing items in the original time series according to the mask matrix to obtain a reshaped time series; The reshaped time series is sequentially passed through the fully connected layer, position encoding, and multi-dimensional attention residual encoder with the same structure as the coarse interpolation module, and then passed through the feedforward network to obtain the fine interpolation output; The fine interpolation module is used to generate an optimized interpolation result according to the preliminary interpolation result and the mask matrix.

[0010] Preferably, the diagonally zeroed multi-head self-attention mechanism is used to identify the complex relationship between different time points in the sequence and effectively capture the time dependency and feature correlation at the time level; The layer feed-forward residual network is used to learn the deep features and long-term dependencies of the data, extract the implicit correlation between the missing data and the observed data, and assist in missing value estimation.

[0011] Preferably, the weighting module comprises: The average of the attention weights of all diagonally zeroed self-attention mechanisms in the multidimensional attention residual encoder in the block; The fully connected layer is used to concatenate the average value of the attention weights with the mask matrix and output the weighted coefficients; The weighted combination module is used to dynamically adjust the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the weighted coefficients, and calculate the output after filling.

[0012] Preferably, the weighting module dynamically adjusts the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series, and calculates the output after filling; The weighting module is used to dynamically adjust the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series data.

[0013] Preferably, the specific contents of inputting the sample data set into the TransSMITS model and training the TransSMITS model to obtain the standard TransSMITS model are: The TransSMITS model adopts an early stopping strategy to prevent overfitting; The TransSMITS model uses mean absolute error (MAE) to constrain the optimization of coarse interpolation, fine interpolation and weighted modules respectively; The TransSMITS model is provided with a loss function; The TransSMITS model calculates evaluation indicators for interpolation results and true values; The evaluation indicators include: MAE, root mean square error RMSE and mean relative error MRE.

[0014] Preferably, the loss function is expressed as: In the formula, is the output result of the coarse interpolation module, is the output result of the fine interpolation module. is the output of the weighted module, is the mask matrix of X (missing values ​​in the X matrix are set to 0 and non-missing values ​​are set to 1, is the mask vector at time t, is the dimension value of X matrix, is the length of the X matrix, For the dimension, For the moment.

[0015] Preferably, the expression of RMSE is: MRE is expressed as: In the formula, X represents the true value, is the interpolation result, and M represents the mask matrix of the sequence.

[0016] In summary, the Transformer-based severe missing multivariate time series interpolation algorithm of the present invention has the following advantages compared with traditional data interpolation technology: 1. High interpolation accuracy. Through the multi-head self-attention mechanism with diagonal zeroing and the deep feedforward residual network, it can effectively capture the long-term dependency and feature correlation of the time series, and deeply mine the multivariate time series features, thereby achieving high-precision interpolation.

[0017] 2. Strong robustness. The weighted module dynamically adjusts the weights of the outputs of the coarse interpolation module and the fine interpolation module, effectively alleviating the overfitting problem and enhancing the model's understanding and utilization of the deep-level features of time series data.

[0018] 3. Wide scope of application: the present invention is applicable to various types of multivariate time series data, including medical, sensor networks, meteorology and other fields.

[0019] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an overall framework diagram of a Transformer-based severe missing multivariate time series interpolation algorithm of the present invention; Figure 2This is a diagram of a self-attention mechanism for setting diagonal zeros in a Transformer-based severe missing multivariate time series interpolation algorithm according to the present invention.

[0021] Figure 3 It is the interpolation performance diagram of the TransSMITS model at different missing rates of Air-Quality.

[0022] Figure 4 is a time series interpolation visualization of the dataset. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below by means of the accompanying drawings and embodiments. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​described in these embodiments do not limit the scope of the present application.

[0024] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0025] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0026] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0027] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0028] This application proposes a Transformer-based algorithm for interpolating severely missing multivariate time series. Aiming at the limitations of existing models in capturing correlations between complex dimensions, processing high missing ratio data, and maintaining efficient computing capabilities, the TransSMITS model is proposed. The TransSMITS model strengthens the learning of long-term dependencies and feature correlations of time series, and achieves efficient and accurate interpolation of severely missing time series data by simulating diverse missing patterns and deepening the network structure.

[0029] S1, analyzing the database with time as the series to obtain a multivariate time series data set, and preprocessing the multivariate time series data set to obtain a sample data set; If the sampling of the multivariate time series data set is synchronized, the sample data is directly obtained. If the sampling is not completely synchronized, the multivariate time series data is aligned in the time domain to obtain the sample data; Divide the sample data into training and test sets according to the target ratio, and then divide the target ratio from the training set as the validation set; The target observations are randomly masked in the validation and test sets for evaluation.

[0030] Three public datasets were obtained, including the 2012 PhysioNet Mortality Prediction Challenge dataset (PhysioNet2012), the Intel Berkeley Research Laboratory Sensor Dataset (Inter-Sensor), and the Beijing Multi-point Air Quality Dataset (Air-Quality). The PhysioNet2012 dataset and the Inter-Sensor dataset were selected by dividing the training and test sets in an 8:2 ratio, and then 20% of the training set was divided as the validation set; 10% of the observations in the validation and test sets were randomly masked for evaluation; the data from 2014 / 11-2017 / 02 in the Air-Quality dataset were selected as the training set, the data from 2014 / 01-2014 / 10 as the validation set, and the data from 2013 / 03 - 2013 / 12 as the test set.

[0031] In the Inter-Sensor dataset, since the sampling of each sensor is not completely synchronized, it is preprocessed before data division. The 0th and 30th seconds of each minute are selected as the reference points, and the temperature data of all sensors within 15 seconds before and after these two time points are merged to complete the time domain alignment. This paper adds 10% artificial missing values ​​to the aligned dataset, making the total missing rate of the dataset reach 51%.

[0032] S2, constructing a TransSMITS model, wherein the TransSMITS model includes a sequence interpolation module and a weighting module; like Figure 1 and Figure 2 As shown, the sequence interpolation module is used to interpolate missing multivariate time series data, including a coarse interpolation module and a fine interpolation module. Figure 1 a in the figure is the TransSMITS structure diagram. Figure 1 Figure b is the MARE structure diagram.

[0033] The coarse interpolation module horizontally splices the time series and the mask matrix, projects them through the fully connected layer, and adds them to the position code to obtain a feature matrix; Inputting the feature matrix into a multi-dimensional attention residual encoder, wherein the multi-dimensional attention residual encoder includes a multi-head self-attention mechanism with diagonal zeroing and a deep feed-forward residual network; The fine interpolation module uses the output missing item prediction value of the coarse interpolation module to replace the missing items in the original time series according to the mask matrix to obtain a reshaped time series; The reshaped time series is sequentially passed through the fully connected layer, position encoding, and multi-dimensional attention residual encoder with the same structure as the coarse interpolation module, and then passed through the feedforward network to obtain the fine interpolation output; The fine interpolation module is used to generate an optimized interpolation result according to the preliminary interpolation result and the mask matrix. As a learner of the coarse interpolation module, the fine interpolation module forms a series structure, and improves the filling effect of the model by gradually deepening the learning and feature extraction.

[0034] The coarse interpolation module introduces a diagonal zeroing attention mechanism to effectively capture the temporal dependency and feature correlation at the temporal level; a deep feedforward residual network is introduced to extract the implicit correlation between missing data and observed data to assist in missing value estimation.

[0035] The diagonally zeroed multi-head self-attention mechanism is used to identify the complex relationship between different time points in the sequence and effectively capture the time dependency and feature correlation at the time level; The layer feed-forward residual network is used to learn the deep features and long-term dependencies of the data, extract the implicit correlation between the missing data and the observed data, and assist in missing value estimation.

[0036] The weighting module comprises: The average of the attention weights of all diagonally zeroed self-attention mechanisms in the multidimensional attention residual encoder in the block; The fully connected layer is used to concatenate the average value of the attention weights with the mask matrix and output the weighted coefficients; The weighted combination module is used to dynamically adjust the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the weighted coefficients, and calculate the output after filling.

[0037] The weighting module dynamically adjusts the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series, and calculates the output after filling; The weighting module is used to dynamically adjust the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series data.

[0038] The weighted module dynamically adjusts the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series, achieving efficient and accurate interpolation of severely missing time series data.

[0039] S3, training the TransSMITS network TransSMITS is implemented based on PyTorch1.12.1, using the Adam optimizer, with the batch size set to 128, and an early stopping strategy to prevent overfitting: when the loss function value on the validation set does not decrease for 30 consecutive epochs, the model is considered to have converged.

[0040] During the training process, the mean absolute error (MAE) is used to constrain the optimization of coarse interpolation, fine interpolation and weighted modules respectively. The loss function can be expressed as: In the formula, is the output result of the coarse interpolation module, is the output result of the fine interpolation module. is the output of the weighted module, is the mask matrix of X (missing values ​​in the X matrix are set to 0 and non-missing values ​​are set to 1, is the mask vector at time t, is the dimension value of X matrix, is the length of the X matrix, For the dimension, For the moment.

[0041] The interpolation results and the true values ​​are evaluated by calculating the indicators, including MAE, Root Mean Square Error (RMSE) and Mean Relative Error (MRE). RMSE is expressed as: MRE is expressed as: In the formula, X represents the true value, is the interpolation result, and M represents the mask matrix of the sequence.

[0042] On the three data sets, the relevant ablation experimental results of each module of the method of the present invention are shown in Table 1, which shows the ablation experimental data of the multi-head self-attention mechanism with diagonal zeroing, the number of encoders, and the necessity of weighted modules. The comparison results of the method of the present invention and the most advanced method are shown in Table 2, which shows the interpolation performance of the model on the three data sets; Figure 3 In order to explore the impact of different missing rates, different masking ratios of {10%, 20%, ..., 80%, 90%} were used in the Air-Quality dataset to carry out interpolation experiments on data with different missing rates. Figure 3 (a) is the mean absolute error between the predicted value and the true value. Figure 3 (b) is the root mean square error between the predicted value and the true value. Figure 3(c) is the average relative error between the predicted value and the true value, MRNN is a multi-directional recurrent neural network (M-RNN), BRITS is a bidirectional recurrent neural network (BRITS), Transform is a time series interpolation model based on the self-attention mechanism, which uses its powerful sequence modeling ability to handle the missing value problem in multivariate time series data, SAITs is a SOTA interpolation model that relies entirely on self-attention, and TransSMITs is a Transformer-based multivariate time series interpolation algorithm with severe missing values ​​proposed in this paper. Figure 4 Some visualization results are shown, including Figure 4 (a) is an example of interpolation on PhysioNet-2012. Figure 4 (b) is the interpolation result on Inter-Sensor. Figure 4 (c) is the interpolation result on Air-Quality.

[0043] The experimental environment of the present invention is configured as follows: the computer processor is Intel(R) Xeon(R) W-2175 CPU @2.50GHz, 64GB running memory, NVIDIA GeForce RTX 2080Ti GPU, and PyTorch framework.

[0044] Table 1

[0045] Table 2

[0046] Example: The specific implementation of the present invention is described by taking the PhysioNet2012 dataset as an example.

[0047] Data preprocessing: The PhysioNet2012 dataset was divided into training and test sets in an 8:2 ratio, and 20% of the training set was divided as the validation set; 10% of the observations in the validation set and the test set were randomly masked for evaluation.

[0048] Model training: Adam optimizer is used, batch size is set to 128, and early stopping strategy is adopted to prevent overfitting.

[0049] Model evaluation: MAE, RMSE and MRE are used to evaluate the interpolation performance of the model.

[0050] Result analysis: The interpolation performance of TransSMITS on the PhysioNet2012 dataset is better than that of other comparison methods, which shows that the present invention can effectively solve the problem of poor accuracy of existing models in filling multivariate time series with serious missing data.

[0051] Further description of the embodiments: The sequence interpolation module, reshaping module and weighting module in the present invention can be adjusted and optimized according to specific application scenarios. For example, the appropriate number of encoders, number of network layers and activation function can be selected according to the characteristics of the data set.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A Transformer-based algorithm for interpolation of severely missing multivariate time series, characterized in that: The following steps are involved: S1. Analyze the database with time series as the data set to obtain a multivariate time series data set, and preprocess the multivariate time series data set to obtain a sample data set; S2. constructing a TransSMITS model, wherein the TransSMITS model includes a sequence interpolation module and a weighting module; S3. Input the sample data set into the TransSMITS model and train the TransSMITS model to obtain the standard TransSMITS model.

2. According to claim 1, a Transformer-based severe missing multivariate time series interpolation algorithm is characterized in that: In S1, the database with time as the series is analyzed to obtain a multivariate time series data set, and the multivariate time series data set is preprocessed to obtain a sample data set. The specific contents are as follows: If the sampling of the multivariate time series data set is synchronized, the sample data is directly obtained. If the sampling is not completely synchronized, the multivariate time series data is aligned in the time domain to obtain the sample data; Divide the sample data into training and test sets according to the target ratio, and then divide the target ratio from the training set as the validation set; The target observations are randomly masked in the validation and test sets for evaluation.

3. According to claim 2, a Transformer-based severe missing multivariate time series interpolation algorithm is characterized in that: The sequence interpolation module is used to interpolate missing multivariate time series data, including coarse interpolation module and fine interpolation module; The coarse interpolation module horizontally splices the time series and the mask matrix, projects them through the fully connected layer, and adds them to the position code to obtain a feature matrix; Inputting the feature matrix into a multi-dimensional attention residual encoder, wherein the multi-dimensional attention residual encoder includes a multi-head self-attention mechanism with diagonal zeroing and a deep feed-forward residual network; The fine interpolation module uses the output missing item prediction value of the coarse interpolation module to replace the missing items in the original time series according to the mask matrix to obtain a reshaped time series; The reshaped time series is sequentially passed through the fully connected layer, position encoding, and multi-dimensional attention residual encoder with the same structure as the coarse interpolation module, and then passed through the feedforward network to obtain the fine interpolation output; The fine interpolation module is used to generate an optimized interpolation result according to the preliminary interpolation result and the mask matrix.

4. The Transformer-based severe missing multivariate time series interpolation algorithm according to claim 3, characterized in that: The diagonally zeroed multi-head self-attention mechanism is used to identify the complex relationship between different time points in the sequence and effectively capture the time dependency and feature correlation at the time level; The layer feed-forward residual network is used to learn the deep features and long-term dependencies of the data, extract the implicit correlation between the missing data and the observed data, and assist in the estimation of missing values.

5. According to claim 4, a Transformer-based severe missing multivariate time series interpolation algorithm is characterized in that: The weighting module comprises: The average of the attention weights of all diagonally zeroed self-attention mechanisms in the multidimensional attention residual encoder in the block; The fully connected layer is used to concatenate the average value of the attention weights with the mask matrix and output the weighted coefficients; The weighted combination module is used to dynamically adjust the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the weighted coefficients, and calculate the output after filling.

6. The Transformer-based severe missing multivariate time series interpolation algorithm according to claim 5, characterized in that: The weighting module dynamically adjusts the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series, and calculates the output after filling; The weighting module is used to dynamically adjust the weights of the outputs of the coarse interpolation module and the fine interpolation module according to the missing information and time correlation of the time series data.

7. The Transformer-based severe missing multivariate time series interpolation algorithm according to claim 6, characterized in that: The specific contents of the standard TransSMITS model obtained by inputting the sample data set into the TransSMITS model and training the TransSMITS model are as follows: The TransSMITS model adopts an early stopping strategy to prevent overfitting; The TransSMITS model uses mean absolute error (MAE) to constrain the optimization of coarse interpolation, fine interpolation and weighted modules respectively; The TransSMITS model is provided with a loss function; The TransSMITS model calculates evaluation indicators for interpolation results and true values; The evaluation indicators include: MAE, root mean square error RMSE and mean relative error MRE.

8. The Transformer-based severe missing multivariate time series interpolation algorithm according to claim 7, characterized in that: The expression of the loss function is: In the formula, is the output result of the coarse interpolation module, is the output result of the fine interpolation module. is the output of the weighted module, The mask matrix of X is set to 0 and the non-missing values ​​are set to 1. is the mask vector at time t, is the dimension value of X matrix, is the length of the X matrix, For the dimension, For the moment.

9. The Transformer-based severe missing multivariate time series interpolation algorithm according to claim 6, characterized in that: The expression of RMSE is: MRE is expressed as: In the formula, X represents the true value, is the interpolation result, and M represents the mask matrix of the sequence.

Images