Multivariate time sequence missing value interpolation method based on self-distillation

Abstract

The invention discloses a multivariate time sequence missing value interpolation method based on self-distillation, and the method comprises the steps: firstly, employing a convolutional neural network and pooling operation, extracting a feature vector of a multivariate time sequence data sample, obtaining a corresponding missing mask matrix, employing the feature vector to supplement a missing value in the missing mask matrix, and obtaining a missing value of the multivariate time sequence data sample; and secondly, splicing the # imgabs1 # and a missing mask matrix, carrying out interpolation by utilizing a multi-head self-attention mechanism, and carrying out normalization to obtain complete data. And then on the basis of the complete data, correcting the interpolation data by using a full connection layer, and carrying out operation in combination with a missing mask matrix and a multivariate time sequence data sample to obtain the interpolation data. And finally, respectively processing complete data and interpolation data of the feature vector # imgabs2 # by using three different full connection layers, and guiding reverse training through self-distillation. The interpolation precision of the network model can be improved, and the quality of the time sequence data is improved.

Description

Technical Field

[0001] The present invention belongs to the field of missing value interpolation of multivariate time series data, and in particular relates to a missing value interpolation method for multivariate time series based on self-distillation. Background Art

[0002] In real life, time series data is ubiquitous in many fields, and its applications in the fields of medicine, meteorology, and finance are very rich, such as stock price prediction, predicting the future status of hospital patients, etc. However, due to failures or errors in various stages of the collection process, such as physical equipment damage, deliberate, and erroneous collection, time series data is often incomplete. For example, the PhysioNet challenge 2012 dataset is a multivariate time series data, which records sample data collected from more than 4,000 patients within 48 hours after admission, with an average missing rate of 80%. Missing data also contains the change pattern of the data. If the missing part is not processed, key information may be missed during data analysis, making it difficult for the established model to extract accurate trend information from the existing data, affecting the prediction accuracy of the model. Therefore, filling the missing part of the data is of great significance for restoring the real data trend, accurately mining the data pattern, and better guiding the subsequent reliability evaluation work.

[0003] According to the characteristics of missing data, it can be divided into the following categories: (1) Missing at random (MAR): missing data in the data set depends on the observed data; (2) Missing completely at random (MCAR): data are independent of external factors, which means that the missing mechanism does not depend on the observed variable of interest or any other variable in the data set; (3) Missing not at random (MNAR): if the missing depends on both the observed variable and the unobserved variable, the data is a MNAR variable. This paper focuses on missing completely at random (MCAR), therefore, data sampling is concentrated on independent missing values, and artificial missing is introduced for comparison with other state-of-the-art methods.

[0004] Interpolation methods are mainly divided into two categories: statistical-based methods and data-driven deep learning methods. In statistical-based methods, missing data are interpolated by studying statistical characteristics and the probability of missing data. However, the interpolation performance of this method may be limited by model assumptions. Due to the lack of learning about the distribution of unobserved data and spatiotemporal correlations in the data set, the authenticity of the interpolation results may be lacking. In recent years, data-driven deep learning methods do not require too many model assumptions, and their powerful fitting and generalization capabilities make them the most commonly used methods for interpolating missing data. However, when the spatiotemporal patterns of the missing data set are more complex, the above methods may face challenges in dealing with highly complex, nonlinear, and non-stationary time series data, and often fail to fully capture these characteristics, making it difficult to obtain satisfactory data interpolation results. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a missing value interpolation method for multivariate time series based on self-distillation, which enhances the representation learning ability of the model by promoting the learning between high-level and low-level semantic information. In the training of the network model, the knowledge learned by the teacher is passed to the students in the way that the teacher teaches the students, so that the students can learn better. At the same time, the knowledge learned by the students can be passed on to the teacher, and different modules of the network guide each other for learning and training. In representation learning, the performance of the model usually depends on the quality of the semantic information extracted from the lower and higher levels. How to effectively promote mutual learning between these two levels without the help of external knowledge is an important guarantee for improving the performance of the model in the model learning process. It can be used in the problem of missing multivariate time series data to reduce errors for tasks such as data analysis and model training.

[0006] The present invention provides a multivariate time series missing value interpolation method based on self-distillation, comprising the following steps:

[0007] Step 1: Based on the multivariate time series data sample, a multi-layer convolutional neural network and pooling operation are used to extract the feature vector f2 of the multivariate time series data sample, and the corresponding missing mask matrix M is obtained. The feature vector f2 is used to supplement the missing values ​​in the missing mask matrix M to obtain the feature vector Specifically:

[0008] (1) Based on multivariate time series data samples Determine its corresponding missing mask matrix

[0009] Specifically, for a multivariate time series data sample It means that at T = {t1, t2, ... t T Data observed during the time For the data X at time t t , contains D features like If the data is missing, the corresponding value of the mask matrix M is missing is 0, if If the data exists, The value of is 1, using express Is there any missing, as shown below:

[0010]

[0011] (2) Based on multivariate time series data samples Through the one-dimensional convolutional network, the feature relationship f1 of the X data is extracted as follows:

[0012] f1=Conv1d(X)

[0013] Conv1d(·) represents a one-dimensional convolution in the time dimension, with f1 as the first feature extracted. The extracted feature f1 is then pooled to remove redundant data to avoid overfitting of the model, thereby reducing parameters and helping the model focus on important features in the data.

[0014] Next, the pooled features are processed again using a convolutional layer to increase the complexity of the features, improve the representation ability of the model, and expand the receptive field to obtain the feature vector f2, as shown below:

[0015] f2 = Conv1d(MaxPool(f1))

[0016] Among them, MaxPool represents the maximum pooling operation.

[0017] (3) Use the same pooling operation to reduce the size and quantity of the data, and use the Hadamard product to operate the missing mask matrix M with X and f2 respectively to cover the missing parts in X and obtain the feature vector As shown below:

[0018] f=MaxPool(f2)

[0019]

[0020] ⊙ represents the Hadamard product, that is, the multiplication of elements in corresponding positions.

[0021] Step 2: Concatenate feature vectors And the missing mask matrix M to remind the model that the input data has observed values ​​and missing values, use the multi-head self-attention mechanism for interpolation, and get the complete data through normalization Specifically:

[0022] (1) The features extracted in step 1 The vector in is concatenated with the missing mask matrix M, and the embedded vector em is obtained through the fully connected layer. After adding the position information, it is processed through the multi-head self-attention mechanism, as shown below:

[0023]

[0024] em=inW e +b e

[0025] p=PE(em)

[0026]

[0027] in, represents the connection operation, W e ,b e is a hyperparameter; PE and FFN represent trigonometric function position encoding and a fully connected feedforward neural network, respectively, and MHA represents a multi-head self-attention mechanism.

[0028] (2) After linear operation, normalization is performed to obtain complete data

[0029]

[0030] Among them, W t ,b t is the corresponding parameter, and BatchNorm1d(·) represents normalization.

[0031] Step 3: Based on complete data Use a simple fully connected layer to correct the interpolated data, combine the missing mask matrix M with the multivariate time series data sample X to calculate the interpolated data Specifically:

[0032]

[0033]

[0034] Among them, W e ,b e is the corresponding hyperparameter.

[0035] Step 4: Transform the feature vector Complete data Interpolation data Three different fully connected layers are used for processing, and reverse training is guided by self-distillation, specifically:

[0036] (1) The feature vector Complete data Interpolation data Use three different fully connected layers to process and obtain knowledge a1, a2, and a3. Specifically:

[0037]

[0038]

[0039]

[0040] Where W1, W2, W3 are the corresponding weights, and b1, b1, b1 are the corresponding biases.

[0041] (2) The training of the network model is guided by the self-distillation operation. The knowledge a1, a2, and a3 obtained by steps 1, 2, and 3 respectively are used for self-distillation. Different modules of the network guide each other for learning and training. Specifically, a3 is used as a teacher to guide the learning of a2, and a2 transfers knowledge to a1.

[0042] According to the network model described in steps 1, 2, and 3, its training loss function is:

[0043] During the self-distillation training process, the distillation loss function of the training is loss kd , specifically:

[0044]

[0045]

[0046] Among them, ||·|| F represents the matrix norm Frobenius norm, a t represents the teacher network, a s represents the student network, α is a hyperparameter, and b is the number of batches.

[0047] Combine the data obtained in steps 2 and 3 to calculate the interpolation loss, and combine it with the distillation loss as the loss function for the entire network model training. Specifically:

[0048]

[0049] in, represents the mean square error.

[0050] The beneficial effects of the present invention are:

[0051] The present invention provides a multivariate time series missing value interpolation method based on self-distillation. For the existing multivariate time series data interpolation missing value, a self-distillation method is used to guide the network model training. The network is trained by combining the distillation loss function and the interpolation loss function. The interpolation accuracy of the network model can be improved, and the deviation and irregularity of the data in the downstream analysis can be avoided. At the same time, the quality of the time series data is improved, and the continuity of the data is maintained, so as to achieve higher analysis and modeling. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of the overall architecture of the present invention;

[0053] Figure 2 It is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0054] The following further describes a multivariate time series interpolation method based on self-distillation proposed by the present invention in conjunction with the accompanying drawings.

[0055] like Figure 1 Shown is an overall flow chart of the method of the present invention.

[0056] Take the Physionet Challenge 2012 dataset as an example. This dataset comes from the desensitized records of 8,000 intensive care patients. Each record consists of approximately 48 hours of multivariate time series data, recording 37 indicators of patients at different time periods during hospitalization, such as respiratory rate, blood sugar, etc. Its missing rate reaches 80%. The specific steps for missing value interpolation of the Physionet Challenge 2012 dataset based on self-distillation are as follows:

[0057] Step 1: Set the batch size to 128 and the time step to 48, read the time series feature vector X: [128, 48, 37], set the artificial random missing rate to 10%, generate the missing mask matrix M: [128, 48, 37], and set the missing value to 0; perform a one-dimensional convolution with a kernel of 1 on the time dimension of the processed feature vector to expand the time step from 48 to 192, and then perform the next one-dimensional convolution with a kernel of 1 after pooling to restore the time step from 192 to 48. After another pooling operation, combine it with the missing mask to obtain a feature vector that supplements the time series semantic information. Finally, a fully connected layer is used to fuse the feature dimension information of the time series to obtain the feature vector a1.

[0058] Step 2: Connect the result of step 1 with the missing mask in the feature dimension to get input: [128, 48, 74], then encode the features through the fully connected layer, expand the number of features from 74 to d_model = 256, then add the position code to the feature vector, stack 5 layers of multi-head attention, interpolate the missing values ​​in the feature vector, and then use the fully connected layer to fuse the feature dimension to restore it to 37, and then normalize it to get the feature vector Finally, a fully connected layer is used to obtain the feature vector a2.

[0059] Step 3: Use the fully connected layer to perform weight fusion on the feature dimension for the result of step 2, and then combine it with the multivariate time series data sample X and perform corresponding operations on the missing mask to obtain the complete data after interpolation. Finally, a fully connected layer is used to obtain the feature vector a3.

[0060] Step 4: Use the self-distillation operation to guide the results a1, a2, and a3 respectively, calculate the distillation loss of the student network and the teacher network, and combine it with the interpolation loss as the loss function for the entire network model training. The model learning rate is set to 1e-3, and the early stopping strategy is adopted for stopping the model training. When the model loss is 50 epochs, the training is stopped. The result obtained in step 3 is used as the final interpolation result of the model.

[0061] The following is a specific application of the method of the present invention, an experiment and analysis of a multivariate time series missing value interpolation method based on self-distillation:

[0062] The application platform is a machine with Nvidia Tesla T4 GPU, 16G video memory and 16G memory. The data set selected is the Physionet Challenge 2012 data set, which is divided into a training set (80%) and a test set (20%). 20% of the samples are separated from the training set as the validation set, and 10% of the observations are randomly eliminated from the validation set and the test set, which are used as the evaluation results.

[0063] Table 1 Performance comparison of interpolation tasks

[0064]

[0065] Table 1 summarizes the evaluation results of the model on the Physionet Challenge 2012 dataset, and each row in the table shows the results of different models performing interpolation tasks on the dataset. Compared with other models (Median, Last, M-RNN, GP-VAE, Transformer, BRITS and SAITS), the interpolation performance of the present invention is better than other methods under different indicators, indicating that the present invention can show better interpolation performance in the absence of this dataset, thereby improving the interpolation accuracy.

[0066] Figure 2 This is a system architecture diagram of a multivariate time series missing value interpolation method based on self-distillation implemented by the present invention. As shown in the figure, first, based on the multivariate time series data samples, multi-layer convolution and pooling are used to extract data features; secondly, a multi-head self-attention mechanism is used to interpolate missing values ​​in the extracted features to obtain complete data; then a simple fully connected layer is used to correct the interpolated complete data to obtain new complete data; finally, a self-distillation operation is used to transfer the knowledge learned by the deep network model to the shallow network, and the distillation loss is combined to guide the network model to train, which can improve the interpolation accuracy of the network model.

Claims

1. A method for interpolating missing values ​​in multivariate time series based on self-distillation, characterized in that: The following steps are involved: Step 1: Use a multi-layer convolutional neural network and pooling operation to extract the eigenvector f2 of the multivariate time series data sample X, and obtain the corresponding missing mask matrix M. Use the eigenvector f2 to supplement the missing values ​​in the missing mask matrix M to obtain the eigenvector Step 2: Concatenate feature vectors And the missing mask matrix M, use the multi-head self-attention mechanism for interpolation, and get the complete data through normalization Step 3: Based on complete data The interpolated data is corrected using the fully connected layer, and the missing mask matrix M is combined with the multivariate time series data sample X to obtain the interpolated data Step 4: Transform the feature vector Complete data Interpolation data Three different fully connected layers are used for processing, and reverse training is guided by self-distillation.

2. The method for interpolating missing values ​​of multivariate time series based on self-distillation according to claim 1, characterized in that: The specific implementation process of step 1 is as follows: Step 1.1: Based on multivariate time series data samples Determine its corresponding missing mask matrix Step 1.2: Based on multivariate time series data samples Through the one-dimensional convolutional network, the feature f1 of the X data is extracted, and then the feature f1 is pooled; the pooled feature is processed again using the convolution layer to obtain the feature vector f2; Step 1.3: Use pooling operation and use Hadamard product to operate the missing mask matrix M with X and f2 respectively to cover the missing part of X and obtain the feature vector f=MaxPool(f2) ⊙ represents the Hadamard product, that is, the multiplication of elements in corresponding positions.

3. The method for interpolating missing values ​​of multivariate time series based on self-distillation according to claim 2, characterized in that: The determination of the corresponding missing mask matrix The specific process is as follows: For a multivariate time series data sample It means that at T = {t1, t2, ... t T Data observed during the time For the data X at time t t , contains D features like If the data is missing, the corresponding value of the mask matrix M is missing is 0, if If the data exists, The value of is 1, using express Is there anything missing? 4. The method for interpolating missing values ​​of multivariate time series based on self-distillation according to claim 3, characterized in that: The specific implementation process of step 2 is as follows: Step 2.1: The features The vector in is concatenated with the missing mask matrix M, and the embedded vector em is obtained through the fully connected layer. After adding the position information, it is processed through the multi-head self-attention mechanism, as shown below: in=inW e +b e p=PE(em) in, represents the connection operation, W e ,b e is a hyperparameter; PE and FFN represent trigonometric position encoding and fully connected feedforward neural network, respectively; MHA represents multi-head self-attention mechanism; Step 2.2: After linear operation, normalization is performed to obtain complete data Among them, W t ,b t is the corresponding parameter, and BatchNorm1d(·) represents normalization.

5. The method for interpolating missing values ​​of multivariate time series based on self-distillation according to claim 4, characterized in that: The final interpolation data is obtained Specifically: Among them, W e ,b e is the corresponding hyperparameter.

6. The method for interpolating missing values ​​of multivariate time series based on self-distillation according to claim 5, characterized in that: The specific implementation process of step 4 is as follows: Step 4.1: Transform the feature vector Complete data Interpolation data Use three different fully connected layers for processing to obtain knowledge a1, a2, and a3: Where W1, W2, W3 are the corresponding weights, and b1, b1, b1 are the corresponding biases; Step 4.2: Based on knowledge a1, a2, and a3, learning and training are performed through self-distillation operation.

7. The method for interpolating missing values ​​of multivariate time series based on self-distillation according to claim 6, characterized in that: The learning training described in step 4.2 is as follows: a3 is used as a teacher to guide the learning of a2, and a2 transfers knowledge to a1. The training loss function is: During the self-distillation training process, the distillation loss function of the training is loss kd , specifically: Among them, ||·|| F represents the matrix norm, a t represents the teacher network, a s represents the student network, α is a hyperparameter, and b is the number of batches; Calculate its interpolation loss and combine it with the distillation loss as the loss function for the entire training: in, represents the mean square error.

Images