Time sequence self-supervised representation learning method based on multi-density mask

Abstract

The invention provides a time sequence self-supervised representation learning method based on a multi-density mask, and belongs to the field of representation learning. The invention provides a time sequence self-supervised representation learning method based on multi-density masks, aiming at the problem that high-dimensional time sequence data depends on labels and the problems that a traditional time sequence representation learning method has relatively strong induction deviation and lacks multi-level modeling on a time sequence relationship. The invention innovatively provides a new data enhancement strategy: multi-density mask, the enhancement mode does not introduce induction deviation which is not emphasized and is easy to capture fine-grained information of the sequence, and meanwhile, the context relation of the time sequence can be captured through a cross-view prediction task. And finally, the multi-granularity time dependency relationship of the time sequence can be further modeled through multi-level comparative learning. Therefore, the quality of representation learning is improved. The expression learning framework is applied to downstream tasks of time sequence classification, the purpose of improving the accuracy can be achieved, and the effectiveness of the method is verified through experimental results.

Description

Technical field:

[0001] The present invention relates to a technology of data self-supervised representation learning, and in particular to a time series self-supervised representation learning method based on multi-density mask. Background technology:

[0002] In the context of the big data Internet era, time series data has become an important form of data in today's society. Massive electronic devices record data information of various automation equipment and sensors at a set sampling rate. At present, the research on time series analysis has shown great significance in various fields, such as health care, industrial diagnosis and financial forecasting. Therefore, how to effectively use time series data, perform operations such as denoising, feature extraction, modeling, fitting, regression, classification and clustering on it, and then extract effective patterns from massive complex data has become a problem worthy of in-depth study.

[0003] In order to extract and infer useful information from complex raw time series data, time series representation learning has become an effective learning paradigm. However, most time series data cannot be effectively utilized due to the lack of accurate labels and the expensive and time-consuming manual annotation. Unsupervised representation learning can extract potential representations from complex raw time series data without explicit supervision. As an important branch of unsupervised learning, self-supervised learning uses different proxy tasks to improve the feature extraction ability of the model, thereby learning effective low-dimensional representations.

[0004] Currently, self-supervised learning has achieved remarkable results in the fields of computer vision and natural language processing due to carefully designed proxy tasks. However, time series have their own unique characteristics, such as seasonality, trend and other time-related information, but these proxy tasks designed for CV / NLP do not consider the semantics related to time series data and cannot be directly transferred to time series data. Mainstream representation learning methods use data-specific augmentation strategies to design contrastive learning to learn invariant representations within time series by distinguishing positive and negative samples. Despite their effectiveness and popularity, the invariance assumption may not always hold in real-world scenarios. In addition, when developing data augmentation strategies, too many inductive biases are introduced. At the same time, however, existing contrastive methods usually only focus on a specific level of information, ignoring the multi-granularity temporal patterns of time series.

[0005] To solve the above problems, we proposed an unsupervised representation learning framework for time series. Our framework designs a new data augmentation strategy: multi-density masking. This masking operation does not change the size of the time series and does not introduce too much inductive bias. At the same time, we use masking and cross-prediction tasks to learn the fine-grained information and coarse-grained contextual relationships of the local time series, and further obtain multi-granular representations of the time series through multi-level contrastive learning. Through the above methods, the quality of representation learning can be improved, and the learned feature representations can be applied to downstream tasks such as classification, ultimately achieving the purpose of improving the accuracy of self-supervised time series classification.

[0006] The applications and advantages of the time series self-supervised representation learning method based on multi-density masks are listed as follows.

[0007] Case 1: Applied to human activity recognition. By analyzing the data from wearable devices, different physical activities such as walking, running, cycling, etc. can be identified and applied to health monitoring and fitness tracking.

[0008] Case 2: By analyzing equipment sensor data, abnormal conditions or impending failures of equipment can be detected in advance, and preventive maintenance can be performed to reduce downtime.

[0009] Case 3: Applied to electroencephalogram (EEG) diagnosis, by classifying EEG data, abnormal brain activity can be detected, epileptic seizures, sleep stages, etc. can be identified, assisting neurologists in diagnosis and treatment. However, obtaining high-quality, fully annotated activity datasets is very time-consuming and expensive, and EEG data usually contains multiple channels, each with high-frequency sampling points, resulting in very high data dimensions, which increases the difficulty of processing and analysis. The use of this method can extract potential representations from complex raw time series without relying on data labeling. At the same time, the cross-prediction task of this method can effectively learn the contextual relationship of the time series, and the multi-density mask can capture fine-grained temporal dependencies to obtain a multi-granular representation of the time series, which improves the quality of representation learning, thereby facilitating the execution of downstream tasks and achieving the purpose of improving the accuracy of human activity recognition.

[0010] In summary, the time series self-supervised representation learning method based on multi-density mask can learn the fine-grained time dependency information and contextual relationship of the time series, improve the quality of representation learning, and apply it to the scenario of time series classification to achieve the purpose of improving the accuracy of self-supervised classification. Summary of the invention:

[0011] The purpose of the present invention is to solve the problem of multivariate time series data's dependence on labels and the problem that most representation learning methods have strong inductive bias and lack multi-level modeling of time series relationships, and to provide a time series self-supervised representation learning method based on multi-density masks.

[0012] (I) Technical solution

[0013] The basic idea of ​​the self-supervised representation learning method for time series based on multi-density masking is: on the one hand, through a new data augmentation strategy: the multi-density masking method will not introduce unappreciated inductive biases and easily capture the fine-grained information of the sequence, and capture the contextual relationship of the time series through the cross-view prediction task. On the other hand, through multi-level contrastive learning, the multi-granularity time dependency of the time series can be further modeled, thereby improving the quality of representation learning.

[0014] It includes the following specific steps:

[0015] 1. Data preprocessing

[0016] like Figure 1 As shown in the method flow chart, the first step of the method is to input data and perform data preprocessing. The process of data preprocessing is mainly to standardize the data. In order to ensure that the network can converge well, before the relative importance of each dimension is clear, standardization makes the distribution of each input data similar, so that we do not overly bias towards data with large variance during network training.

[0017] Step 1_1 Load unlabeled data And labeled data

[0018] and The definition of is as follows:

[0019]

[0020]

[0021] U stands for unlabeled and L stands for labeled. represents the i-th sample in the unlabeled data, represents the jth timestamp vector of the i-th sample in the unlabeled data, and T represents the sequence length. U represents the number of unlabeled samples, X U Represents the set of all unlabeled data. Represents the i-th sample data in the labeled data, represents the jth timestamp vector of the i-th sample in the labeled data, Represents the corresponding label value, n L represents the number of labeled samples, X L Represents the set of all labeled data, Y LRepresents the set of all labeled data labels; this paper adopts a self-supervised training method, uses unlabeled data for pre-training, and then uses labeled data to fine-tune the model and train the classifier at the same time.

[0022] Step 1_2 Data normalization. Use the z-score normalization method to normalize the data. The z-score is normalized based on the mean and standard deviation of the original data. The formula is as follows:

[0023]

[0024] where x i represents the i-th value of the time series data, represents the mean of the time series data, and s represents the variance of the time series data. This process is performed on each time series data.

[0025] Step 1_3 divides the dataset into training set, validation set, and test set in a ratio of 6:2:2. The training set is used to train the model, the validation set is used to adjust the model parameters, and the test set is used to test the final performance of the model.

[0026] 2. Unlabeled data model pre-training stage

[0027] Pre-trained models such as Figure 2 As shown in Figure 1, it consists of a transformer encoder E(·), a cross-prediction module, and a contrastive learning module. Unlabeled data is input into the transformer encoder for primary feature extraction, and the model pre-training is guided by cross-prediction proxy tasks and multi-level contrastive learning, thereby improving the model's feature extraction ability and learning an effective low-dimensional representation.

[0028] Step 2_1 uses multi-density mask data enhancement on the training set data, that is, the original time series x i Perform two time step masking with different masking rates to obtain two different masked time series and

[0029] Step 2_2 Input the two different but related mask sequences obtained above into the transformer feature extractor for feature extraction to obtain two views and

[0030] Step 2_3 calculates the prediction loss through the cross-view prediction task, that is, the past of one view (the previous 0-t moment) predicts the future of another view (the next Tt moment). Specifically, we use a linear model To save and The mutual information, where i is the index of the sample and t is the timestamp, represents the features from time 0 to t, represents the feature representation at time t + k (0 < k < K, K = T - t). Then we maximize the dot product of the predicted representation and the true future features of the same sample, while minimizing the dot product with the features of other samples in the batch. That is, we minimize the following loss

[0031]

[0032]

[0033]

[0034] Step 2_4 conducts a consistency comparison on the features extracted by the encoder, reducing the distance between positive samples and increasing the distance between negative samples. Multilevel comparison is adopted, including time-level comparison and instance-level comparison. The time-level loss regards the representations at the same timestamp in the two views as positive samples, while the representations at different timestamps from the same time series are regarded as negative samples. and represent the representations from the same timestamp t after two augmentations. Then the time contrast loss of the i-th sample at timestamp t can be expressed as:

[0035]

[0036] The instance-level loss regards the representations at the same timestamp in the two views of the same sample as positive samples, and the representations of other time series at timestamp t in the same batch as negative samples. Then the instance contrast loss of the i-th time series at timestamp t can be expressed as:

[0037]

[0038] Therefore, the consistency loss is:

[0039]

[0040] Step 2_5 calculates the total loss The formula is as follows:

[0041]

[0042] where λ represents the trade-off parameter of the total loss, ranging from [0, 1].

[0043] Step 2_6 performs gradient backpropagation according to the total loss value in Step 2_4

[0044] ​Step 2_7 determines whether the current number of iterations is less than or equal to 500. If so, jump to step 2_1 to continue training. Otherwise, exit the self-supervised pre-training and enter the fine-tuning stage.

[0045] 3. Model fine-tuning stage

[0046] Fine-tune the training model as Figure 3 As shown in the figure, there are three main modules, feature extractor E(·) and classifier c(·). E(·) has been preliminarily trained in the pre-training stage and has a certain feature extraction capability. In this stage, the feature extractor E(·) is fine-tuned with labeled data, and the classifier c(·) is trained so that the model can classify well.

[0047] Step 3_1 Input the labeled training set data.

[0048] Step 3_2 inputs the training data into the pre-trained transformer feature extractor for feature extraction.

[0049]

[0050] Step 3_3 Input the extracted features into the classifier for label classification:

[0051] Step 3_4 uses the true label and predicted label of the sample to calculate the classification loss through the cross entropy loss function

[0052]

[0053] Step 3_5 Backpropagate classification loss The gradient of is used to fine-tune the model.

[0054] Step 3_6 calculates the validation set accuracy. If there is an improvement, calculate the test set accuracy and record the maximum test accuracy.

[0055] 4. Model testing phase

[0056] The model in the testing phase is Figure 4 As shown in the figure, the validation set and test set data are passed through the feature extractor E(·) and classifier c(·) trained in the fine-tuning phase in turn, and finally the true label is obtained. The purpose of the validation set is to adjust the hyperparameters of the model, and the division of the test set can enable the model to have a better generalization effect.

[0057] Step 4_1 Load the validation set and test set data.

[0058] Step 4_2 Verify data through Figure 4 For example, for the i-th validation set sample First, the data is classified by the feature extractor E(·) and then by the classifier c(·) to obtain the predicted label The formula is as follows:

[0059]

[0060] Step 4_3 compares the predicted label with the true label and calculates the validation set accuracy. If there is an improvement, use the same operation as step 4_2 to calculate the test set accuracy and use the current test accuracy to update the final classification accuracy.

[0061] Step 4_4 determines whether the current number of iterations is less than or equal to 50. If so, jump to step 3_1 to continue training, otherwise end.

[0062] (II) Beneficial effects

[0063] 1. The present invention solves the problem of dependence on a large number of data labels during the training of multivariate time series data, avoids tedious data labeling work and improves training efficiency. At the same time, it can also use labeled data to fine-tune the pre-trained model to obtain an effective self-supervised classification model.

[0064] 2. Compared with traditional self-supervised representation learning, the present invention extracts features more effectively. The time series self-supervised representation learning method based on multi-density mask proposed in the present invention, on the one hand, through a new data enhancement strategy: the multi-density mask method will not introduce unappreciated inductive bias and easily capture the fine-grained information of the sequence, and capture the contextual relationship of the time series through the cross-view prediction task. On the other hand, through multi-level contrastive learning, the multi-granularity time dependency of the time series can be further modeled. Thereby improving the quality of representation learning. Applying it to the scenario of time series classification can achieve the purpose of improving the accuracy of self-supervised classification. Description of the drawings:

[0065] Figure 1 The present invention is a flow chart of the method.

[0066] Figure 2 This is the pre-training model diagram described in the present invention.

[0067] Figure 3 This is the fine-tuning training model diagram described in the present invention.

[0068] Figure 4 This is a diagram of the test model described in the present invention. Specific implementation method:

[0069] In order to explain the technical solution of the present invention more clearly and completely, the present invention is further explained below in conjunction with the accompanying drawings and examples.

[0070] Below is an EEG dataset FingerMovements, each sample has 28 EEG channel records, each record is 500 milliseconds long. The data is downsampled to 100Hz as recommended, so each channel contains 50 observations. This dataset is used as an example for explanation. Figure 1 As shown, the present invention provides a time series self-supervised representation learning method based on multi-density mask, comprising the following steps:

[0071] Step 1 Figure 1 As shown in the method flow chart, the first step of the method is to input data and perform data preprocessing. The specific steps of data preprocessing are as follows:

[0072] Step 1_1 Load the train.pt and test.pt data files in the FingerMovements folder. After loading, 316 training cases and 100 test cases are obtained.

[0073] Step 1_2: Normalize each data using the z-score standardization method. For example, for a time series data x=(-0.00367,-0.00016,-0.00005.-0.00020,0.00163,...) with a length of 50, its mean is 0.0008 and its standard deviation is 0.0019. Substitute each value of the x sequence into the formula Then we get a new sequence x=(-2.37456,-0.49505,-0.43354,-0.51591,0.46342,...) with a length of 50.

[0074] Step 1_3 merges the training data and the test data, and then divides them into training set, validation set, and test set. The number of data in the training set is 250, the validation set is 83, and the test set is 83, with a ratio of 6:2:2. Each data channel is 28 and the length is 50.

[0075] Step 2: Unsupervised data model pre-training phase. The pre-training model is as follows: Figure 2 As shown, the data is input into the transformer encoder for primary feature extraction, and the model pre-training is guided by cross-prediction proxy tasks and multi-level contrastive learning, thereby improving the model's feature extraction ability and learning effective low-dimensional representations. The specific steps are as follows:

[0076] Step 2_1 uses multi-density mask data enhancement on the training set data, that is, the original time series x i Perform two time step masking with different masking rates, such as masking rates 0.3 and 0.5, to obtain two different masking time series and

[0077] Step 2_2 Input the two different but related mask sequences obtained above into the transformer feature extractor for feature extraction to obtain two views and

[0078] Step 2_3 uses the cross-view prediction task, that is, the past of one view (0-t moments before) predicts the future of another view (Tt moments after). The current data set has a data length of 50, and 60% of the total length is selected as the past, that is, the features of 0-30 moments predict the features of the next 20 timestamps of another view. Calculate the prediction loss

[0079]

[0080]

[0081]

[0082] Step 2_4 performs consistency comparison on the features extracted by the encoder to shorten the distance between positive samples and increase the distance between negative samples. Multi-level comparison is adopted, which is divided into time-level comparison and instance-level comparison. That is, the representation of the same timestamp (such as timestamp 0) in two views of the same sample is regarded as a positive sample, and the time-level comparison is the representation of different timestamps (non-timestamp 0) from the same time series as negative samples. Instance-level comparison is the representation of other time series with timestamp t (such as timestamp 0) in the same batch as negative samples. Calculate the time-level loss and instance-level loss:

[0083]

[0084]

[0085] Therefore, the consistency loss is:

[0086]

[0087] Step 2_5 Calculate the total loss The formula is as follows:

[0088]

[0089] Where λ represents the trade-off parameter of the total loss and ranges from [0, 1].

[0090] Step 2_6 Based on the total loss value in step 2_4 Perform gradient backpropagation.

[0091] Step 2_7 determines whether the current number of iterations is less than or equal to 500. If so, jump to step 2_1 to continue training. Otherwise, exit pre-training and enter the fine-tuning stage.

[0092] Step 3: Model fine-tuning phase, fine-tuning the model as follows Figure 3 As shown in the figure, there are two important modules, feature extractor E(·) and classifier c(·). E(·) has been preliminarily trained in the pre-training stage and has a certain feature extraction capability. In this stage, the feature extractor E(·) is fine-tuned with labeled data, and the classifier c(·) is trained so that the model can classify well. The specific steps are as follows:

[0093] Step 3_1 Input the labeled training set data.

[0094] Step 3_2 Input the training data into the pre-trained transformer feature extractor to extract features and obtain potential features

[0095] Step 3_3 Input the extracted features into the classifier for label classification:

[0096] Step 3_4 uses the true label and predicted label of the sample to calculate the classification loss through the cross entropy loss function

[0097]

[0098] Step 3_5 Backpropagate classification loss The gradient of is used to fine-tune the model.

[0099] Step 4: Model testing phase. The model in the testing phase is as follows: Figure 4 As shown in Figure 1, the validation set and test set data are passed through the feature extractor E(·) and classifier c(·) trained in the fine-tuning phase in turn, and then the accuracy is calculated. The specific steps are as follows:

[0100] Step 4_1 Load the validation set and test set data.

[0101] Step 4_2 Verify data through Figure 4 For example, for the i-th validation set sample First, the data is classified by the feature extractor E(·) and then by the classifier c(·) to obtain the predicted label The formula is as follows:

[0102]

[0103] Step 4_3 compares the predicted label with the true label and calculates the validation set accuracy to be 0.5375, which is an improvement over the previous validation accuracy of 0.4417. 0.5375 is taken as the current maximum test accuracy.

[0104] Step 4_4 determines whether the current number of iterations is less than or equal to 250. If so, jump to step 4_1 to continue training, otherwise end.

[0105] The data set used in this example is an electroencephalogram (EEG) data set FingerMovements. The self-supervised classification accuracy, area under the curve AUROC, precision and F1-Score are selected as the classification performance indicators of the algorithm. Compared with the traditional self-supervised representation learning method, the classification performance of the present invention is improved, as shown in Table 1, where MDMRL represents the time series self-supervised representation learning method based on multi-density mask proposed by the present invention. The first row of the table is the evaluation index, and the first column is the comparison method.

[0106] Table 1

[0107]

[0108] The above is a detailed description of the embodiments of the present invention in conjunction with the accompanying drawings, and the specific implementation methods herein are only used to help understand the method of the present invention. For those of ordinary skill in the art, according to the idea of ​​the present invention, changes and modifications can be made in the specific implementation methods and application scopes, so the present invention should not be understood as limiting the present invention.

Claims

1. A self-supervised representation learning method for time series based on multi-density masks, characterized by The method comprises the following steps: Step 1: Data preprocessing, including data standardization and data set division. Step 2: In the model pre-training phase, the model is trained using unlabeled data using a self-supervised time series representation learning method based on multi-density masks. Step 3: Model fine-tuning phase: Use labeled data to fine-tune the model and train an effective classifier. Step 4: Model testing phase: Use the trained model to test the data.

2. According to claim 1, a self-supervised time series representation learning method based on multi-density masking is characterized in that: The data preprocessing module in step 1 comprises the following specific steps: Step 1_1 Load the data file. Step 1_2 Data normalization. Use the z-score normalization method to normalize the data. The z-score is normalized based on the mean and standard deviation of the original data. The formula is as follows: Step 1_3 divides the training set, validation set, and test set into a ratio of 6:2:

2.

3. According to claim 1, a self-supervised time series representation learning method based on multi-density masking is characterized in that: The pre-training module in step 2 includes the following specific steps: Step 2_1 uses multi-density mask data enhancement on the training set data, that is, the original time series is masked with two time steps of different masking rates to obtain two different masked time series. Step 2_2 Input the two different but related mask sequences obtained above into the transformer feature extractor for feature extraction to obtain two views Z m1 and Z m2 . Step 2_3 calculates the prediction loss through the cross-view prediction task, that is, the past of one view predicts the future of another view. Specifically, we use a linear model To save and We then maximize the dot product of the predicted representation with the true future features of the same sample, while minimizing the dot product with other samples in the batch. That is, we minimize the following loss Step 2_4 performs consistency comparison on the features extracted by the encoder, shortens the distance between positive samples, and expands the distance between negative samples. Multi-level comparison is adopted, which is divided into time-level comparison and instance-level comparison. The time-level loss regards the representation of the same timestamp in two views as positive samples, and the representation of different timestamps from the same time series as negative samples. Then the time comparison loss of the i-th time series at timestamp t can be expressed as: The instance-level loss considers the representation of the same timestamp in two views of the same sample as positive samples, and the representation of other time series with timestamp t in the same batch as negative samples. Then the instance contrast loss of the i-th time series at timestamp t can be expressed as: Therefore, the consistency loss is: Step 2_5 Calculate the total loss The formula is as follows: Where λ represents the trade-off parameter of the total loss and ranges from [0, 1]. Step 2_6 Based on the total loss value in step 2_4 Perform gradient backpropagation. Step 2_7 determines whether the current number of iterations is less than or equal to 500. If so, jump to step 2_1 to continue training. Otherwise, exit the self-supervised pre-training and enter the fine-tuning stage.

4. The self-supervised time series representation learning method based on multi-density mask according to claim 1, characterized in that: The fine-tuning module in step 3 includes the following specific steps: Step 3_1 Input the labeled training set data. Step 3_2 inputs the training data into the pre-trained transformer feature extractor for feature extraction. Step 3_3 Input the extracted features into the classifier for label classification. Step 3_4 uses the true label and predicted label of the sample to calculate the classification loss through the cross entropy loss function Step 3_5 Backpropagate classification loss The gradient of is used to fine-tune the model. Step 3_6 calculates the validation set accuracy. If there is an improvement, calculate the test set accuracy and record the maximum test accuracy.

5. The method for self-supervised time series representation learning based on multi-density mask according to claim 1, characterized in that: The model testing module in step 4 includes the following specific steps: Step 4_1 Load the validation set and test set data. Step 4_2 extracts features and classifies the verification data through the test model. Step 4_3 compares the predicted label with the true label and calculates the validation set accuracy. If there is an improvement, use the same operation as step 4_2 to calculate the test set accuracy and use the current test accuracy to update the final classification accuracy. Step 4_4 determines whether the current number of iterations is less than or equal to 50. If so, jump to step 3_1 to continue training, otherwise end.

Images