Time sequence analysis method based on double-flow Transform architecture

Abstract

The invention discloses a time sequence analysis method based on a double-flow Transform architecture, and belongs to the technical field of time sequence analysis, and the method comprises the steps: obtaining a multivariable time sequence data set, and carrying out the preprocessing and division of the data set; a TSTNet model is constructed; performing preliminary training on the TSTNet model to obtain a final feature sequence; based on the obtained final feature sequence, a gradient descent method is applied to optimize a loss function, and primary training of a TSTNet model is completed; the TSTNet model is verified; carrying out overall training on the TSTNet model; and after the training is completed, evaluating a final feature sequence output by the TSTNet model by using the test data set to obtain a final loss score of the TSTNet on the data set. By the adoption of the method, the relation between personality and generality between processing channels is clearly processed by building the TSTNet model, and the complex dependency relation between different channels is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of time series analysis, and in particular to a time series analysis method based on a dual-stream Transformer architecture. Background Art

[0002] In the real world, time series data is everywhere. Analysis based on existing time series data is applied to all walks of life, including but not limited to meteorological element forecasting, missing data filling, disease probability prediction, monitoring data anomaly detection and business decision-making in the financial field. Due to its wide range of application value, time series modeling has always received great attention from the academic community.

[0003] Multivariate time series (MTS) has multiple interrelated data dimensions. Unlike text data and video data, time series data must be continuous data recorded on the timeline. Since data at a single moment cannot contain enough semantic information, past methods have focused more on the temporal changes of time series data. In particular, with the development of deep learning, many models have proposed methods to capture cross-temporal dependencies in MTS data. The earliest method used recurrent neural networks (RNNs) to summarize past information using internal states to achieve the purpose of modeling continuous time points. However, as the amount of data increases, the RNN method cannot capture feature information from a long time ago well, and when facing long sequences, the model efficiency becomes very low due to its sequential calculation characteristics. Another method uses a time-dimensional convolutional neural network (TCN) to extract local temporal changes. Due to the limitations of the convolution kernel, its ability to capture long-term dependencies is still insufficient.

[0004] Information across time is extremely important, but the correlation and dependency between channels is also crucial in MTS. The traditional Transformer model projects all channels into a high-dimensional hidden space and uses the attention mechanism to capture the dependencies at different time steps. This method mixes all channels together and fully utilizes the commonalities (correlations) between channels, but due to the extreme commonalities, the individuality (independence) between single channels cannot be reflected. Methods such as DLinear independently process the cross-time information between channels, and different channels do not affect each other. However, this processing method only utilizes the individual characteristics of the channels and completely ignores the common characteristics between channels, so that the hidden information in MTS cannot be fully reflected. Summary of the invention

[0005] The purpose of the present invention is to provide a time series analysis method based on a dual-stream Transformer architecture, which can accurately capture the complex relationship between individual features and common features between channels, help to more fully utilize the information in time series data and thus improve the utilization of relevant time data.

[0006] To achieve the above object, the present invention provides a time series analysis method based on a dual-stream Transformer architecture, comprising the following steps:

[0007] S1. Obtain a multivariate time series data set and preprocess the data set to obtain a training data set, a validation data set, and a test data set;

[0008] S2. Build TSTNet model;

[0009] S3, input the training data set into the TSTNet model for preliminary training to obtain the final feature sequence;

[0010] S4. Based on the final feature sequence obtained, the gradient descent method is used to optimize the loss function to complete the first training of the TSTNet model;

[0011] S5. Use the validation data set for verification, input all group data into the trained TSTNet model, evaluate the final feature sequence output by the TSTNet model, and determine whether the TSTNet model continues to optimize;

[0012] S6, repeat steps S3 to S5 until the evaluation index obtained in step S5 is no longer improved, and the TSTNet model training is completed;

[0013] S7. Using the test data set, input all group data into the trained TSTNet model, evaluate the final feature sequence output by the TSTNet model, and obtain the final loss score of TSTNet on the data set.

[0014] Preferably, the TSTNet model in step S2 includes a stream-specific channel selection module, a dual-stream feature interaction module and a stream-level adaptive fusion module, and the dual-stream feature interaction module includes an explicit network and an implicit network.

[0015] Preferably, in step S3, the training data set is input into the TSTNet model for preliminary training, and the final feature sequence obtained includes:

[0016] S31, randomly select a certain amount of training data, input the training data into the TSTNet model, and perform normalization processing on the input sequence in parallel and independently;

[0017] S32, inputting the normalized input sequence into the stream-specific channel selection module, calculating the correlation scores between different channels through the stream-specific channel selection module, and distinguishing the relationship between the channels according to the values ​​of these scores, obtaining explicit related channels, implicit related channels and irrelevant channels, inputting the explicit related channels and implicit related channels into the explicit network and the implicit network in parallel, and discarding the irrelevant channels;

[0018] S33, the explicit network extracts time domain features based on the explicit correlation channel to obtain explicit correlation time domain features;

[0019] S34, the implicit network extracts features based on the implicit correlation channels and the correlation scores to obtain implicit correlation time domain features;

[0020] S35. The obtained explicit correlated time domain features and implicit correlated time domain features are input into the stream-level adaptive fusion module, and the two feature sequences are adaptively fused to capture the inter-channel dependency to obtain the final feature sequence.

[0021] Preferably, the normalization processing formula in step S31 is:

[0022]

[0023] In the formula, Represents an L×C real number set representing the input sequence, L represents the length of the input sequence, C represents the number of channels of the input sequence, represents a set of real numbers, mean and var represent the mean and variance of the input sequence respectively.

[0024] Preferably, calculating the correlation scores between different channels by the stream-specific channel selection module in step S32 includes:

[0025] Calculate the correlation scores between different channels, the formula is:

[0026]

[0027] In the formula, R i represents the correlation scores between different channels, from The data of the ith channel separated from the C channels; Represents the correlation score between the i-th channel and all channels;

[0028] Select the first k+1 values ​​with the largest scores and use To express;

[0029]

[0030] In the formula, k represents a hyperparameter, Corresponding channel is the dominant correlation channel, Corresponding channel is the implicitly related channel, and the rest are irrelevant channels.

[0031] Preferably, in step S33, the explicit network extracts time domain features based on explicit correlation channels, performs feature analysis on the data of the time dimension based on different scales, and then aggregates features of different scales together to obtain explicit correlation time domain features.

[0032] Preferably, the explicit network consists of an MTE module, and the MTE module formula is:

[0033]

[0034] In the formula, Sampling(*) is the downsampling scale information, Padding(*) is the end padding 0 to achieve complete downsampling, and is the sequence before and after sampling, l is the scale after sampling, Transformer(*) represents a single-layer encoder, Proj(*) is a simple linear layer that maps the captured feature information to the length required for the task, O represents the output of the MTE module, and here represents the output of the explicit module.

[0035] Preferably, in step S34, the implicit network performs feature extraction based on implicit correlation channels and correlation scores. First, the time dimension data is filtered based on the implicit correlation channels, and then feature analysis is performed based on different scales. Finally, the features of different scales are aggregated together according to the correlation scores to obtain implicit correlation time domain features. The implicit network is composed of a feature filter and an MTE module, and the feature filter filters irrelevant features of the correlation channels.

[0036] Preferably, the step of aggregating different scale features together according to the correlation scores to obtain implicitly correlated time domain features comprises:

[0037]

[0038] In the formula, Represents the correlation score, reflecting the selected j-th implicit channel and original channel C i The relative importance of represents the output of the feature filter in the hidden network, O represents the output of the MTE module, and here represents the output of the hidden network.

[0039] Preferably, the explicit correlated time domain features and the implicit correlated time domain features obtained in step S35 are input into a stream-level adaptive fusion module, and the two feature sequences are adaptively fused to capture the inter-channel dependency, and the final feature sequence obtained includes:

[0040]

[0041] In the formula, O1 and O2 are the outputs of the explicit network and the implicit network respectively, α represents the learnable parameter, and the dependency between the two streams is dynamically learned. is the output after capturing the time dependency and inter-channel dependency for the i-th channel, Represents the final feature sequence obtained by connecting the outputs of the corresponding C channels together.

[0042] Therefore, the present invention adopts the above-mentioned time series analysis method based on the dual-stream Transformer architecture, which has the following beneficial effects:

[0043] (1) By constructing the TSTNet model, we explicitly handle the relationship between individuality and commonality between channels and solve the complex dependencies between different channels;

[0044] (2) Improve the understanding and utilization of time-dependent data to improve the accuracy of downstream tasks, which is applicable to multiple fields.

[0045] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0047] Figure 2 This is a diagram of the dual-stream Transformer architecture of an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0049] Example

[0050] Reference Figure 1 The present invention provides a time series analysis method based on a dual-stream Transformer architecture, the steps comprising:

[0051] S1. Obtain a multivariate time series dataset and preprocess the dataset to obtain a training dataset, a validation dataset, and a test dataset.

[0052] In this embodiment, real data sets in eight different fields are collected, including energy, economy, transportation, weather and disease. The data sets are segmented and preprocessed to meet the model's data processing requirements. The length of each set of data is uniformly set to 96, and the entire data set is segmented by a sliding window, with 70% of the data set used as a training data set, 10% of the data set used as a validation data set, and 20% of the data set used as a test data set.

[0053] S2. Build the TSTNet model, as shown in the following diagram Figure 2 As shown in Figure 1, the TSTNet model includes a stream-specific channel selection module, a two-stream feature interaction module, and a stream-level adaptive fusion module. The two-stream feature interaction module includes an explicit network and an implicit network, which are used to process explicit related channels and implicit related channels respectively.

[0054] S3. Input the training data set into the TSTNet model for preliminary training to obtain the final feature sequence. The training process includes:

[0055] S31. Randomly select 64 sets of training data, input the training data into the TSTNet model, and perform normalization processing on the input sequences in parallel and independently. Normalization processing is a concept drift removal operation commonly performed in the field of time series. The specific formula is as follows:

[0056]

[0057] In the formula, Represents an L×C real number set representing the input sequence, L represents the length of the input sequence, C represents the number of channels of the input sequence, represents a set of real numbers, mean and var represent the mean and variance of the input sequence respectively.

[0058] S32. The input sequence after normalization is input into the flow-specific channel selection module. The flow-specific channel selection module calculates the correlation scores between different channels, and distinguishes the relationship between channels according to the size of these score values, so as to obtain explicit related channels, implicit related channels and irrelevant channels. The explicit related channels and implicit related channels are input into the explicit network and the implicit network in parallel, and the irrelevant channels are discarded.

[0059] In this embodiment, calculating the correlation scores between different channels by the stream-specific channel selection module includes:

[0060] Calculate the correlation scores between different channels, the formula is:

[0061]

[0062] In the formula, R i represents the correlation scores between different channels, from The data of the ith channel separated from the C channels; Represents the correlation score between the i-th channel and all channels;

[0063] Select the first k+1 values ​​with the largest scores and use To express;

[0064]

[0065] In the formula, k represents a hyperparameter, Corresponding channel is the dominant correlation channel, Corresponding channel is the implicitly related channel, and the rest are irrelevant channels.

[0066] S33. The explicit network extracts time domain features based on explicit correlation channels, performs feature analysis on the data in the time dimension based on different scales, and then aggregates features of different scales together to obtain explicit correlated time domain features.

[0067] In this embodiment, the explicit network captures the individual characteristics of the channel in a channel-independent manner. The explicit network consists of an MTE module, and the MTE module formula is:

[0068]

[0069] In the formula, Sampling(*) is the downsampling scale information, Padding(*) is the end padding 0 to achieve complete downsampling, and is the sequence before and after sampling, l is the scale after sampling, Transformer(*) represents a single-layer encoder, Proj(*) is a simple linear layer that maps the captured feature information to the length required for the task, O represents the output of the MTE module, and here represents the output of the explicit module.

[0070] Specifically for Transformer, the traditional Transformer encoder is used to capture the features contained in the time series. The specific process includes:

[0071] First, through the learnable parameter matrix The downsampled sequence of length l Projected into the hidden state space of Transformer, local information is expanded to address the limited information represented by a single point.

[0072] Then, through the learnable relative position encoding To embed the time order to solve the time-insensitive characteristics of the attention mechanism, the entire embedding layer can be expressed as:

[0073]

[0074] In the formula, represents the sequence after multi-scale downsampling, W d represents the learnable parameter matrix, W p represents the learnable relative position encoding, X enc Represents the output after calculation;

[0075] Then, the multi-head attention mechanism will input Convert to query matrix for each head separately key matrix Sum Matrix Then the output Q is calculated through the attention mechanism h , and finally concatenate the outputs of all heads together, the formula is:

[0076]

[0077] MultiHead(X enc )=Concat(Head1,…,head H )×W O ;

[0078] In the formula, and is the parameter mapping matrix, K h The transposed matrix of , d represents a hyperparameter, specifically the dimension of the hidden layer in the Transformer encoder. The original dimension entering the encoder is mapped to a d-dimensional space, then evenly distributed and processed by the multi-head attention module. When there are H heads, for each head: After the multi-head attention module, the final output is obtained through residual connection, batch normalization (bacthNorm) and position-wise feed-forward network (position-wisefeed-forward). The specific formula is:

[0079] X mul=BatchNorm(MutilHead(X enc )+X enc ;

[0080]

[0081] In the formula, FFN(*) consists of two fully connected layers plus a ReLU activation function in the middle, X mul Represents the intermediate data after passing through the multi-head attention module, is the output of the encoder.

[0082] S34. The implicit network extracts features based on implicit correlation channels and correlation scores. First, the time dimension data is filtered based on the implicit correlation channels, and then feature analysis is performed based on different scales. Finally, the features of different scales are aggregated together according to the correlation scores to obtain implicit correlation time domain features.

[0083] In this embodiment, the implicit network is parallel to the explicit network to obtain the common characteristics between channels. The implicit network is composed of a feature filter and an MTE module. The feature filter filters irrelevant features of related channels, and the MTE module is the same as the process in step S33.

[0084] Different scale features are aggregated together according to the correlation scores, and the implicitly correlated time domain features obtained include:

[0085]

[0086] In the formula, Represents the correlation score, reflecting the selected j-th implicit channel and original channel C i The relative importance of Represents the output of the feature filter, O represents the output of the MTE, and here also represents the output of the implicit network.

[0087] S35. The obtained explicit correlated time domain features and implicit correlated time domain features are input into the stream-level adaptive fusion module, and the two feature sequences are adaptively fused to capture the inter-channel dependency to obtain the final feature sequence.

[0088] In this embodiment, the specific process of obtaining the final feature sequence includes:

[0089]

[0090] In the formula, O1 and O2 are the outputs of the explicit network and the implicit network respectively, α represents the learnable parameter, and the dependency between the two streams is dynamically learned. is the output after capturing the time dependency and inter-channel dependency for the i-th channel, Represents the final feature sequence obtained by connecting the outputs of the corresponding C channels together.

[0091] S4. Based on the final feature sequence obtained, the gradient descent method is used to optimize the loss function to complete the training of the TSTNet model.

[0092] In this embodiment, the loss function is the L2 loss function, and the mean square error and the average square error are used as evaluation indicators.

[0093] S5. Use the validation data set for verification, input all group data into the trained TSTNet model, evaluate the final feature sequence output by the TSTNet model, and determine whether the TSTNet model continues to optimize.

[0094] S6. Repeat steps S3 to S5 until the evaluation index obtained in step S5 no longer improves, and the TSTNet model training is completed.

[0095] S7. Using the test data set, input all group data into the trained TSTNet model, evaluate the final feature sequence output by the TSTNet model, and obtain the final loss score of TSTNet on the data set.

[0096] In order to verify the TSTNet model proposed in the present invention, multiple public data sets are used to conduct comparative experiments with multiple models, including iTransformer (ICLR-2024), FreTS (Nips-2023), Koopa (Nips-2023), TimesNet (ICLR-2023), Crossformer (ICLR-2023), DLinear (AAAI-2023), FEDformer (Nips-2022) and Pyraformer (ICLR-2022), and the comparison results are shown in Tables 1 and 2. It can be seen from the table that the present invention has achieved good results on multiple public data sets.

[0097] Table 1 Comparison of long time series prediction results

[0098]

[0099] Table 2 Missing value processing results

[0100]

[0101] Therefore, the present invention adopts the above-mentioned time series analysis method based on the dual-stream Transformer architecture, and uses the TSTNet model based on the dual-stream network architecture. The TSTNet model decouples the complex dependencies between channels, fully captures the cross-channel and cross-time features in the time series data, and obtains the optimal information utilization efficiency.

[0102] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A time series analysis method based on a dual-stream Transformer architecture, characterized in that the steps include: S1. Obtain a multivariate time series data set, and preprocess the data set to obtain a training data set, a validation data set, and a test data set; S2. Build TSTNet model; S3, input the training data set into the TSTNet model for preliminary training to obtain the final feature sequence; S4. Based on the final feature sequence obtained, the gradient descent method is used to optimize the loss function to complete the first training of the TSTNet model; S5. Use the validation data set for verification, input all group data into the trained TSTNet model, evaluate the final feature sequence output by the TSTNet model, and determine whether the TSTNet model continues to optimize; S6, repeat steps S3 to S5 until the evaluation index obtained in step S5 is no longer improved, and the TSTNet model training is completed; S7. Using the test data set, input all group data into the trained TSTNet model, evaluate the final feature sequence output by the TSTNet model, and obtain the final loss score of TSTNet on the data set.

2. According to claim 1, a time series analysis method based on a dual-stream Transformer architecture is characterized in that: In step S2, the TSTNet model includes a stream-specific channel selection module, a dual-stream feature interaction module and a stream-level adaptive fusion module, and the dual-stream feature interaction module includes an explicit network and an implicit network.

3. According to claim 2, a time series analysis method based on a dual-stream Transformer architecture is characterized in that: In step S3, the training data set is input into the TSTNet model for preliminary training, and the final feature sequence obtained includes: S31, randomly select a certain amount of training data, input the training data into the TSTNet model, and perform normalization processing on the input sequence in parallel and independently; S32, inputting the normalized input sequence into the stream-specific channel selection module, calculating the correlation scores between different channels through the stream-specific channel selection module, and distinguishing the relationship between the channels according to the values ​​of these scores, obtaining explicit related channels, implicit related channels and irrelevant channels, inputting the explicit related channels and implicit related channels into the explicit network and the implicit network in parallel, and discarding the irrelevant channels; S33, the explicit network extracts time domain features based on the explicit correlation channel to obtain explicit correlation time domain features; S34, the implicit network extracts features based on the implicit correlation channels and the correlation scores to obtain implicit correlation time domain features; S35. The obtained explicit correlated time domain features and implicit correlated time domain features are input into the stream-level adaptive fusion module, and the two feature sequences are adaptively fused to capture the inter-channel dependency to obtain the final feature sequence.

4. According to claim 3, a time series analysis method based on a dual-stream Transformer architecture is characterized in that: The normalization processing formula in step S31 is: In the formula, Represents an L×C real number set representing the input sequence, L represents the length of the input sequence, C represents the number of channels of the input sequence, represents a set of real numbers, mean and var represent the mean and variance of the input sequence respectively.

5. According to claim 4, a time series analysis method based on a dual-stream Transformer architecture is characterized in that: Calculating the correlation scores between different channels by the stream-specific channel selection module in step S32 includes: Calculate the correlation scores between different channels, the formula is: In the formula, R i represents the correlation scores between different channels, from The data of the ith channel separated from the C channels; Represents the correlation score between the i-th channel and all channels; Select the first k+1 values ​​with the largest scores and use To express; In the formula, k represents a hyperparameter, Corresponding channel is the dominant correlation channel, Corresponding channel is the implicitly related channel, and the rest are irrelevant channels.

6. The time series analysis method based on the dual-stream Transformer architecture according to claim 5 is characterized by: In step S33, the explicit network extracts time domain features based on the explicit correlation channel, performs feature analysis on the data of the time dimension based on different scales, and then aggregates the features of different scales together to obtain the explicit correlation time domain features.

7. A time series analysis method based on a dual-stream Transformer architecture according to claim 6, characterized in that: The explicit network consists of an MTE module, and the MTE module formula is: In the formula, Sampling(*) is the downsampling scale information, Padding(*) is the end padding 0 to achieve complete downsampling, and is the sequence before and after sampling, l is the scale after sampling, Transformer(*) represents a single-layer encoder, Proj(*) is a simple linear layer that maps the captured feature information to the length required for the task, O represents the output of the MTE module, and here represents the output of the explicit module.

8. The time series analysis method based on the dual-stream Transformer architecture according to claim 7 is characterized in that: In step S34, the implicit network extracts features based on the implicit correlation channels and the correlation scores. First, the time dimension data is filtered based on the implicit correlation channels, and then feature analysis is performed based on different scales. Finally, the features of different scales are aggregated together according to the correlation scores to obtain implicit correlation time domain features. The implicit network is composed of a feature filter and an MTE module. The feature filter filters irrelevant features of the correlation channels.

9. A time series analysis method based on a dual-stream Transformer architecture according to claim 8, characterized in that: The method of aggregating different scale features together according to the correlation scores to obtain implicitly correlated time domain features includes: In the formula, Represents the correlation score, reflecting the selected j-th implicit channel and original channel C i The relative importance of Represents the output of the feature filter in the hidden network. O is the output of the MTE module, which also represents the output of the hidden network.

10. A time series analysis method based on a dual-stream Transformer architecture according to claim 9, characterized in that: The explicit correlated time domain features and implicit correlated time domain features obtained in step S35 are input to the stream-level adaptive fusion module, which adaptively fuses the two feature sequences to capture the inter-channel dependency, and obtains the final feature sequence including: In the formula, O1 and O2 are the outputs of the explicit network and the implicit network respectively, α represents the learnable parameter, and the dependency between the two streams is dynamically learned. is the output after capturing the time dependency and inter-channel dependency for the i-th channel, Represents the final feature sequence obtained by connecting the outputs of the corresponding C channels together.

Images