Time sequence data prediction method based on time convolution and multi-head attention mechanism

Abstract

The invention relates to a time sequence data prediction method based on time convolution and a multi-head attention mechanism. The method comprises the following steps: preprocessing a time sequence data set, dividing the time sequence data set into a plurality of batches, and dividing each batch into a plurality of time windows; processing any batch of time sequence data to obtain a scale sequence with internal correlation; completing prediction according to the different scale sequences with the internal correlation; and the time series data of all batches are predicted. According to the method, periodic mode extraction is performed on a time sequence, and extracted features are mapped to a feature space related to a key time scale, so that effective capture of correlation of different time scales is realized; by combining an adaptive learning loss function and an early stop mechanism, the generalization ability and prediction precision of the model in processing complex time sequence data are improved.

Description

Technical Field

[0001] The present application relates to the technical field of time series data prediction, and in particular to a time series data prediction method based on time convolution and multi-head attention mechanism. Background Art

[0002] Time series data is typically the result of observing an underlying process at a given sampling rate over equally spaced time periods. Therefore, there are close connections between data values ​​at different moments. Time series forecasting uses historical time series data to predict future data, providing data support for developing appropriate strategies. It has wide applications in fields such as economics, finance, and meteorology. It can be categorized into univariate and multivariate time series forecasting, depending on the variable dimension. Univariate time series forecasting assumes that variable values ​​are only correlated with their own historical values; multivariate time series forecasting, on the other hand, assumes that each variable is influenced not only by its own historical values ​​but also by close connections with other variables. Deep learning methods typically establish channels based on the variable dimensions in the sequence to capture the correlations between different features.

[0003] In recent years, deep models have been widely used in time series analysis tasks, with recurrent neural networks (RNNs), timed convolutional networks (TCNs), and transformer networks being particularly prominent. However, RNN and TCN methods primarily focus on capturing changes between adjacent moments and lack the ability to model long-term dependencies. While the Transformer has inherent advantages in modeling long-term dependencies, the complexity of real-world temporal variations makes it difficult to reliably extract temporal dependencies using attention mechanisms between discrete time points. Furthermore, existing deep learning models have significant shortcomings in capturing the dynamics of inter-sequential correlations across multiple time series at different time scales. For example, in finance, the correlations between asset prices, including stocks, bonds, and commodities, may increase during periods of market instability due to risk aversion, while they may decrease during periods of economic growth as investors diversify their portfolios to pursue diverse opportunities. Similarly, in ecosystems, the dynamics of variables controlling species populations and environmental factors exhibit complex temporal correlations across different time scales. The correlations between series at different time scales can vary significantly due to market or environmental fluctuations, and existing models fail to effectively capture these complex dynamic correlations between series across different periods. Summary of the Invention

[0004] Therefore, it is necessary to address the above technical issues and provide a time series data prediction method based on temporal convolution and multi-head attention mechanism that can capture the complex dynamic correlations between sequences in different periods, including:

[0005] S1: Preprocess the time series dataset and divide it into several batches, and then divide each batch into several time windows;

[0006] S2: Each time window of any batch of time series data is embedded and positionally encoded. The periodic features of each time window are identified through spectral analysis. The time series data is converted into a two-dimensional time series based on the selected frequency and periodic features. Two-dimensional time convolution is performed. Based on the convolved sequence, a multi-head attention mechanism is used at each time scale to capture intra-sequence correlations and obtain a scale sequence with intra-correlation.

[0007] S3: Aggregate different scale sequences with internal correlation into a one-dimensional time series, project the sequence in the time dimension and variable dimension, and complete the prediction by combining the adaptive learning loss function and early stopping mechanism;

[0008] S4: Return to step S2 for the next batch of time series data until all batches of time series data are predicted.

[0009] Furthermore, in step S1, the step of preprocessing the time series data set is specifically as follows: the time series data set includes a plurality of time series data;

[0010] Perform min-max normalization on time series data, and the expression is as follows:

[0011]

[0012] Where t is time, xt is the unnormalized time series data, min(T) and max(T) are the minimum and maximum values ​​in the time series before normalization, and xt′ is the normalized time series data.

[0013] Furthermore, in step S2, the steps of embedding and adding position encoding to each time window of any batch of time series data are as follows:

[0014] A fully connected layer is used to map the original data into the embedding space, which is expressed as follows:

[0015] X embed =W·x t ′ +b

[0016] Among them, X embed ∈R T×dmodel is the embedded time data, W is the weight matrix, xt′ is the normalized time series data, and b is the bias term;

[0017] The temporal information is introduced through position coding, which is added to the embedded data. The expression is as follows:

[0018] X withpos =X embed +PE

[0019] Among them, X withpos ∈R T×dmodel It is the time series data after embedding and adding position encoding. PE is the position encoding matrix with the same shape as X embed same.

[0020] Furthermore, in step S2, the step of identifying the periodic characteristics of each time window through spectrum analysis includes:

[0021] The discrete wavelet transform is used to transform each time window into the frequency domain, and its expression is as follows:

[0022]

[0023] ∈{1,…,k}.

[0024] Where DWT(·) represents the discrete wavelet transform of the input data to convert the time series from the time domain to the frequency domain. In the frequency domain, the periodic pattern of the data is expressed as the amplitude of different frequencies. Amp(·) is used to calculate the amplitude value of each frequency point after DWT. The larger the amplitude, the more significant the periodic component of the frequency in the time series. The vector F contains the average amplitude value of all frequencies. This amplitude is in dimension d. model The above is averaged by the function Avg(·).

[0025] Furthermore, in step S2, the time series data is converted into a two-dimensional time series based on the selected frequency and periodicity features as follows:

[0026] Considering the sparsity of the frequency domain and avoiding the noise caused by meaningless high frequencies, only the first k amplitude values ​​are selected and the most significant frequencies {f1,…,f k}, where k is a hyperparameter. The frequency chosen corresponds to k period lengths {p1,…,p k}, based on the selected frequencies {f1,…,f k} and the corresponding period lengths {p1,…,p k}, the time series data, i.e., one-dimensional time series, is converted into multiple two-dimensional tensors through the following equation, which is expressed as follows:

[0027]

[0028] Where Padding(·) is to extend the time series by zero along the time dimension to make it consistent with Compatible, where pi and fi represent the number of rows and columns of the transformed 2D tensor, respectively. represents the time series after the i-th reshaping based on time scale i.

[0029] Furthermore, in step S2, the steps of performing two-dimensional time convolution are specifically as follows:

[0030] The converted two-dimensional tensor is processed through a parameter-efficient Inception module. This module uses multi-scale two-dimensional convolution kernels to effectively capture features at different scales. The Inception module processes the input tensor through multi-scale two-dimensional convolution operations to extract multi-level spatiotemporal features. The convolution kernel simultaneously captures local details and global trends in the time series. Its expression is as follows:

[0031]

[0032] Among them, the Inception module performs convolution operations on the input tensor by using convolution kernels of different scales in parallel, and splices the outputs of each branch along the feature dimension, thereby achieving effective extraction of multi-scale features.

[0033] Furthermore, in step S2, the steps of using the multi-head attention mechanism at each time scale based on the convolution sequence are as follows:

[0034] At each time scale, a multi-head attention mechanism is used to capture the correlation within the sequence. Specifically, for each time scale tensor, a multi-head attention mechanism of self-attention is applied on the time scale dimension of the tensor, which is expressed as follows:

[0035]

[0036] Among them, MHA s (·) represents the multi-head attention function applied on the time scale dimension, which is used to extract local and global correlations between different time points in the sequence, thereby enhancing the model's ability to model temporal features.

[0037] Furthermore, in step S3, the different scale sequences with internal correlation are aggregated into a one-dimensional time series as follows:

[0038] Inspired by autocorrelation, the amplitude F can reflect the relative importance of the selected frequency and period, and thus corresponds to the importance of each transformed two-dimensional tensor. Therefore, the one-dimensional representations of different scales are weightedly aggregated based on the amplitude, and its expression is as follows:

[0039]

[0040] in, is the amplitude corresponding to each scale, which is calculated using discrete wavelet transform; the SoftMax function is applied to the amplitude Normalization is performed to determine the weight of each scale, and the outputs of different scales are aggregated into a one-dimensional time series through weighted summation.

[0041] Furthermore, in step S3, the steps of projecting the sequence on the time dimension and the variable dimension are specifically as follows:

[0042] Linear projection is used in both time dimension and variable dimension. Convert to The conversion expression is as follows:

[0043]

[0044] Where W t ∈R T×L , W s ∈R dmodel×N and b∈R T are learnable parameters, and the result is the predicted data, N represents the number of variables, L represents the length of the input sequence, and T represents the prediction range.

[0045] Furthermore, in step S3, the steps of completing the prediction by combining the adaptive learning loss function and the early stopping mechanism are as follows:

[0046] During the training process, mean square error and mean absolute error are used as loss functions to evaluate model performance. Mean square error is used to measure the square error between the predicted value and the true value, and mean absolute error is used to measure the absolute error between the predicted value and the true value. Its expression is:

[0047]

[0048] Among them, Y i represents the true value at the i-th time point, represents the predicted value at the i-th time point, and N represents the length of the time series, that is, the total number of time points.

[0049] During model training, an early stopping mechanism is used to prevent overfitting. The specific steps are as follows:

[0050] The validation loss based on mean square error and mean absolute error is monitored on the validation set respectively. When the validation loss no longer decreases in several consecutive rounds of training, the training process is terminated early.

[0051] The present invention extracts periodic patterns from time series and maps the extracted features to the feature space related to the key time scale, thereby effectively capturing the correlations at different time scales. The model extracts the two-dimensional changing features of time series data and integrates the multi-head self-attention mechanism to achieve the synchronous capture of intra-sequence correlations. By combining the adaptive learning loss function and the early stopping mechanism, the generalization ability and prediction accuracy of the model are improved when processing complex time series data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 1. A flow chart of a method for predicting time series data based on temporal convolution and multi-head attention mechanism in one embodiment; DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0054] Example 1:

[0055] This embodiment provides Figure 1 The time series data prediction method based on temporal convolution and multi-head attention mechanism shown in FIG includes:

[0056] S1: Preprocess the time series dataset and divide it into several batches, and then divide each batch into several time windows;

[0057] S2: Each time window of any batch of time series data is embedded and positionally encoded. The periodic features of each time window are identified through spectral analysis. The time series data is converted into a two-dimensional time series based on the selected frequency and periodic features. Two-dimensional time convolution is performed. Based on the convolved sequence, a multi-head attention mechanism is used at each time scale to capture intra-sequence correlations and obtain a scale sequence with intra-correlation.

[0058] S3: Aggregate different scale sequences with internal correlation into a one-dimensional time series, project the sequence in the time dimension and variable dimension, and complete the prediction by combining the adaptive learning loss function and early stopping mechanism;

[0059] S4: Return to step S2 for the next batch of time series data until all batches of time series data are predicted.

[0060] This embodiment extracts periodic patterns from time series and maps the extracted features to feature spaces related to key time scales, thereby effectively capturing correlations at different time scales. The model extracts two-dimensional variation features of time series data and integrates a multi-head self-attention mechanism to achieve synchronous capture of intra-sequence correlations. By combining an adaptive learning loss function and an early stopping mechanism, the model's generalization ability and prediction accuracy are improved when processing complex time series data.

[0061] Example 2:

[0062] This embodiment further discloses the following on the basis of the first embodiment:

[0063] Furthermore, in step S1, the step of preprocessing the time series data set is specifically as follows: the time series data set includes a plurality of time series data;

[0064] Perform min-max normalization on time series data, and the expression is as follows:

[0065]

[0066] Where t is time, xt is the unnormalized time series data, min(T) and max(T) are the minimum and maximum values ​​in the time series before normalization, and xt′ is the normalized time series data.

[0067] Furthermore, in step S2, the steps of embedding and adding position encoding to each time window of any batch of time series data are as follows:

[0068] A fully connected layer is used to map the original data into the embedding space, which is expressed as follows:

[0069] X embed =W·x t ′ +b

[0070] Among them, X embed ∈R T×dmodel is the embedded time data, W is the weight matrix, xt′ is the normalized time series data, and b is the bias term;

[0071] The temporal information is introduced through position coding, which is added to the embedded data. The expression is as follows:

[0072] X withpos =X embed +PE

[0073] Among them, X withpos ∈R T×dmodelIt is the time series data after embedding and adding position encoding. PE is the position encoding matrix with the same shape as X embed same.

[0074] Furthermore, in step S2, the step of identifying the periodic characteristics of each time window through spectrum analysis includes:

[0075] The discrete wavelet transform is used to transform each time window into the frequency domain, and its expression is as follows: ∈{1,…,k}.

[0076] Where DWT(·) represents the discrete wavelet transform of the input data to convert the time series from the time domain to the frequency domain. In the frequency domain, the periodic pattern of the data is expressed as the amplitude of different frequencies. Amp(·) is used to calculate the amplitude value of each frequency point after DWT. The larger the amplitude, the more significant the periodic component of the frequency in the time series. The vector F contains the average amplitude value of all frequencies. This amplitude is in dimension d. model The above is averaged by the function Avg(·).

[0077] Furthermore, in step S2, the time series data is converted into a two-dimensional time series based on the selected frequency and periodicity features as follows:

[0078] Considering the sparsity of the frequency domain and avoiding the noise caused by meaningless high frequencies, only the first k amplitude values ​​are selected and the most significant frequencies {f1,…,f k}, where k is a hyperparameter. The frequency chosen corresponds to k period lengths {p1,…,p k}, based on the selected frequencies {f1,…,f k} and the corresponding period lengths {p1,…,p k}, the time series data, i.e., one-dimensional time series, is converted into multiple two-dimensional tensors through the following equation, which is expressed as follows:

[0079]

[0080] Where Padding(·) is to extend the time series by zero along the time dimension to make it consistent with Compatible, where pi and fi represent the number of rows and columns of the transformed 2D tensor, respectively. represents the time series after the i-th reshaping based on time scale i.

[0081] Furthermore, in step S2, the steps of performing two-dimensional time convolution are specifically as follows:

[0082] The converted two-dimensional tensor is processed through a parameter-efficient Inception module. This module uses multi-scale two-dimensional convolution kernels to effectively capture features at different scales. The Inception module processes the input tensor through multi-scale two-dimensional convolution operations to extract multi-level spatiotemporal features. The convolution kernel simultaneously captures local details and global trends in the time series. Its expression is as follows:

[0083]

[0084] Among them, the Inception module performs convolution operations on the input tensor by using convolution kernels of different scales in parallel, and splices the outputs of each branch along the feature dimension, thereby achieving effective extraction of multi-scale features.

[0085] Furthermore, in step S2, the steps of using the multi-head attention mechanism at each time scale based on the convolution sequence are as follows:

[0086] At each time scale, a multi-head attention mechanism is used to capture the correlation within the sequence. Specifically, for each time scale tensor, a multi-head attention mechanism of self-attention is applied on the time scale dimension of the tensor, which is expressed as follows:

[0087]

[0088] Among them, MHA s (·) represents the multi-head attention function applied on the time scale dimension, which is used to extract local and global correlations between different time points in the sequence, thereby enhancing the model's ability to model temporal features.

[0089] Furthermore, in step S3, the different scale sequences with internal correlation are aggregated into a one-dimensional time series as follows:

[0090] Inspired by autocorrelation, the amplitude F can reflect the relative importance of the selected frequency and period, and thus corresponds to the importance of each transformed two-dimensional tensor. Therefore, the one-dimensional representations of different scales are weightedly aggregated based on the amplitude, and its expression is as follows:

[0091]

[0092] in, is the amplitude corresponding to each scale, which is calculated using discrete wavelet transform; the SoftMax function is applied to the amplitude Normalization is performed to determine the weight of each scale, and the outputs of different scales are aggregated into a one-dimensional time series through weighted summation.

[0093] Furthermore, in step S3, the steps of projecting the sequence on the time dimension and the variable dimension are specifically as follows:

[0094] Linear projection is used in both time dimension and variable dimension. Convert to The conversion expression is as follows:

[0095]

[0096] Where W t ∈R T×L , and b∈R T are learnable parameters, and the result is the predicted data, N represents the number of variables, L represents the length of the input sequence, and T represents the prediction range.

[0097] Furthermore, in step S3, the steps of completing the prediction by combining the adaptive learning loss function and the early stopping mechanism are as follows:

[0098] During the training process, mean square error and mean absolute error are used as loss functions to evaluate model performance. Mean square error is used to measure the square error between the predicted value and the true value, and mean absolute error is used to measure the absolute error between the predicted value and the true value. Its expression is:

[0099]

[0100] Among them, Y i represents the true value at the i-th time point, represents the predicted value at the i-th time point, and N represents the length of the time series, that is, the total number of time points.

[0101] During model training, an early stopping mechanism is used to prevent overfitting. The specific steps are as follows:

[0102] The validation loss based on mean square error and mean absolute error is monitored on the validation set respectively. When the validation loss no longer decreases in several consecutive rounds of training, the training process is terminated early.

[0103] This embodiment extracts periodic patterns from time series and maps the extracted features to feature spaces related to key time scales, thereby effectively capturing correlations at different time scales. The model extracts two-dimensional variation features of time series data and integrates a multi-head self-attention mechanism to achieve synchronous capture of intra-sequence correlations. By combining an adaptive learning loss function and an early stopping mechanism, the model's generalization ability and prediction accuracy are improved when processing complex time series data.

Claims

1. A time series data prediction method based on temporal convolution and multi-head attention mechanism, characterized in that: The following steps are involved: S1: Preprocess the time series dataset and divide it into several batches, and then divide each batch into several time windows; S2: Each time window of any batch of time series data is embedded and positionally encoded. Timing information is introduced based on the positional encoding to convert the time series from the time domain to the frequency domain. In the frequency domain, spectral analysis is performed to identify the periodic features of each time window and select frequencies. Based on the selected frequencies and periodic features, the time series data is converted into a two-dimensional time series. Two-dimensional time convolution is performed, and a multi-head attention mechanism is used on each time scale based on the convolved sequence to capture intra-sequence correlations and obtain a scale sequence with intra-correlation. S3: Aggregate different scale sequences with internal correlation into a one-dimensional time series, project the sequence in the time dimension and variable dimension, and complete the prediction by combining the adaptive learning loss function and early stopping mechanism; S4: Return to step S2 for the next batch of time series data until all batches of time series data are predicted.

2. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S1, the steps of preprocessing the time series data set are specifically as follows: the time series data set includes a plurality of time series data; Perform min-max normalization on time series data, and the expression is as follows: Where t is time, xt is the unnormalized time series data, min(T) and max(T) are the minimum and maximum values ​​in the time series before normalization, and xt′ is the normalized time series data.

3. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S2, the steps of embedding and adding position encoding to each time window of any batch of time series data are as follows: A fully connected layer is used to map the original data into the embedding space, which is expressed as follows: X embed =W·x t ′ +b Among them, X embed ∈R T×dmodel is the embedded time data, W is the weight matrix, xt′ is the normalized time series data, and b is the bias term; The temporal information is introduced through position coding, which is added to the embedded data. The expression is as follows: X withpos =X embed +PE Among them, X withpos ∈R T×dmodel It is the time series data after embedding and adding position encoding. PE is the position encoding matrix with the same shape as X embed same.

4. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S2, the step of identifying the periodic characteristics of each time window through spectrum analysis includes: The discrete wavelet transform is used to transform each time window into the frequency domain, and its expression is as follows: Where DWT(·) represents the discrete wavelet transform of the input data to convert the time series from the time domain to the frequency domain. In the frequency domain, the periodic pattern of the data is expressed as the amplitude of different frequencies. Amp(·) is used to calculate the amplitude value of each frequency point after DWT. The larger the amplitude, the more significant the periodic component of the frequency in the time series. The vector F contains the average amplitude value of all frequencies. This amplitude is in dimension d. model The above is averaged by the function Avg(·).

5. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S2, the time series data is converted into a two-dimensional time series based on the selected frequency and periodicity features. Specifically: Considering the sparsity of the frequency domain and avoiding the noise caused by meaningless high frequencies, only the first k amplitude values ​​are selected and the most significant frequencies {f1,…,f k }, where k is a hyperparameter. The frequency chosen corresponds to k period lengths {p1,…,p k }, based on the selected frequencies {f1,…,f k } and the corresponding period lengths {p1,…,p k }, the time series data, i.e., one-dimensional time series, is converted into multiple two-dimensional tensors through the following equation, which is expressed as follows: Where Padding(·) is to extend the time series by zero along the time dimension to make it consistent with Compatible, where pi and fi represent the number of rows and columns of the transformed 2D tensor, respectively. represents the time series after the i-th reshaping based on time scale i.

6. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S2, the steps of performing two-dimensional temporal convolution are as follows: The converted two-dimensional tensor is processed through a parameter-efficient Inception module. This module uses multi-scale two-dimensional convolution kernels to effectively capture features at different scales. The Inception module processes the input tensor through multi-scale two-dimensional convolution operations to extract multi-level spatiotemporal features. The convolution kernel simultaneously captures local details and global trends in the time series. Its expression is as follows: Among them, the Inception module performs convolution operations on the input tensor by using convolution kernels of different scales in parallel, and splices the outputs of each branch along the feature dimension, thereby achieving effective extraction of multi-scale features.

7. The method for predicting time series data based on temporal convolution and multi-head attention mechanism according to claim 1, characterized in that: In step S2, the steps of using the multi-head attention mechanism at each time scale based on the convolution sequence are as follows: At each time scale, a multi-head attention mechanism is used to capture the correlation within the sequence. Specifically, for each time scale tensor, a multi-head attention mechanism of self-attention is applied on the time scale dimension of the tensor, which is expressed as follows: Among them, MHA s (·) represents the multi-head attention function applied on the time scale dimension, which is used to extract local and global correlations between different time points in the sequence, thereby enhancing the model's ability to model temporal features.

8. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S3, the different scale sequences with internal correlation are aggregated into a one-dimensional time series as follows: Inspired by autocorrelation, the amplitude F can reflect the relative importance of the selected frequency and period, and thus corresponds to the importance of each transformed two-dimensional tensor. Therefore, the one-dimensional representations of different scales are weightedly aggregated based on the amplitude, and its expression is as follows: in, is the amplitude corresponding to each scale, which is calculated using discrete wavelet transform; the SoftMax function is applied to the amplitude Normalization is performed to determine the weight of each scale, and the outputs of different scales are aggregated into a one-dimensional time series through weighted summation.

9. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S3, the steps of projecting the sequence on the time dimension and the variable dimension are as follows: Linear projection is used in both time dimension and variable dimension. Convert to The conversion expression is as follows: Where W t ∈R T×L , W s ∈R dmodel×N and b∈R T are learnable parameters, and the result is the predicted data, N represents the number of variables, L represents the length of the input sequence, and T represents the prediction range.

10. The time series data prediction method based on temporal convolution and multi-head attention mechanism according to claim 1 is characterized in that: In step S3, the steps to complete the prediction by combining the adaptive learning loss function and the early stopping mechanism are as follows: During the training process, mean square error and mean absolute error are used as loss functions to evaluate the model performance. The mean square error is used to measure the square error between the predicted value and the true value, and the mean absolute error is used to measure the absolute error between the predicted value and the true value. Its expression is: in, Y i represents the true value at the i-th time point, represents the predicted value at the i-th time point, and N represents the length of the time series, that is, the total number of time points. During model training, an early stopping mechanism is used to prevent overfitting. The specific steps are as follows: The validation loss based on mean square error and mean absolute error is monitored on the validation set respectively. When the validation loss no longer decreases in several consecutive rounds of training, the training process is terminated early.

Images