Time sequence prediction system based on variable dependence and long-time dependence
By introducing the timing channel adapter and post-constraint module in the time series large language model GPT-2, the shortcomings of the existing models in modeling multivariate timing and long-term timing dependencies are solved, and higher prediction accuracy and interpretability are achieved.
Patent Information
- Application Number
- CN202411465424.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-03-07
AI Technical Summary
The existing time series large language models have shortcomings in modeling multivariate timing and long-term timing dependencies, resulting in poor prediction results, especially when the prediction length increases rapidly.
A timing prediction system based on variable dependence and long-term dependence is designed, using the large language model GPT-2 with pre-trained weights, and adding a timing channel adapter and post-constraint module to its architecture to dynamically adjust the variable channel weights and learn long-term context timing characteristics.
It effectively improves the modeling ability of multiple timing and long-term timing, and improves the accuracy of time series prediction tasks, especially in long-term prediction scenarios.
Smart Images
Figure CN119578459A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of time series analysis, and in particular relates to a time series prediction system. Background Art
[0002] Time series prediction refers to predicting future values or states based on existing time series data. It is an important task in data science and machine learning. This type of problem usually exists in various scenarios, such as financial market forecasting, weather forecasting, energy consumption forecasting, traffic flow forecasting, etc. The impact of time series prediction technology on various fields is reflected in improving efficiency, optimizing resources, reducing risks, and many other aspects. The application of this technology not only makes decision-making more scientific and forward-looking, but also effectively improves the quality of production, management, and services. Especially with the continuous development of big data and deep learning, time series prediction has become more accurate and reliable, bringing important value enhancement to all walks of life. At the same time, with the rapid progress of large models in fields such as natural language processing and image processing, it is urgent to study time series large models. However, there are still many problems with existing time series large language models. For example, in most time series large models, the focus is on independent channel modeling, ignoring the dependencies between variables in the time series. The loss function used in the training process is mostly the mean square error loss function, which calculates the gap between the predicted value and the true value point by point, but ignores the modeling of long-term dependencies in the time series. This rough training process leads to poor performance of the time series large model, especially as the prediction length increases, the prediction effect decreases rapidly. Therefore, it is urgent to design a time series large model framework that integrates variable dependency and long-term dependency to improve the prediction effect of the model. Summary of the invention
[0003] The purpose of the present invention is to provide a time series prediction system based on variable dependency and long-term dependency, so as to effectively improve the modeling ability of multivariate time series and long-term time series, and improve the accuracy of time series prediction tasks in real application scenarios.
[0004] The time series prediction system based on variable dependency and long-term dependency provided by the present invention uses a variety of new technical means, including: a large language model for multivariate time series prediction; a time series channel adapter to model multivariate time series dependency; a post-constraint module to model long-term time series dependency and improve the long-term prediction capability of the model; specifically as follows:
[0005] (1) The large language model for multivariate time series prediction specifically uses the large language model GPT-2 loaded with pre-trained weights (cited from “A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019.”), continues to be trained on the time series dataset, fixes the weights of most parameters, and only updates the weights of the layer normalization part;
[0006] (2) The temporal channel adapter is to add an adapter to the architecture of the large language model GPT-2 to learn the dependencies between multiple variables and dynamically adjust the weights of each variable channel;
[0007] (3) The post-constraint module adds a contrastive learning constraint module to the representation space output by the large language model GPT-2, driving the large language model to learn more robust and predictive representations.
[0008] Further:
[0009] (1) The large language model for multivariate time series prediction, wherein:
[0010] To use a large language model to process time series data, for a time series containing n variables First, standard normalization is performed. Specifically, for the i-th variable, the normalized variable z i The calculation process is as follows:
[0011]
[0012] Among them, μ i is the variable x i The mean value, σ i is the variable x i The standard deviation of i Divide into non-overlapping time series segments, called time series patches; i After a linear layer f(·), it is mapped to the input space of the large model so that the time series patch can be directly input into the large model. The calculation process is as follows:
[0013] f(z i )=W 1 z i +b 1 #(2)
[0014] Among them, W 1 and b 1 are the weights and biases of the linear layer respectively.
[0015] The large language model receives the time series patch as input, and extracts a good time series representation by modeling the dependencies between the time series patches through the multi-head self-attention mechanism and the feedforward neural network. Specifically, the pre-trained autoregressive language model GPT-2 is selected for the experiment. The selected GPT-2 model consists of six layers of stacked GPT-2 blocks. The output of the l-1th block is the input of the lth block. Assume that the output of the l-1th block is h l-1 , the detailed calculation process of the lth block is as follows:
[0016]
[0017] FFN(m l )=ReLU(m l W 2 +b 2 )W 3 +b 3 #(4)
[0018] h l =LN(FFN(m l )+m l ) #(5)
[0019] Among them, MHSA is a multi-head attention mechanism module. represents the sequential channel adapter module, LN represents the LayerNormalization module, ReLU represents the ReLU (·) activation function, FFN (Feed-Forward Network) is a feed-forward neural network composed of two layers of perceptrons, W 2 , W 3 represents the weight of the linear layer of the FFN module, b 2 、b 3 Represents the linear layer bias of the FFN module.
[0020] Assume that the temporal representation h of the last layer output of the GPT-2 model k , after the post-constraint module and the linear decoder Decoder, the predicted value is obtained The process is as follows:
[0021]
[0022] Among them, P(·) represents the post-constraint module, W 4 and b 4 Represent the weight matrix and bias vector of the linear layer respectively.
[0023] During the model training process, in order to achieve the purpose of efficient parameter fine-tuning, the pre-trained parameters of the GPT-2 model are loaded, and the parameters of other components of GPT-2 except the Layer Normalization layer are frozen, which greatly accelerates the fine-tuning training process.
[0024] (2) The timing channel adapter is designed to dynamically capture the dependencies between variables and learn the weights of each variable channel, where:
[0025] The hidden state output of the self-attention layer is denoted as h l , where l represents the lth layer; h l The shape of is (B×C×n×d), where B is the batch size, C is the number of variables, n is the number of patches after segmentation, and d is the model dimension. In order to aggregate global information in each variable channel, we first apply global pooling to these hidden representations h. l , we get p l :
[0026]
[0027] Feature l It is then projected into a higher-dimensional space, passed through a ReLU (·) activation function, and then reduced back to the original dimension through a fully connected layer. These two fully connected layers integrate cross-channel feature information to compensate for the impact of global pooling operations being limited to a single channel. This structure introduces enhanced nonlinearity, enabling the model to more effectively capture complex inter-variable correlations. The normalized weight g is then generated using the Sigmoid gating mechanism l , whose values range from 0 to 1. These weights are applied to the channel features h through a weighted operation l , to obtain a new representation This representation aggregates cross-channel information and replaces the original h l , as the input of the subsequent residual block. The entire calculation process is as follows:
[0028] g l =Sigmoid(W 6 (ReLU(W 5 p l ))) #(8)
[0029]
[0030] in, And M>C.
[0031] This weighting operation enables the model to focus on features in individual channels while adjusting the relative importance across channels based on global information, thereby enhancing its ability to capture dynamic relationships between variables.
[0032] The post-constraint module described in (3) aims to impose constraints on the output space of the large language model so that the model can learn a more robust and predictive temporal representation.
[0033] Given the temporal representation h generated by the last layer of GPT-2 0：t+v First, use a two-layer GRU model from h 0 to h t Extract long-term contextual temporal features to obtain a high-dimensional context vector c 0：t :
[0034] c 0：t =GRU(h 0 ,......,h t ) #(10)
[0035] Context vector c 0：t is used to predict from h t+1 to h t+v To this end, the projection matrix W is used 7 c 0：t Mapping to h t+1：t+v The goal is to make the prediction space from c 0：t The projected value is as close as possible to the time series representation of the latter segment. The cosine similarity is used to measure the predicted value c 0：t W 7 and the true value h t+1：t+v Distance between:
[0036]
[0037] in, In the segments from the same temporal context, maximize the context information c 0：t With future fragment h t+1：t+v The similarities between them encourage the model to infer more global structures and thus better capture the relationship between consecutive time points.
[0038] Contrastive learning is also used to improve multivariate time series representation. Specifically:
[0039] Given a set H = {h (1) , ..., h (r) , ..., h (M)}, where M is the total number of time series representations of the randomly selected batch, which is The associated ones are considered as positive samples The rest are considered negative samples. Calculate the contrast loss as follows:
[0040]
[0041] Contrastive loss It aims to maximize the similarity between the predicted representation of the positive sample and the true representation, while minimizing the similarity with the negative sample. This helps the model better distinguish between the correct future segments and irrelevant segments, ensuring that the learned representation h 0：t+v Captures the most important features, including long-term dependencies of multivariate time series.
[0042] The present invention has at least the following beneficial effects:
[0043] The present invention designs a large language model for multivariate time series prediction, which effectively mines the dependency between multiple variables in the time series, and makes up for the deficiencies and technical limitations of the existing time series prediction methods in variable dependency modeling. The present invention selects a pre-trained large language model GPT-2, first pre-processes the time series data, including normalization and segmentation, and then inputs the mapped time series segments into the GPT-2 model, and trains the model to obtain a robust time series representation. At the same time, the design adds a time series adapter to the original GPT-2 model, introduces a small amount of trainable parameters, and effectively mines the dependency between time series variables. And the post-constraint module is designed to further impose contrastive learning constraints on the time series representation output by the large model, help the model to mine global context dependencies, learn more robust and more predictive time series representations, and realize the extraction of time series prediction representations in longer prediction length scenarios. Finally, a linear decoder is used to map the learned time series representation to the required prediction length. The present invention effectively improves the prediction accuracy and the interpretability of the prediction process in the long-term time series prediction scenario, and has a high practical application value. At the same time, the present invention has good portability and versatility, and can be quickly migrated from one application scenario to a new application scenario. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 It is a framework diagram of the present invention.
[0045] Figure 2 Schematic diagram of the rear constraint module. DETAILED DESCRIPTION
[0046] The present invention is further described in detail below in conjunction with the accompanying drawings.
[0047] like Figure 1 As shown, for a time series containing n variables First, preprocessing is performed, including applying standard normalization and dividing the time series segments. Then, the divided time series segments are linearly mapped and input into the GPT-2 model loaded with pre-trained parameters. The GPT-2 model consists of six layers of stacked GPT-2 blocks. The output of the l-1th block is the input of the lth block. Assume that the output of the l-1th block is h l-1 , the detailed calculation process of the lth block is as follows:
[0048]
[0049] FFN(m l )=ReLU(m l W 2 +b 2 )W 3 +b 3 #(14)
[0050] h l =LN(FFN(m l )+m l )#(15)
[0051] Among them, MHSA is a multi-head attention mechanism module, LN is a Layer Normalization module, ReLU represents the ReLU (·) activation function, FFN (Feed-Forward Network) is a feed-forward neural network composed of two layers of perceptrons, and W 2 , W 3 represents the weight of the linear layer of the FFN module, b 2 、b 3 Represents the linear layer bias of the FFN module. represents the temporal channel adapter module, and the hidden state output of the self-attention layer is represented as y l , where l represents the lth layer. l The shape of is (B×C×n×d), where B is the batch size, C is the number of variables, n is the number of patches after segmentation, and d is the model dimension. In order to aggregate global information in each variable channel, global pooling is applied to these hidden representations y l , we get p l :
[0052]
[0053] Feature lIt is then projected into a higher-dimensional space, passed through a ReLU (·) activation function, and then reduced back to the original dimension through a fully connected layer. These two fully connected layers integrate cross-channel feature information to compensate for the impact of global pooling operations being limited to a single channel. This structure introduces enhanced nonlinearity, enabling the model to more effectively capture complex inter-variable correlations. The normalized weight g is then generated using the Sigmoid gating mechanism l , whose values range from 0 to 1. These weights are applied to the channel features h through a weighted operation l , to obtain a new representation This representation aggregates cross-channel information and replaces the original h l , as the input of the subsequent residual block. The entire calculation process is as follows:
[0054] g l =Sigmoid(W 6 (ReLU(W 5 p l )))#(17)
[0055]
[0056] in, And M>C.
[0057] Assume that the hidden state h of the last layer output of the GPT-2 model k , after the post-constraint module and the linear decoder Decoder, the predicted value is obtained The process is as follows:
[0058]
[0059] Among them, P(·) represents the post-constraint module, W 4 and b 4 Represent the weight matrix and bias vector of the linear layer respectively. During the training process, in order to achieve the purpose of efficient parameter fine-tuning, the pre-trained parameters of the GPT-2 model are loaded, and the parameters of other components of GPT-2 except the Layer Normalization layer are frozen, which greatly accelerates the fine-tuning training process.
[0060] like Figure 2 As shown, the post-constraint module is given the temporal representation h generated by the last layer of GPT-2 0：t+v , we first use a two-layer GRU model from h 0 to h t Extract long-term contextual temporal features to obtain a high-dimensional context vector c 0 :t:
[0061] c 0：t=GRU(h 0 ,......,h t )#(20)
[0062] Context vector c 0：t It is then used to predict the t+1 to h t+v To this end, we use the projection matrix W 7 c 0：t Mapping to h t+1：t+v Our goal is to make the prediction space from c 0：t The projected value is as close as possible to the time series representation of the latter segment. The cosine similarity is used to measure the predicted value c 0：t W 7 and the true value h t+1：t+v Distance between:
[0063]
[0064] in, In the segments from the same temporal context, maximize the context information c 0：t With future fragment h t+1：t+v The similarities between them encourage the model to infer more global structures and thus better capture the relationship between consecutive time points.
[0065] Contrastive learning is also used to improve the representation of multivariate time series. Specifically, given a set H = {h (1) , ..., h (r) , ..., h (M)}, where M is the total number of time series representations of the randomly selected batch, which is The associated ones are considered as positive samples The rest are considered negative samples. Calculate the contrast loss as follows:
[0066]
[0067] Contrastive loss It aims to maximize the similarity between the predicted representation of the positive sample and the true representation, while minimizing the similarity with the negative sample. This helps the model better distinguish between the correct future segments and irrelevant segments, ensuring that the learned representation h 0：t+v Captures the most important features, including long-term dependencies of multivariate time series.
[0068] This application also provides a verification test to further prove the technical effect of this application.
[0069] We selected seven time series data sets to verify the present invention, including four transformer temperature (Electric Transformer Temperature, ETT) data sets (ETTh1, ETTh2, ETTm1 and ETTm2), an electricity data set (Electricity), a financial data set (Exchange Rate), and a meteorological data set (Weather). As usual, the data sets used in the experiment are all normalized.
[0070] To verify the superiority of the present method, on the above seven data sets, the present embodiment is compared with the following existing time series prediction methods: Unitime (excerpted from "X.Liu, J.Hu, et al., "Unitime: a language-empowered unified model for cross-domain time series forecasting," in Proceedings of theACM on Web Conference 2024.WWW 2024, Singapore, May 13-17, 2024, pp.4095-4106."), GPT4TS (excerpted from "T.Zhou, P.Niu, et a1., "One fits all: power general time series analysis by pretrained lm," Advances in Neural Information Processing Systems, vol.36, pp.43322-43355, 2023."), PatchTST (excerpted from "Y.Nie, NHNguyen, et al., "A timeseries is worth 64 words: long-term forecasting with transformers," in TheEleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023."), TimesNet (from "H.Wu, T.Hu, Y.Liu, et al., "Timesnet: temporal 2d-variation modeling for general time series analysis," in TheEleventh International Conference on Learning Representations.ICLR 2023, Kigali, Rwanda, May 1-5, 2023.") and DLinear (from "A.Zeng, M.Chen, L Zhang, and Q.Xu, "Are transformers effective for time series forecasting?" in Thirty-SeventhAAAI Conference on Artificial Intelligence,AAAI,B.Williams,Y.Chen,andJ.Neville,Eds.,2023,pp.11 121-11 128.").
[0071] This example uses mean square error (MSE) and mean absolute error (MAE) as evaluation criteria to measure the performance of each algorithm. To ensure a fair comparison, the same experimental settings as the previous method are adopted, the length of the input historical time window is fixed to 96, and the model performance is measured under four different prediction lengths, from short to long, which are 96, 192, 336 and 720.
[0072] Table 1 gives the prediction accuracy of various time series prediction methods on seven data sets.
[0073] Table 1: Comparison of prediction accuracy of six methods on seven datasets
[0074]
[0075] The highest accuracy in the table is marked in bold, and the second highest accuracy is marked with an underline. It can be seen from the data in the table that the present embodiment as a whole has achieved a prediction accuracy far higher than other benchmark methods. Specifically, the present embodiment has achieved 26 lowest mean square errors and 27 lowest mean absolute errors, as well as two second lowest mean square errors and one second lowest mean absolute error in a total of 28 experiments on seven data sets. The present embodiment has achieved all the lowest mean square errors and lowest mean absolute errors on two transformer temperature data sets (ETTh1, ETTh2), an electric power data set (Electricity) and a weather data set (Weather). At the same time, as the length of the prediction window increases, the prediction effects of all models decline to varying degrees, but the downward trend of the present embodiment is the smallest. For a longer prediction window (336,720), the present embodiment has achieved the lowest mean square error and mean absolute error on all data sets. This shows that the present embodiment has considerable advantages in long-term time series prediction. In summary, the present embodiment can effectively handle time series prediction tasks, and can achieve higher prediction accuracy than its contemporary benchmark methods.
[0076] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and the implementation modes, and they can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.
Claims
1. A time series prediction system based on variable dependency and long-term dependency, characterized in that: include: Large language model for multivariate time series prediction; Time series channel adapter, modeling multivariate time series dependencies; The post-constraint module models long-term temporal dependencies and improves the long-term prediction capabilities of the model; the details are as follows: (1) The large language model for multivariate time series prediction specifically uses the large language model GPT-2 loaded with pre-trained weights, continues to be trained on the time series dataset, fixes the weights of most parameters, and only updates the weights of the layer normalization part; (2) The temporal channel adapter is loaded on the architecture of the large language model GPT-2 and is used to learn the dependencies between multiple variables and dynamically adjust the weights of each variable channel; (3) The post-constraint module is loaded into the representation space output by the large language model GPT-2 and is used to compare learning constraints, driving the large language model to learn more robust and predictive representations.
2. The time series prediction system according to claim 1, characterized in that: The large language model for multivariate time series prediction, wherein: For a time series containing n variables First, standard normalization is performed; specifically, for the i-th variable, the normalized variable z i The calculation process is as follows: Among them, μ i is the variable x i The mean value, σ i is the variable x i The standard deviation of z i Divide into non-overlapping time series segments, called time series patches; z i After being mapped to the input space of the large model through a linear layer f(·), the calculation process is as follows: f(z i )=W1z i +b1 #(2) Among them, W1 and b1 are the weight and bias of the linear layer respectively; The GPT-2 model consists of six layers of stacked GPT-2 blocks. The output of the l-1th block is the input of the lth block. Assume that the output of the l-1th block is h l-1 , the calculation process of the lth block is as follows: <h2 style=";text-align:left;direction:ltr">FFN(m<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> )=ReLU(m<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> W2+b2)W3+b3 #(4) h l =LN(FFN(m l ) + m l ) #(5) Among them, MHSA is a multi-head attention mechanism module. represents the temporal channel adapter module, LN is the LayerNormalization module, ReLU is the ReLU(·) activation function, FFN is a feedforward neural network consisting of two layers of perceptrons, W2 and W3 represent the weights of the linear layer of the FFN module, b2 and b3 represent the linear layer bias of the FFN module; assuming that the hidden state of the last layer output of the GPT-2 model is h k , after a post-constraint module and a linear decoder Decoder to get the predicted value The process is as follows: Where P(·) represents the post-constraint module, W4 and b4 represent the weight matrix and bias vector of the linear layer, respectively; During the model training process, in order to achieve the purpose of efficient parameter fine-tuning, the pre-trained parameters of the GPT-2 model are loaded, and the parameters of other components of GPT-2 except the LayerNormalization layer are frozen to greatly accelerate the fine-tuning training process.
3. The time series prediction system according to claim 2, characterized in that: The timing channel adapter, wherein: Let the hidden state output of the self-attention layer be represented as h l , l represents the lth layer; h l The shape of is (B×C×n×d), where B is the batch size, C is the number of variables, n is the number of patches after segmentation, and d is the model dimension; in order to aggregate global information in each variable channel, global pooling is applied to these hidden representations h l , we get the feature p l : The feature pl is then projected into a higher-dimensional space, passed through a ReLU(·) activation function, and then reduced back to the original dimension through a fully connected layer; these two fully connected layers integrate cross-channel feature information to compensate for the impact of the global pooling operation being limited to a single channel; the normalized weight g is then generated using the Sigmoid gating mechanism l , whose values range from 0 to 1; these weights are used for channel features h through weighted operations l , to obtain a new representation This representation aggregates cross-channel information and replaces the original h l , as the input of the subsequent residual block; the entire calculation process is as follows: <h2 style=";text-align:left;direction:ltr">g<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> =Sigmoid(W6(ReLU(W5p<h2 style=";text-align:left;direction:ltr"> l <h2 style=";text-align:left;direction:ltr"> ))) #(8) in, And M>C.
4. The time series prediction system according to claim 3, characterized in that: The rear restraint module, wherein: Given the temporal representation h0 generated by the last layer of GPT-2: t+v First, a two-layer GRU model is used to extract long-term contextual temporal features from h0 to ht to obtain a high-dimensional context vector c 0：t : c 0：t =GRU(h0,......,h t ) #(10) Context vector c 0：t will be used to predict the t+1 to h t+v The latter part of the timing representation, for this purpose, the projection matrix W7 is used to transform c 0：t Mapping to h t+1：t+v The goal is to make the prediction space from c 0：t The projected value is as close as possible to the time series representation of the latter segment; the cosine similarity is used to measure the predicted value c 0：t W7 and true value h t+1：t+v Distance between: in, In segments from the same temporal context, maximize the context information c0:t with the future segment h t+1：t+v The similarity between them encourages the model to infer more global structures and better capture the relationship between consecutive time points; We also use contrastive learning to improve multivariate time series representation; specifically: Given a set H = {h (1) , ..., h (r) , ..., h (M) }, M is the total number of time series representations of randomly selected batches, which is The associated ones are considered as positive samples The rest are considered as negative samples; calculate the contrast loss as follows: Contrastive loss It aims to maximize the similarity between the predicted representation of the positive sample and the true representation, while minimizing the similarity with the negative sample; this helps the model better distinguish between the correct future fragments and irrelevant fragments, ensuring that the learned representation h 0:t+v Captures the most important features, including long-term dependencies of multivariate time series.