Time series prediction method and device for multi-level text alignment using large models

Abstract

The application provides a time series prediction method and device for multi-level text alignment using a large model, belonging to the technical field of time series prediction based on large models in rail transit systems. The method comprises the following steps: inputting a multivariate time series into a plurality of single-variable time series according to feature dimensions, performing additive decomposition on each single-variable time series, and performing fragmentation processing on each time series component after decomposition; aligning the fragments with the text embedding space of a pre-trained language model; combining the structured prompts with the aligned time series representation to form the input of the large model; feeding the input combined with the prompt and the aligned representation into a frozen large language model to obtain the output representation of the model, and mapping the output representation to the final prediction result through a linear projection layer. The application effectively aligns and fuses time series data and natural language modalities, significantly improving the accuracy and interpretability of the prediction.

Description

Technical Field

[0001] This invention belongs to the field of time series prediction technology for rail vehicles, and in particular relates to a time series prediction method and apparatus that utilizes a large model for multi-level text alignment. Background Technology

[0002] In rail transit systems, time series data is widely used in several key aspects such as train operation monitoring, equipment condition assessment, and passenger flow management. This data typically includes trends in various physical quantities such as train speed, acceleration, energy consumption, temperature, and vibration. Due to the wide variety of sensors, inconsistent sampling frequencies, and data transmission delays, the acquired time series data often suffers from heterogeneity, missing values, and temporal misalignment, posing significant challenges to subsequent data analysis and prediction. Therefore, efficiently processing multi-source heterogeneous time series data and achieving high-precision prediction has become an important research direction for intelligent operation and maintenance of rail transit.

[0003] In recent years, with the development of deep learning technology, especially the successful application of large models in natural language processing, computer vision, and other fields, their potential in time series modeling has gradually attracted attention. Large models have powerful representational capabilities, enabling them to learn complex patterns from massive amounts of data and generalize to different tasks. For example, in the pre-training stage, large models can extract general time series features through self-supervised learning, and then fine-tune them on specific tasks (such as prediction and anomaly detection), thereby improving the model's adaptability and accuracy. However, directly applying large models to time series prediction still faces many challenges, one of the most critical issues being how to handle the alignment problem between time series from different sources or with different sampling rates.

[0004] Time series alignment refers to matching multiple time series along a time dimension to eliminate inconsistencies caused by differences in sampling frequency, time offset, or missing data. Traditional methods mainly rely on Dynamic Time Warping (DTW) or interpolation-based methods to align time series, but these methods have high computational complexity when dealing with large-scale, high-dimensional data and struggle to capture long-term dependencies. Furthermore, time series data in rail transit systems often come with a wealth of contextual information, such as equipment models, operating environments, and historical maintenance records, which are typically stored in text format. Effectively integrating textual information with time series data to improve model interpretability and prediction accuracy is a key research focus.

[0005] Against this backdrop, time series prediction methods utilizing large models for multi-level text alignment have emerged as a potential solution to the aforementioned problems. This method aims to leverage the powerful representational capabilities of large models, combining textual information with time series data, to achieve more accurate time series alignment and improve the robustness and generalization ability of the prediction model. This technology not only helps improve the intelligence level of rail transit systems but also provides strong support for equipment health management, energy consumption optimization, and operational scheduling.

[0006] Time series forecasting, a crucial task in data analysis, has long relied on statistical and machine learning methods. Early forecasting models primarily included statistical methods such as Autoregressive Integrated Moving Average (ARIMA) and Exponential Smoothing, which were suitable for linear, stationary time series but had limited performance when dealing with nonlinear, non-stationary data. Subsequently, machine learning methods such as Support Vector Machines (SVM) and Random Forest were introduced into the field of time series forecasting. While these methods improved the ability to model nonlinear relationships, their strong reliance on feature engineering made it difficult to automatically extract complex time series patterns.

[0007] In the field of rail transit, time series forecasting faces increasingly complex challenges. First, data on train operating status and equipment health often originate from different types of sensors, resulting in significant variations in sampling frequency and data quality, making data alignment difficult. Second, the operating environment of rail transit systems is complex, influenced by factors such as weather, passenger flow, and signal control, causing time series data to exhibit highly nonlinear and non-stationary characteristics. Furthermore, traditional forecasting models lack effective utilization of contextual information, such as equipment models, maintenance records, and operational strategies—textual information crucial for improving forecast accuracy and interpretability. Therefore, effectively integrating textual information with time series data to enhance model adaptability and robustness has become a key research direction.

[0008] In time series prediction tasks, the application of large models is mainly reflected in two aspects. First, large models can be used for pre-training of time series data, that is, extracting general time series features from massive amounts of unlabeled data through self-supervised learning. For example, the BERT-style mask prediction method has been applied to time series modeling, learning the global dependencies of the sequence by reconstructing the masked time steps. This pre-training strategy enables the model to achieve better generalization ability with limited labeled data, thereby improving prediction performance. Second, large models have stronger context modeling capabilities, can handle multivariate time series simultaneously, and combine external information (such as text descriptions, event logs, etc.) for joint modeling, thereby enhancing the interpretability and accuracy of prediction results.

[0009] Furthermore, the large-scale model demonstrates excellent adaptability in multi-task learning and zero-shot prediction. Since rail transit systems involve various prediction tasks, such as train operation status prediction, equipment fault warning, and passenger flow prediction, the large-scale model can simultaneously learn patterns from multiple related tasks through parameter sharing and knowledge transfer, thereby improving the overall performance of the model. Simultaneously, in situations where data is scarce, the large-scale model can also utilize its learned knowledge to perform zero-shot predictions, making reasonable predictions even without specific task training data. This is of great significance for the intelligent operation and maintenance of rail transit systems.

[0010] In rail transit systems, time-series data is often accompanied by rich contextual information, such as equipment models, operating environments, maintenance records, and operational strategies. This information is typically stored in text format. However, traditional prediction models often rely solely on numerical time-series data, neglecting the value of textual information, which limits the interpretability and accuracy of prediction results. Therefore, effectively integrating textual information with time-series data has become a key issue in improving prediction performance. Multi-Level Text Alignment (MLAL) technology is a solution proposed to address this need. It aims to associate textual descriptions with corresponding time series data through multi-level information matching, thereby enhancing the model's ability to understand the context.

[0011] The core objective of aligning time-series and text data is to establish correspondences between textual information and time series data at different granularities. First, at the token level, keywords in textual descriptions (such as "air conditioning system," "braking device," and "high-temperature environment") may be associated with specific time-series features (such as temperature, vibration, and energy consumption). By embedding these keywords and aligning them with time-series features, the model's ability to identify key variables can be enhanced. Second, at the sentence level, a complete textual description may contain multiple related contextual information, such as operating procedures and fault cause analysis in equipment maintenance records. This information can be modeled using a semantic encoder and matched with the historical state of the time series to help predict future trends. Finally, at the document level, an entire textual document (such as a maintenance manual or operation report) may cover contextual information from multiple time points. By modeling this information holistically and aligning it with the historical changes of the time series, the model's ability to model long-term dependencies can be further improved.

[0012] To effectively integrate textual information and time-series data, researchers have proposed various methods combining multi-level text alignment with time-series prediction. One typical approach employs a dual-channel modeling structure, processing time-series and textual data separately, and using a cross-modal attention mechanism for interactive modeling. This method ensures alignment between textual information and time-series data at different levels, thereby enhancing the model's ability to understand context.

[0013] Specifically, the time series portion typically employs sequence modeling techniques such as Transformer or LSTM to extract features from the original time series; while the text portion uses pre-trained language models such as BERT or RoBERTa to encode textual information. Subsequently, a cross-attention mechanism is used to fuse textual features with time series features, enabling the model to adjust its predictions based on the textual context. Furthermore, some studies have attempted to introduce Graph Neural Networks (GNNs) to model the complex relationships between text and time series, such as the causal relationship between equipment maintenance records and equipment status.

[0014] Beyond innovations in model structure, data preprocessing and feature engineering are also crucial steps in fusing multi-level text alignment with time series prediction. For example, textual information typically requires cleaning, word segmentation, and entity recognition to extract key semantic information; while time series data requires standardization, missing value imputation, and outlier handling to improve data quality. Furthermore, to enhance the model's generalization ability, meta-learning or transfer learning strategies can be introduced, enabling the model to quickly adapt to new prediction tasks or data distributions.

[0015] In rail transit systems, time series forecasting is used not only for trend inference under normal conditions but also plays a crucial role in anomaly detection. The goal of anomaly detection is to identify anomalous behaviors in time series data, such as equipment failures, signal anomalies, and sudden increases in energy consumption. Traditional anomaly detection methods primarily rely on statistical models or threshold-based rule-based judgments, but these methods often fail to achieve ideal results when dealing with complex and nonlinear rail transit data.

[0016] On the other hand, the accuracy of time series prediction models directly affects the effectiveness of anomaly detection. Therefore, improving the accuracy of prediction models is of great significance for enhancing anomaly detection capabilities. Furthermore, combining multi-level text alignment mechanisms can further improve the interpretability of anomaly detection. For example, when the model detects a large prediction error at a certain time point, the corresponding text context (such as maintenance records, weather conditions, signal status, etc.) can be analyzed to determine whether the anomaly is caused by external factors, thereby reducing the false alarm rate and improving diagnostic efficiency.

[0017] Therefore, the results of anomaly detection can also feed back into the training process of time series prediction models. For example, during the training phase, an anomaly perception mechanism can be introduced, allowing the model to consider the impact of historical anomalous events during prediction, thereby improving its ability to predict sudden events. Furthermore, online learning strategies can be employed to enable the model to update its understanding of data distribution in real time, thus adapting to the ever-changing rail transit environment. Summary of the Invention

[0018] The present invention aims to overcome the shortcomings of the prior art and provide a method and apparatus for time series prediction using large models for multi-level text alignment. By using artificial intelligence large model technology for time series modeling and prediction, the accuracy, interpretability, practicality and deployment efficiency of the prediction are significantly improved, and the model's adaptability to complex time series changes is enhanced.

[0019] To achieve the above-mentioned objectives, this invention provides a time series prediction method for multi-level text alignment using a large model, comprising the following steps:

[0020] Time series input decomposition steps: Split the multivariate time series input into multiple univariate time series according to feature dimensions, perform additive decomposition on each univariate time series to obtain trend components, seasonality components and residual components; and perform sharding on each decomposed time series component.

[0021] Multi-level text alignment steps: The segmented words are input into the multi-level text alignment module; using a multi-head cross-attention mechanism, the segmented embeddings are aligned with the text embedding space of the pre-trained language model to generate an aligned time series representation;

[0022] Component-specific hints step: Construct structured hints for specific components, which include at least the dataset context, input statistical features, and specific component task instructions for trend, seasonal, and residual components; combine the structured hints with the aligned time series representation to form the input of the large model;

[0023] Output mapping step: The input, which combines the prompts and alignment representations, is fed into the frozen large language model to obtain the model's output representation, and then the output representation is mapped to the final prediction result through a linear projection layer.

[0024] Furthermore, in the time series input decomposition step, the trend component is extracted using the moving average method, and the additive decomposition is performed using the classic additive seasonal trend decomposition method or the local weighted regression scatter smoothing method.

[0025] Furthermore, the multi-level text alignment step specifically includes:

[0026] For the segmented embedding of the trend component, it is used as a query and multi-head cross-attention calculation is performed with the predefined sparse trend anchors to achieve alignment between the trend component and the text anchors;

[0027] For the segmented embeddings of the seasonal components and residual components, they are used as queries and multi-head cross-attention calculations are performed with the seasonal text corpora and residual text corpora selected from the pre-trained word embeddings to achieve alignment between the seasonal components and residual components and the text corpora.

[0028] Furthermore, the large language model is a GPT-3 model, and in the output mapping step, only the first six layers of the model are used for forward inference. The parameters of the large language model are frozen and do not require fine-tuning.

[0029] Furthermore, the output mapping step also includes denormalizing the output of the linear projection layer and summing the denormalization results corresponding to each component to reconstruct the final predicted value.

[0030] The present invention also provides an apparatus for implementing the above method, characterized in that it comprises:

[0031] The input decomposition module is used to receive multivariate time series inputs, split them into univariate sequences, and then perform component decomposition and segmentation.

[0032] The multi-level text alignment module is used to achieve multi-level alignment between time series segments and text embedding space through a multi-head cross-attention mechanism;

[0033] The Component-Specific Hints module is used to generate and integrate structured hints for component-specific task instructions that include dataset context, input statistical features, and specific component-specific task instructions for trend, seasonal, and residual components.

[0034] The output mapping module is used to process the output of large language models, generating prediction results through linear projection and inverse normalization.

[0035] Furthermore, the multi-level text alignment module includes:

[0036] Trend alignment unit, configured to use sparse anchors for attention alignment;

[0037] Seasonal alignment unit, configured to perform attention alignment using a filtered corpus of seasonal text;

[0038] The residual alignment unit is configured to perform attention alignment using a filtered residual text corpus.

[0039] The beneficial effects of this invention are mainly reflected in the following aspects. First, by introducing a multi-level text alignment strategy, time series data and natural language modalities are effectively aligned and fused, enabling large language models to better understand and reason about time series data, thereby significantly improving the accuracy and interpretability of predictions. Second, this invention uses a frozen GPT-3 as the backbone model, using only its first six layers without fine-tuning, avoiding the high computational cost and resource consumption caused by large-scale model training, and improving the practicality and deployment efficiency of the method. Third, through a time series input decomposition mechanism, the original data is split into trend, seasonal, and residual components, and modeled separately, which helps to extract more refined data characteristics. The invention enhances the model's adaptability to complex time series changes by incorporating specific component hints. Furthermore, the introduction of these hints provides the model with joint guidance based on task context, statistical features, and component information, further optimizing the input representation and improving prediction performance. Finally, the output mapping module ensures the stability of the final output by inversely normalizing and reconstructing the prediction results. Overall, this invention demonstrates good generalization ability and application prospects in time series prediction within the rail transit field. It is not only suitable for multivariate, long-sequence prediction scenarios but also possesses strong interpretability, contributing to decision support and operation and maintenance management. It exhibits significant technological advancement and practical application value. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0041] To better understand the purpose, structure, and function of this invention, the following detailed description, in conjunction with the accompanying drawings, provides a method and apparatus for time series prediction using a large model for multi-level text alignment.

[0042] A time series prediction method and apparatus in the field of rail transit that utilizes a large model for multi-level text alignment. For example... Figure 1 As shown, the proposed method consists of four core modules: (1) time series input decomposition; (2) multi-level text alignment; (3) specific component hints; and (4) output mapping. This process first divides the multivariate time series into... There are 3 univariate time series, each processed independently, where the 1st series is the 2nd series. A sequence, denoted as Before aligning with time-series anchors and text corpora, a series of steps are involved, including decomposition, normalization, patching, and embedding. To enhance the reasoning ability of large language models on time-series data, this invention introduces specific component cues and aligned embeddings, enabling the model to generate meaningful output representations. These representations are then projected through an output linear layer to generate the final prediction result. With the primary goal of improving interpretability, this invention employs GPT-3, using its first six layers as the backbone model for time series prediction, without fine-tuning the base model.

[0043] Problem Definition. Given the first L timestamps of observations, the task of multivariate time series forecasting is to predict the next... The value of a timestamp can be represented formally as:

[0044]

[0045] in, It corresponds to the first i Channels of each feature Starting from timestamp t Step prediction vector, given historical values Large language model Usage Tips These predictions are made. Leveraging the powerful reasoning capabilities of pre-trained large language models, this invention aims to align time-series data with text, enabling large language models to interpret input sequences and accurately predict the future. Therefore, the overall objective of this invention is to minimize the mean square error between the true value and the predicted value, expressed as:

[0046] .

[0047] (1) Decomposition of time series input

[0048] For time series data, the model decomposes complex inputs into meaningful components, such as trends, seasonality, and residual elements, which helps to optimally extract valuable information. In this invention, given a multivariate time series input... Where N is the feature dimension and L is the length of the time series, its additive decomposition can be expressed as follows:

[0049]

[0050] in, Feature indexes and trend components of multivariate time series inputs Capture potential long-term patterns in the data, which can be represented as: where m=2k +1 and k It is the average step size; seasonal components. Reflecting recurring short-term cycles, estimation can be performed after removing the trend; residual components This represents the remaining data portion after extracting the trend and seasonal elements. Generally, several methods can be used to perform additive seasonal trend decomposition. A common method is the classic additive seasonal trend decomposition, which first uses a moving average method to extract the long-term trend, obtaining the trend component, then estimates the seasonal component by averaging the trend time series, and subtracts the estimated trend and seasonal components to obtain the residual component. Another widely used method is seasonal trend decomposition using locally weighted regression scatter smoothing (STL). In this invention, the choice of decomposition method is specifically determined based on the validation results.

[0051] Subsequently, the present invention segments the time series decomposition component; specifically, for the first... Based on the normalized trend components, the word segmentation obtained after segmentation in this invention is: ,in Indicates the fragment length. Indicates the number of fragments. Similarly, this invention applies this segmentation process to both seasonal and residual components to obtain segmented words. and Then, these segmented words are input into the multi-level text alignment module to generate aligned time series embeddings.

[0052] (2) Multilevel text alignment

[0053] This invention reprograms segmented embeddings into the pre-trained data representation space of a language model to align time series and natural language modalities, thereby activating the backbone model's time series understanding and reasoning capabilities. Intuitively, this invention can use similarity estimation to align time series and text segmentation embeddings. Although time series segmentation lacks text annotations, this invention can place their embeddings near typical textual descriptions of the time series. Therefore, it is intuitively expected that various time series segmentations can represent various descriptive words, such as rising, falling, and stable. However, the pre-trained word segmentation embedding space is vast and dense, and the selection of text corpora (patterns) is often very loose, sometimes even involving random words unrelated to the time series or clustering of pre-trained word segmentations, which leads to poor interpretability.

[0054] Therefore, this invention proposes a multi-level text alignment strategy to enhance the interpretability of language models in time series prediction. Specifically, this invention first decomposes the time series into trend, seasonality, and residuals, and then... With selected specific trend anchor points Alignment, however, is challenging in accurately defining anchor points for seasonal and residual components. To address this issue, this invention utilizes pre-trained word embeddings from the backbone model. For seasonal components and residual components Reprogramming, in which It refers to the size of the vocabulary list. E is the hidden dimension of the pre-trained language model. Directly utilizing E typically leads to a large and potentially dense reprogramming space; therefore, this invention employs a linear mechanism to probe E. The text corpora for the seasonal components and residual components are represented as follows: and ,in This is because residual components are more inconsistent and more variable compared to seasonal components.

[0055] Therefore, the multi-level text alignment proposed in this invention aims to establish connections between anchor points and trend segments. The selected anchor points are generally sparse. This invention reprograms seasonality and residuals using text corpora selected from pre-trained word embeddings, connecting time-series segments through a denser reprogramming space. To achieve this, this invention employs a multi-head cross-attention layer for each component. Specifically, for the first... i The input features, in this invention, define the trend as a query matrix. The key matrix is The value matrix is The query matrix is ​​defined as follows: Key matrix The value matrix is Residual definition query matrix The key matrix is Value matrix By employing a multi-head attention mechanism, this invention reprograms each time series component. For example, the trend after adopting the multi-head attention mechanism is defined as:

[0056]

[0057] in, It represents the dimension of each head in the multi-head attention module. After the multi-head attention step, each component undergoes linear projection to align the hidden dimensions with the backbone model.

[0058] (3) Specific component prompts

[0059] Existing prompting techniques have proven highly effective across a variety of tasks by leveraging task-specific knowledge encoded in prompts. This success stems from the fact that prompts provide a structured framework that aligns the model's output with the desired objective, improving accuracy and coherence. However, directly translating time series data into natural language presents significant challenges, complicating the creation of instruction-compliant datasets and effective, on-the-spot prompts. Recent advances have demonstrated that prompts can enrich the input context and guide the effective transformation of reprogrammed time series segments. To utilize the semantic information of time series components, this invention proposes a component-specific prefix prompting strategy, comprising three elements: dataset context, input statistical features, and component-specific task instructions for trend, seasonality, and residual components. For example, the task description "Predict the next 96 steps based on the previous 512 steps | [trend, seasonality, residual]" serves as a template for the task instructions in this invention, which is then concatenated with the corresponding component-aligned time series representation and fed into a large model.

[0060] (4) Output mapping

[0061] By freezing the core language model, packaging and inputting prompts and embeddings for specific components, this invention retains the embeddings of each component and applies a linear mapping to the output representation. The input, combining prompt and alignment representations, is fed into the frozen large language model to obtain the model's output representation. By inversely normalizing and summing these representations, the prediction result for the task of this invention is finally obtained. .

[0062] A time series prediction device that utilizes a large model for multi-level text alignment includes:

[0063] The input decomposition module is used to split multivariate time series into univariate series according to features, and to perform additive decomposition and slicing of trend, seasonality and residual components.

[0064] A multi-level text alignment module is used to align time series slices with the text embedding space through a multi-head cross-attention mechanism;

[0065] The specific component prompting module is used to generate structured prompts that include task instructions, statistical features, and component information;

[0066] The output mapping module is used to project the model output representation into the prediction result through a linear layer, and then perform inverse normalization and reconstruction.

[0067] This invention has significant advantages in predicting complex time-series data in rail transit. It can effectively combine textual information and numerical data to achieve high-precision and interpretable predictions, providing reliable technical support for intelligent operation and maintenance.

Claims

1. A time series prediction method using a large model for multi-level text alignment, characterized in that, Includes the following steps: Time series input decomposition steps: Split the multivariate time series input into multiple univariate time series according to feature dimensions, perform additive decomposition on each univariate time series to obtain trend components, seasonality components and residual components; and perform sharding on each decomposed time series component. Multi-level text alignment steps: The segmented words are input into the multi-level text alignment module; using a multi-head cross-attention mechanism, the segmented embeddings are aligned with the text embedding space of the pre-trained language model to generate an aligned time series representation; Component-specific hint steps: Construct structured hints for specific components, which include at least the dataset context, input statistical features, and specific component task instructions for trend, seasonal, and residual components; The structured hints are combined with the aligned time series representation to form the input of the large model; Output mapping step: The input, which combines the prompts and alignment representations, is fed into the frozen large language model to obtain the model's output representation, and then the output representation is mapped to the final prediction result through a linear projection layer.

2. The time series prediction method using a large model for multi-level text alignment according to claim 1, characterized in that: In the time series input decomposition step, the trend component is extracted using the moving average method, and the additive decomposition is performed using the classic additive seasonal trend decomposition method or the local weighted regression scatter smoothing method.

3. The method according to claim 1, characterized in that, The multi-level text alignment steps specifically include: For the segmented embedding of the trend component, it is used as a query and multi-head cross-attention calculation is performed with the predefined sparse trend anchors to achieve alignment between the trend component and the text anchors; For the segmented embeddings of the seasonal components and residual components, they are used as queries and multi-head cross-attention calculations are performed with the seasonal text corpora and residual text corpora selected from the pre-trained word embeddings to achieve alignment between the seasonal components and residual components and the text corpora.

4. The method according to claim 1, characterized in that, The large language model is the GPT-3 model, and in the output mapping step, only the first six layers of the model are used for forward inference. The parameters of the large language model are frozen and do not require fine-tuning.

5. The method according to claim 1, characterized in that, The output mapping step further includes denormalizing the output of the linear projection layer and summing the denormalization results corresponding to each component to reconstruct the final prediction value.

6. An apparatus for implementing the method as described in any one of claims 1 to 5, characterized in that, include: The input decomposition module is used to receive multivariate time series inputs, split them into univariate sequences, and then perform component decomposition and segmentation. The multi-level text alignment module is used to achieve multi-level alignment between time series segments and text embedding space through a multi-head cross-attention mechanism; The Component-Specific Hints module is used to generate and integrate structured hints for component-specific task instructions that include dataset context, input statistical features, and specific component-specific task instructions for trend, seasonal, and residual components. The output mapping module is used to process the output of large language models, generating prediction results through linear projection and inverse normalization.

7. The apparatus according to claim 6, characterized in that, The multi-level text alignment module includes: Trend alignment unit, configured to use sparse anchors for attention alignment; Seasonal alignment unit, configured to perform attention alignment using a filtered corpus of seasonal text; The residual alignment unit is configured to perform attention alignment using a filtered residual text corpus.

Images