Large language model time sequence prediction method and device based on integer decimal decomposition

Abstract

The invention discloses a large language model time sequence prediction method and device based on integer and decimal decomposition. The method comprises the steps that each numerical value in time series data is subjected to integer part and decimal part decomposition, correlation characteristics between the integer parts and the decimal parts are extracted, characteristic representation is obtained, and the correlation characteristics are used for assisting residual understanding; aligning a time sequence mode of the time sequence data with a text mode through a double cross attention mechanism to obtain a mode alignment representation; fusing the time sequence data, the feature representation and the modal alignment representation to obtain a final input feature; and based on the final input features, reasoning is carried out by using a large language model, and a prediction result is output. The technical problem that the reasoning result of the large language model is not accurate enough is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a method and device for predicting time series using a large language model based on integer-decimal decomposition. Background Art

[0002] In the era of big data, systems and entities continuously generate massive amounts of time series data during their ongoing operations. This data is widely used in numerous application scenarios, including weather forecasting, financial modeling, and energy planning. To extract valuable information from this data, time series forecasting has become a key research area. By analyzing the temporal characteristics of historical data, it predicts future numerical trends and provides support for practical decision-making.

[0003] In recent years, with the development of deep learning technology, researchers have proposed a variety of model schemes for multivariate time series prediction, which are mainly divided into two categories: non-pretrained models and pretrained models. Among them, the non-pretrained model based on Transformer has achieved remarkable results, but it still has certain limitations, such as limited parameter scale, insufficient model depth and poor generalization ability, especially when facing cross-domain data or small sample tasks. It is easy to overfit. To this end, some studies have begun to try to apply large language models (LLMs) to time series prediction tasks. Due to the powerful context modeling capabilities brought by large-scale pre-training, LLMs have shown good cross-domain transfer capabilities and have shown great potential in few-sample and zero-sample learning scenarios, becoming a new choice to replace traditional methods.

[0004] However, directly applying LLMs to time series forecasting still faces two challenges: First, LLMs have weak fitting capabilities for residuals and are easily affected by outliers and fluctuations, making it difficult to extract stable and effective time series features; second, time series data is fundamentally different from natural language in structure and semantics. LLMs are natively good at processing text data, and how to mobilize their pre-trained knowledge to understand time series becomes a technical difficulty.

[0005] To address these issues, existing technologies lack a universal solution that fully integrates the structural characteristics of time series with the expressive power of LLMs. Therefore, a new time series modeling framework suitable for LLMs is urgently needed to improve the model's understanding of time series patterns and prediction accuracy, while also ensuring generalization across diverse task scenarios. Summary of the Invention

[0006] The embodiments of the present invention provide a method and apparatus for predicting time series using a large language model based on integer-decimal decomposition, so as to at least solve the technical problem that the inference results of a large language model are not accurate enough.

[0007] According to one aspect of an embodiment of the present invention, a large language model time series prediction method based on integer-decimal decomposition is provided, comprising: decomposing each numerical value in time series data into an integer part and a decimal part, extracting correlation features between the integer part and the decimal part, and obtaining a feature representation, wherein the correlation features are used to assist residual understanding; aligning the time series modality of the time series data with the text modality through a double cross-attention mechanism to obtain a modality alignment representation; fusing the time series data, the feature representation, and the modality alignment representation to obtain a final input feature; and performing reasoning in a large language model based on the final input feature to output a prediction result.

[0008] According to another aspect of an embodiment of the present invention, a large language model time series prediction device based on integer-decimal decomposition is provided, comprising: a decomposition module, configured to decompose each numerical value in the time series data into an integer part and a decimal part, extract the correlation features between the integer part and the decimal part, and obtain a feature representation, wherein the correlation features are used to assist residual understanding; an alignment module, configured to align the time series modality of the time series data with the text modality through a double cross-attention mechanism to obtain a modality alignment representation; a fusion module, configured to fuse the time series data, the feature representation and the modality alignment representation to obtain a final input feature; a prediction module, configured to perform inference in a large language model based on the final input feature and output a prediction result.

[0009] In an embodiment of the present invention, each numerical value in the time series data is decomposed into an integer part and a decimal part, and the correlation features between the integer part and the decimal part are extracted to obtain a feature representation, wherein the correlation features are used to assist in residual understanding; the time series modality of the time series data is aligned with the text modality through a double cross-attention mechanism to obtain a modality alignment representation; the time series data, the feature representation, and the modality alignment representation are fused to obtain the final input features; based on the final input features, reasoning is performed using a large language model to output a prediction result. The above solution solves the technical problem of inaccurate reasoning results of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0011] Figure 1 is a flowchart of a large language model time series prediction method based on integer-decimal decomposition according to an embodiment of the present invention;

[0012] Figure 2 is an optional IDDLLM model architecture diagram according to an embodiment of the present invention;

[0013] Figure 3 is a flowchart of another large language model time series prediction method based on integer-decimal decomposition according to an embodiment of the present invention;

[0014] Figure 4 is an optional residual composition graph according to an embodiment of the present invention;

[0015] Figure 5 This is a flowchart of an optional simulation method for large language model time series prediction based on integer-decimal decomposition according to an embodiment of the present invention;

[0016] Figure 6 is a comparison diagram of modality alignment ablation experiment results according to an embodiment of the present invention;

[0017] Figure 7 2 is a structural diagram of a large language model time series prediction device based on integer-decimal decomposition according to an embodiment of the present invention;

[0018] Figure 8 A schematic structural diagram of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0019] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0021] According to an embodiment of the present invention, a method embodiment of a large language model time series prediction method based on integer-decimal decomposition is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] Figure 1 is a large language model time series prediction method based on integer-decimal decomposition according to an embodiment of the present invention, such as Figure 1 As shown, the method includes the following steps:

[0023] Step S102 : Decompose each value in the time series data into an integer part and a decimal part, extract the correlation features between the integer part and the decimal part, and obtain a feature representation, wherein the correlation features are used to assist in understanding the residual.

[0024] First, preprocessing is performed on the time series data. The preprocessing includes at least one of the following: data cleaning, missing value filling, and outlier processing; reversible normalization is performed on the preprocessed time series data based on the distribution of the training set, so that the mean of each dimension feature is 0 and the variance is 1; the normalized time series data is divided into continuous non-overlapping blocks of a preset length and converted into a feature format that meets the input requirements of the large language model.

[0025] Next, decomposition is performed. The integer part of each value in the time series data is extracted using a first function, and the decimal part is extracted using a second function. A cross-attention mechanism is used, using the integer part as the query vector and the decimal part as the key vector and value vector, to extract the attention representation between the integer part and the decimal part as the association feature.

[0026] Step S104: aligning the time series modality and the text modality of the time series data through a double cross attention mechanism to obtain a modality-aligned representation.

[0027] In the first stage, the text description is used as the query vector, the pre-trained word embedding is used as the key vector and the value vector, and the first cross-attention result is calculated; in the second stage, the time series representation is used as the query vector, the first cross-attention result is used as the key vector and the value vector, and the cross-attention is calculated again to obtain the modality alignment representation.

[0028] Step S106: Fusing the time series data, the feature representation, and the modality alignment representation to obtain a final input feature.

[0029] The time series data representation, the feature representation and the modality alignment feature are weightedly summed or concatenated to obtain a fused input feature as the final input feature.

[0030] Step S108: Based on the final input features, reasoning is performed in a large language model to output a prediction result.

[0031] The final input features are linearly transformed to adjust them to be consistent with the input dimensions of the large language model, and the adjusted final input features are input into the large language model; reasoning is performed using the large language model, and the final hidden state output of the large language model is mapped to the target prediction sequence length and dimension through a linear mapping layer to obtain the prediction result of the final time series.

[0032] The embodiment of the present application also provides another IDDLLM method based on integer-decimal decomposition and cross-modal fine-tuning. IDDLLM mainly enhances the time series prediction performance through two key mechanisms. First, the SDC time series splitting module decomposes the time series into an integer part and a decimal part, and analyzes the intrinsic relationship between them to capture more refined time features and enhance the model's understanding of time-dependent patterns. Secondly, the modal alignment module is optimized through a dual cross-attention mechanism to achieve effective cross-modal alignment between time series and text. This mechanism strengthens the semantic association between the two modalities through interactive learning, thereby ultimately improving the adaptability of LLM in time series prediction.

[0033] As described above, in terms of data processing, this embodiment preprocesses the time series data X into Xi, and then inputs the processed time series Xi into the SDC module to obtain Xs. In this module, the time series is decomposed into an integer part and a decimal part to extract relevant information. At the same time, Xi is input into the modal alignment module, where it is aligned with the filtered text prototype to generate Xa to achieve alignment between the time series and the text. It is then merged with Xi to obtain the final input Xe, as shown in formula (1). The input sequence is input into the LLM, and the output result is obtained after inference. Finally, the corresponding linear transformation is applied to the output result of the LLM to obtain the final prediction result.

[0034]

[0035] Here, SDC() and alignment() represent the data processing steps of the SDC module and the alignment module, respectively. Formula (1) summarizes the overall processing flow of time series data before inputting it into the model. The abbreviations here are only used to illustrate the complete process; the detailed operation of each module will be described below.

[0036] IDDLLM framework such as Figure 2 As shown, the lower left corner is the data preprocessing and innovative SDC module, which is responsible for the preprocessing of time series and extracting intrinsic correlations through the segmentation mechanism; the lower right corner is the modality alignment module, which uses a two-way cross-attention mechanism to align the pre-trained word embeddings and time series data after text description filtering; the upper area is the pre-trained large language model and its output adaptation layer, which realizes the output conversion of the large language model and the generation of prediction results.

[0037] First, clarify the current task: given a historical value X containing m features and corresponding to t historical time steps, T-t+1:T )∈R t×m , build a model to predict the value of n time steps in the future To minimize the mean square error goal.

[0038] In this application, all experiments use GPT2 as the base model. Although the proposed method is model-independent in principle and can be applied to any large language model, all evaluations in this application are completed based on the GPT2 model.

[0039] The IDDLLM framework can be executed as Figure 3 The method shown comprises the following steps:

[0040] Step S302: input preprocessing.

[0041] The input time series first needs to be preprocessed to reduce the impact of time series distribution shift. This process includes data cleaning, missing value filling, and outlier handling. Subsequently, the data is standardized and normalized to ensure consistency between different time series distributions. By using reversible instance normalization, each feature of the input sequence is independently normalized to a mean of 0 and a variance of 1 based on the distribution of the training set. This normalization process helps improve numerical stability and promotes model convergence.

[0042] Then, the processed time series is divided into several blocks (patches) to reduce memory usage and computational burden while preserving the time series information as much as possible. Therefore, the time series is divided into continuous non-overlapping patches of length P, and the total number of patches is Where L is the total length of the time series, P is the length of each block after division, and S is the length of each slide.

[0043] Finally, it is transformed into a shape X suitable for model processing through linear transformation e ∈R m*d, where m is the number of attributes and d is the hidden dimension of the large language model (LLMs). In this way, the model can learn the correlation between time steps, making it possible to input time series into LLMs in a more concise way.

[0044] Step S304: splitting using the SDC module.

[0045] The embodiment of the present application uses the SDC time series splitting module to extract more information, thereby reducing the difficulty of the model to understand the time series and improving reasoning performance. Traditional time series forecasting methods generally rely on residual learning to capture time-dependent patterns. However, residuals often contain noise or unpredictable factors, making it difficult for the model to learn its patterns, thereby affecting forecasting performance. This limitation also means that sudden events, outliers, or short-term disturbances may have a negative impact on the model fitting residuals.

[0046] To address the above limitations, this embodiment proposes a new method to enable the model to understand this difficult-to-model component, thereby reducing the fitting difficulty and improving the prediction performance.

[0047] This embodiment re-examines the composition of the residual and determines that the residual is mainly composed of two different parts: an integer part (with a relatively large value but low precision) and a fractional part (with a relatively small value but high precision), such as Figure 4 As shown in Figure 2, the interaction between the two components has a significant impact on the overall structure of the residuals. Based on this finding, the study further explored the intrinsic correlation between the two components to enhance the model's understanding of the composition of the time series.

[0048] Therefore, it is necessary to split each value in the time series. For example, a value 1.234 is decomposed into its integer part 1 and decimal part 0.234. As shown in formula (2), the time series is split into integer components and decimal components:

[0049]

[0050] Among them, τ() represents the integer part of the obtained value, and φ() represents the decimal part of the obtained value.

[0051] After decomposing the integer and decimal components, this example introduces a cross-attention mechanism to extract the correlation between the two and adds it as a bias term to the model to enhance its ability to capture the residual structure. This mechanism is designed to capture the interaction of explicit and implicit information between the integer and decimal parts and dynamically weightedly fuse them through attention distribution, enabling the model to specifically understand the compositional structure of time series residuals.

[0052] This embodiment uses the integer part as the query and the decimal part as the key and value to extract the correlation between the two, as shown in formula (3):

[0053]

[0054] Among them, Wq, Wk, and Wv are the projection matrices of query (Q), key (K), and value (V), respectively, which are used to linearly project the integer and fractional parts to the hidden dimensions of the model to adapt the model.

[0055] Step S306: perform alignment using a modal alignment module.

[0056] A major challenge in using large language models for time series forecasting is how to guide the model to integrate its pre-trained knowledge to understand and process time series. While LLMs excel at processing language tasks, time series are inherently different from text and cannot be directly represented as text in a lossless manner. This fundamental difference in modality necessitates an alignment mechanism to improve the model's ability to process time series data.

[0057] However, the pre-trained word embeddings are too large, and not all words in them are relevant to the time series task. To address this problem, this embodiment proposes a method that can selectively reduce the number of word embeddings while retaining high relevance.

[0058] Research has found that the official description text of the dataset can characterize the time series to a certain extent. As shown in Table 1, this description information can be used to guide the pre-trained word embedding to select parts that are highly relevant to the time series, and thus can be used to optimize the modality alignment module.

[0059] Table 1 (Dataset details and dataset description)

[0060]

[0061] This embodiment uses a dual cross-attention mechanism to build deep correlations between modalities through cross-attention, thereby achieving alignment between text and time series. Specifically, the goal of this module is to filter and refine pre-trained word embeddings using semantic information describing the text, while also integrating time series features to achieve alignment between the two different modalities.

[0062] As shown in formula (4), this embodiment defines the time series Xi∈R m×d , related description text Zd∈R m×d , and the vocabulary embedding Zv∈R m×dFirst, a multi-head cross-attention is performed between Zd and Zv to refine the vocabulary embedding based on the description of time series correlation. Then, the cross-attention is calculated again using the result Xm and the time series Xi. This iterative interaction process fuses the representations of the two modalities to produce a joint representation Xa that contains both text and time series features.

[0063]

[0064] Step S308: fine-tuning and output.

[0065] To improve training efficiency, reduce memory usage, and mitigate operational challenges due to device limitations, this embodiment employs parameter-efficient training techniques to fine-tune the pre-trained large language model. Specifically, a small number of weight parameters are fine-tuned using the Low-Rank Adaptation (LoRA) method.

[0066] In the output part, the model inputs the results processed by the aforementioned modules into the large language model, takes the last layer of hidden state and performs linear projection, adjusts it to different lengths to adapt to the task, and finally outputs the prediction results.

[0067] This application also provides a simulation experiment method based on integer-decimal decomposition and cross-modal fine-tuning, such as Figure 5 As shown, the method includes the following steps:

[0068] Step S502: prepare a data set.

[0069] The embodiments of the present application conduct experiments on six real-world datasets, which are widely used to evaluate the performance of long-term time series models. The datasets used and their specific information are as follows: ETTm1, ETTm2, ETTh1, ETTh2: The ETT (Electricity Transformer Temperature) series datasets record key indicators such as load and temperature of transformers in two regions of China, with two granularities of 1 hour and 15 minutes. Weather: This meteorological dataset records the changes in multiple meteorological variables in Germany within a year, including temperature, humidity and air pressure, with a sampling frequency of once an hour. Electricity: This electricity dataset records the changes in electricity consumption in multiple regions, with a sampling frequency of once an hour. These datasets cover multiple fields such as electricity, energy and meteorology. See Table 1 for more detailed dataset information.

[0070] Step S504: Conduct experiments using a baseline method.

[0071] This application example selects seven prediction methods as baselines, which are divided into the following two categories:

[0072] LLM-based methods: Time-LLM method, GPT4TS method; Transformer-based methods: PatchTST method, iTransformer method, Crossformer method, Autoformer method, Informer method.

[0073] The embodiment of this application adopts IDDLLM and combines a large number of the latest time series prediction models to design comparative experiments to verify the effectiveness of the proposed method. The prediction experiment is based on the first six layers of Transformer of the GPT2 pre-trained model as the backbone network, the loss function adopts L1Loss, and the verification function adopts MSE. The experimental tasks cover long-term prediction, zero-sample prediction, and few-sample prediction. To ensure the repeatability of the experimental results, the embodiment of this application uses 2021 as the random seed to lock the initial parameters. All training processes are completed on a single A10 graphics card.

[0074] For efficient parameter fine-tuning, we used the Low Rank Adaptation (LoRA) strategy with the following hyperparameter settings: rank r = 16, scaling factor α = 32, target modules c_attn and c_proj, and a dropout rate of 0.05 to prevent overfitting. No additional bias parameters were trained. This configuration is suitable for the Causal Language Modeling (CAUSAL_LM) task and is consistent with the model's autoregressive properties.

[0075] At the same time, to ensure the fairness of the experiment, the original settings of other models remain unchanged, and MSE (mean square error) and MAE (mean absolute error) commonly used in time series prediction are uniformly used as evaluation indicators to facilitate standardized comparison between different models.

[0076] Step S506: obtain the experimental results.

[0077] 1) Long-term forecast

[0078] Setup: This example conducts long-term prediction experiments on six real-world datasets: ETTm1, ETTm2, ETTh1, ETTh2, Weather, and Electricity. See Table 1 for detailed data. For uniform comparison, the input sequence length TTT is set to 96, and four different prediction intervals are used for the prediction time length.

[0079] Results: Detailed results for long-term prediction are shown in Table 2. In most cases, the method proposed in this embodiment outperforms other baseline models. More importantly, compared to the state-of-the-art LLM method, TIME-LLM, the proposed method achieves 9% and 8% reductions in MSE and MAE, respectively. Compared to the state-of-the-art Transformer model, PatchTST, the MSE and MAE are reduced by 2% and 3%, respectively.

[0080] Table 2

[0081]

[0082] Table 2 shows the long-term forecast results. The length of all input time series is set to 96. The results are derived from four different forecast horizons H∈(96, 192, 336, 720). The power data set only includes the first three forecast horizons due to equipment limitations. Lower values ​​indicate better performance.

[0083] 2) Few-sample prediction

[0084] Setup: Recent research has shown that LLMs have excellent few-shot learning capabilities, performing well even with only a small amount of training data. This capability is crucial for time series forecasting models. To verify whether the embodiments of this application possess such generalization capabilities, a few-shot prediction experiment was conducted. To ensure experimental fairness, the training set data size for few-shot prediction was uniformly set to no more than the first 10% of the total training set time steps.

[0085] Results: This embodiment of the present application conducted few-shot prediction experiments on four ETT datasets, and the results are shown in Table 3. In most cases, the method proposed in this embodiment of the present application outperformed other baseline models. Notably, compared with the TIME-LLM and PatchTST methods, the average performance improvement of this embodiment of the present application was 12% and 4%, respectively, demonstrating the powerful ability of this embodiment of the present application in few-shot prediction tasks.

[0086] Table 3

[0087]

[0088] Table 3 shows the small sample prediction results. These are based on the first 10% of the training dataset. All results are based on four different prediction ranges H∈(96, 192, 336, 720). Lower values ​​indicate better performance.

[0089] 3) Zero-sample prediction

[0090] Setting: Similarly, LLMs also show excellent zero-shot prediction capabilities. Compared with few-shot prediction, zero-shot prediction has more stringent requirements: after the model is trained on a certain dataset, it must be tested on a completely new dataset that has never been seen before, and still achieve good performance. This ability is particularly critical for time series models. Therefore, the embodiment of the present application further conducts experiments in the zero-shot prediction scenario to verify its generalization ability. In the zero-shot setting, the model is trained on dataset □ and tested on a completely different dataset ⊙ without retraining.

[0091] Results: This embodiment of the present application also conducted zero-shot prediction experiments on four ETT datasets. The results are shown in Table 4. Compared with other baseline models, the method of this embodiment of the present application demonstrated significant advantages. Compared with TIME-LLM and PatchTST, the method of this embodiment of the present application achieved average improvements of more than 5% and 10% respectively, demonstrating its transferability across different domains and further verifying the powerful ability of this method in zero-shot prediction.

[0092] Table 4

[0093]

[0094] Table 4 shows the zero-shot prediction results. The arrow symbol □->⊙ indicates that the model is trained on dataset □ and evaluated on a different dataset ⊙ (without retraining). All results are based on four different prediction horizons H∈(96,192,336,720).

[0095] Step S508: performing an ablation experiment.

[0096] 1) Ablation of SDC module

[0097] In order to analyze the impact of the SDC module, the embodiment of the present application conducted an ablation experiment by removing the SDC module, and the results are shown in Table 5. The three experiments in the table are: the original version (without adding any additional modules), the version with the SDC module, and the version with all improved modules. Among them, without and with the SDC module, the MSE / MAE on the ETTh1 dataset were 0.444 / 0.443, and the MSE / MAE on the Weather dataset were 0.256 / 0.273, respectively, with an average improvement of about 3%. It can be seen that the SDC module achieved the lowest MSE / MAE among all experimental settings and performed the best.

[0098] Table 5

[0099]

[0100] 2) Ablation of the modality alignment module.

[0101] To evaluate the impact of the modality alignment module on the model, the present embodiment removed the modality alignment module in the ablation experiment and compared it with other alignment methods. The experimental results on the Weather dataset are as follows: Figure 6 Among them, Scheme 1 is the alignment scheme used in the embodiment of this application; Scheme 2 is the result after removing the modal alignment module; Scheme 3 is the result of reprogramming using TIME-LLM; Scheme 4 is the result of using CALFPCA.

[0102] By comparing the experimental results, it can be clearly seen that the modal alignment module proposed in the embodiment of the present application is significantly better than other solutions in all indicators, achieving satisfactory results, and proving the effectiveness of the module.

[0103] The embodiments of the present application are based on previous work that mainly involves extracting statistical features and learning residuals to predict time series. However, existing technologies often overlook the difficulties in the residual learning process and the problem that the residuals themselves are easily interfered with. The embodiments of the present application are based on this problem and decompose the time series data into an integer part and a decimal part, thereby providing a more interpretable residual modeling framework. By studying the correlation between the two parts, this method enhances the ability to learn residual patterns, thereby improving the overall understanding of time series. This innovation makes up for the shortcomings of previous work and improves the prediction performance.

[0104] The superior performance of IDDLLM in this example can be attributed to several key factors. First, the SDC module decomposes the time series into integer and decimal components, enabling the model to capture the complex and underlying relationships within the time series, thereby enhancing the model's ability to extract meaningful patterns and dependencies. Second, selective cross-modal alignment using descriptive text effectively aligns the time series with the text, further improving the accuracy of the model's predictions. Furthermore, ablation experiments verified that each improvement significantly contributed to the overall performance improvement.

[0105] This application proposes a novel IDDLLM framework designed to enhance the applicability of large language models (LLMs) for time series forecasting. By using the SDC module to decompose time series into integer and fractional components, this effectively addresses the residual learning difficulties and extreme value sensitivity issues of previous methods. Furthermore, by combining text descriptions to optimize the alignment of text modalities with time series modalities, the model gains a deeper understanding of time series data.

[0106] This application also provides a large language model time series prediction device based on integer decimal decomposition, such as Figure 7As shown, it includes: a decomposition module 72, which is configured to decompose each numerical value in the time series data into an integer part and a decimal part, extract the correlation features between the integer part and the decimal part, and obtain a feature representation, wherein the correlation features are used to assist in residual understanding; an alignment module 74, which is configured to align the time series modality of the time series data with the text modality through a double cross attention mechanism to obtain a modality alignment representation; a fusion module 76, which is configured to fuse the time series data, the feature representation and the modality alignment representation to obtain a final input feature; a prediction module 78, which is configured to perform inference in a large language model based on the final input feature and output a prediction result.

[0107] It should be noted that the large language model time series prediction device based on integer-decimal decomposition provided in the above embodiment is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the large language model time series prediction device based on integer-decimal decomposition provided in the above embodiment and the large language model time series prediction method embodiment based on integer-decimal decomposition belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0108] Figure 8 Schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure is shown. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0109] like Figure 8 As shown, the electronic device includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 into the random access memory (RAM) 1003. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, ROM 1002 and RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0110] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0111] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A large language model time series prediction method based on integer-decimal decomposition, characterized in that: include: Decompose each value in the time series data into an integer part and a decimal part, extract the correlation features between the integer part and the decimal part, and obtain a feature representation, wherein the correlation features are used to assist in understanding the residual; The time series modality of the time series data is aligned with the text modality through a double cross attention mechanism to obtain a modality-aligned representation; fusing the time series data, the feature representation, and the modality alignment representation to obtain a final input feature; Based on the final input features, reasoning is performed in a large language model to output a prediction result.

2. The method according to claim 1, characterized in that Decompose each value in the time series data into an integer part and a decimal part, extract the correlation features between the integer part and the decimal part, and obtain a feature representation. The correlation features are used to assist in understanding the residual, including: Extracting the integer part of each value in the time series data by a first function and extracting the decimal part by a second function; A cross-attention mechanism is used to take the integer part as a query vector and the decimal part as a key vector and a value vector, and an attention representation between the integer part and the decimal part is extracted as the association feature.

3. The method according to claim 1, characterized in that The time series modality of the time series data is aligned with the text modality through a double cross attention mechanism to obtain a modality-aligned representation, including: In the first stage, the text description is used as the query vector, the pre-trained word embedding is used as the key vector and the value vector, and the first cross attention result is calculated; In the second stage, the time series is represented as the query vector, the first cross-attention result is used as the key vector and the value vector, and the cross-attention is calculated again to obtain the modality alignment representation.

4. The method according to claim 1, wherein The time series data, the feature representation and the modal alignment representation are fused to obtain a final input feature, including: weighted summing or concatenating the time series data representation, the feature representation and the modal alignment feature to obtain a fused input feature as the final input feature.

5. The method according to claim 1, wherein Based on the final input features, the large language model is used for inference and outputs prediction results, including: Performing a linear transformation on the final input feature to adjust it to be consistent with the input dimension of the large language model, and inputting the adjusted final input feature into the large language model; The large language model is used for reasoning, and the final hidden state output of the large language model is mapped to the target prediction sequence length and dimension through a linear mapping layer to obtain the prediction result of the final time series.

6. The method according to claim 1, characterized in that Before decomposing each value in the time series data into an integer part and a decimal part, the method further includes: Preprocessing the time series data, wherein the preprocessing includes at least one of the following: data cleaning, missing value filling, and outlier processing; Performing reversible normalization on the preprocessed time series data based on the distribution of the training set so that the mean of each dimension feature is 0 and the variance is 1; The normalized time series data is divided into continuous non-overlapping blocks of a preset length, and converted into a feature format that meets the input requirements of the large language model.

7. A large language model time series prediction device based on integer-decimal decomposition, characterized in that: include: a decomposition module configured to decompose each value in the time series data into an integer part and a decimal part, extract correlation features between the integer part and the decimal part, and obtain a feature representation, wherein the correlation features are used to assist in understanding the residual; an alignment module configured to align the time series modality and the text modality of the time series data through a double cross attention mechanism to obtain a modality-aligned representation; a fusion module configured to fuse the time series data, the feature representation, and the modality alignment representation to obtain a final input feature; The prediction module is configured to perform inference in the large language model based on the final input features and output a prediction result.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

9. A computer device, characterized in that: include: memory and processor, The memory stores a computer program; The processor is configured to execute a computer program stored in the memory, wherein the computer program enables the processor to execute the method according to any one of claims 1 to 6 when the computer program is executed.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images