Plug-and-play parameterized memory enhancement prediction method for time sequence basic model

Abstract

The invention discloses a plug-and-play parameterized memory enhancement prediction method for a time sequence basic model, and belongs to the technical field of time sequence prediction. The method aims at solving the problem that when an existing time sequence basic model is deployed in an actual field, it is difficult to consider the field adaptive capacity and avoid disastrous forgetting. Comprising the following steps: encoding keys into key embedding vectors, and performing paired storage with values to obtain a knowledge base; establishing an index for the key embedding vector so as to retrieve a historical mode as a supervision signal in an offline training stage of a memory decoder; constructing a memory decoder to obtain a memory enhanced future prediction sequence; adjusting model parameters based on an arrangement invariance loss function; carrying out dimension alignment on the memory enhanced future prediction sequence and the original future prediction sequence, splicing and extracting global association features, and calculating to obtain memory correction features; and taking the memory correction characteristic as a residual error updating item through a gating mechanism, and superposing the residual error updating item to an original future prediction sequence to obtain a global correction prediction result. The method is used for predicting the time sequence.

Description

Technical Field

[0001] This invention relates to a plug-and-play parametric memory-enhanced prediction method for time series basic models, belonging to the field of time series prediction technology. Background Technology

[0002] Time series forecasting (TSF), as a core technology of data-driven decision-making systems, plays a fundamental supporting role in key areas such as financial risk management, healthcare monitoring, industrial IoT, and climate modeling. Its forecasting accuracy directly impacts risk control and decision-making efficiency. Its technological development has mainly gone through three key stages:

[0003] Traditional statistics and signal processing stage: represented by ARIMA, state-space models, etc., have good interpretability in trend and seasonality modeling, but rely on strong distribution assumptions and are difficult to deal with high-dimensional, multivariate and highly non-stationary real-world time series data.

[0004] The emergence of deep learning-specific model stages—RNN, LSTM, Temporal CNN, and attention mechanism models—has significantly improved the ability to model complex patterns, but they need to be trained separately for specific tasks and datasets, and their generalization ability is limited in cross-domain and distribution-varying scenarios.

[0005] Large-scale pre-trained foundation model stage: Inspired by natural language processing and computer vision, Time Series Foundation Models (TSFMs) have emerged. These models, pre-trained on massive amounts of heterogeneous time series data, possess powerful representation capabilities and zero-shot prediction performance, establishing performance benchmarks in multiple benchmark tests and fully validating the potential of the transfer learning paradigm in time series analysis.

[0006] Despite significant progress in TSFMs, a core unresolved issue remains when deploying them in practical applications: the structural tension between adaptive capabilities and catastrophic forgetting. This tension manifests itself in the following ways:

[0007] Real-world time series are highly non-stationary, often containing sporadic but crucial pattern changes (such as sudden changes in financial market trends, acute physiological anomalies in medical monitoring, and sudden failures in industrial systems). These low-frequency, high-impact patterns play a decisive role in prediction accuracy and risk control. However, general-purpose pre-trained time series models (TSFMs) lack sensitivity to domain-specific anomalies; and domain adaptation achieved through full parameter tuning or large-scale fine-tuning sacrifices the model's original generalization ability, leading to catastrophic forgetting, a problem particularly prominent in scenarios involving conceptual drift, structural mutations, and long-term evolution.

[0008] To alleviate this contradiction, existing research has developed two main technical approaches, but both have significant limitations:

[0009] I. Domain-Adaptive Pretraining (DAPT): Adaptation is achieved by pretraining the base model on domain data. It is effective in some scenarios, but it has two major drawbacks: high computational cost and high resource consumption; it is difficult to achieve stable cross-domain generalization while retaining general knowledge, and it still faces the risk of forgetting.

[0010] II. Retrieval-Augmented Generation (RAG) Paradigm Shift: Drawing inspiration from natural language processing, this approach dynamically retrieves historical fragments related to the current input during the inference phase to enhance predictive capabilities without modifying the underlying model parameters. Representative works such as TS-RAG, while improving zero-shot performance and providing some interpretability, have fundamental limitations: 1) Online similarity search and retrieval operations introduce significant inference delays, limiting applications in real-time or resource-constrained scenarios; 2) The knowledge injection method is superficial: retrieved historical patterns are only used as input context for prediction and are not parameterized and internalized into the model representation, hindering a structured understanding of key patterns and making robust generalization difficult in similar but not perfectly matched scenarios.

[0011] III. Limitations of Research on Neural Memory Mechanisms: Research on neural memory mechanisms in parallel with RAG aims to alleviate long-term dependence and forgetting by introducing learnable external memory or long-term state modules to store and recall historical information. However, it has significant shortcomings: 1) It is highly coupled with specific network structures, requiring end-to-end joint training, and is difficult to apply directly to frozen basic models; 2) It focuses more on general sequence modeling capabilities and less on explicit modeling of domain-specific, low-frequency but high-impact mutation and drift patterns.

[0012] In summary, current time series foundational models suffer from critical deficiencies in their memory mechanisms: models either update implicit historical patterns through parameter updates (costly and prone to forgetting) or explicitly retrieve historical fragments through external searches (inefficient and difficult to internalize). How to inject key domain knowledge into time series foundational models in a controllable, interpretable, and generalizable form without modifying the host model parameters or requiring online searches has become an urgent open challenge. Summary of the Invention

[0013] To address the problem that existing time series baseline models struggle to balance domain adaptability and avoid catastrophic forgetting when deployed in real-world applications, this invention provides a plug-and-play parametric memory-enhanced prediction method for time series baseline models.

[0014] This invention provides a plug-and-play parametric memory-enhanced prediction method for time series basic models, comprising:

[0015] Building a knowledge base: The input raw time series is divided into multiple segments using a sliding window with a fixed step size. Each segment is divided into a historical context segment and a future segment. The historical context segment is used as the key and the future segment is used as the value. The key is encoded into a key embedding vector and paired with the corresponding value to store as a historical pattern, thus obtaining the knowledge base. An index is built for all key embedding vectors to retrieve historical patterns as supervision signals during the offline training phase of the memory decoder.

[0016] Constructing a memory decoder: A Transformer encoder is used to map the key embedding vector to a shared latent space; multiple independent future branches are configured in the shared latent space for future prediction, and each future branch divides the prediction horizon into multiple consecutive time blocks. Each time block is temporally modeled by a block Transformer encoder; the future prediction results of multiple future branches are passed through a linear layer to obtain multiple memory-enhanced future prediction sequences, forming an unordered prediction set; during the training process, the memory decoder uses a permutation-invariant loss function to calculate the loss based on the memory-enhanced future prediction sequences and the corresponding supervision signals, and adjusts the model parameters accordingly.

[0017] Base model prediction: Keep the base model parameters frozen and obtain the original future prediction sequence based on the original time series;

[0018] Adaptive Memory Fusion: All memory-enhanced future prediction sequences and the original future prediction sequences are dimensionally aligned through projection, discarding, and positional encoding. All aligned prediction sequences are concatenated, and then a lightweight Transformer encoder is used to extract the global association features of each prediction sequence based on the original future prediction sequence. The weights of each future branch and the base model are calculated based on all global association features and weighted pooling is performed to obtain memory-corrected features. The memory-corrected features are used as residual update terms through a gating mechanism and superimposed on the original future prediction sequence to obtain the globally corrected prediction result.

[0019] According to the plug-and-play parametric memory-enhanced prediction method for time series basic models of the present invention, in the process of constructing the knowledge base, the original time series is divided into training set interval, validation set interval and test set interval; and a segment in the training set interval, validation set interval or test set interval is selected as the data segment of the corresponding interval.

[0020] According to the plug-and-play parametric memory-enhanced prediction method for time series-based models of the present invention, the index is organized based on the approximate nearest neighbor method of the inverted file structure.

[0021] The method for selecting the supervision signal in the plug-and-play parametric memory-enhanced prediction method for time series basic models according to the present invention is as follows:

[0022] Set the time exclusion radius so that its length is equal to the length of the segment; combine the current key and its corresponding current value of the memory decoder into a query window; retrieve the preset number of segments in the knowledge base that have the highest similarity to the current key of the query window, and filter out segments that are in the same original time series as the query window and whose time interval is less than the time exclusion radius; use the value of the filtered segment as a supervision signal.

[0023] According to the plug-and-play parametric memory-enhanced prediction method for time series fundamental models of the present invention, the permutation invariance loss function is expressed as follows: :

[0024] ,

[0025] In the formula To enhance the future prediction sequence of the k-th memory, For the first The k-th monitoring signal in the permutation, Let K be the set of all permutations of the K monitoring signals.

[0026] The plug-and-play parametric memory-enhanced prediction method for time series-based fundamental models according to the present invention includes a method for dimensional alignment of all memory-enhanced future prediction sequences and the original future prediction sequences, comprising:

[0027] All memory-enhanced future prediction sequences and original future prediction sequences are mapped to the latent space through a parameter-shared projection network to obtain memory-enhanced future prediction latent vectors and original future prediction latent vectors of the same dimension.

[0028] The memory-enhanced future prediction latent vector and the original future prediction latent vector are randomly discarded by applying Bernoulli masks. Learnable positional codes are added to the memory-enhanced future prediction latent vector and the original future prediction latent vector after the information is randomly discarded, and then layer normalization is performed to obtain dimension-aligned normalized memory-enhanced future prediction latent vector and normalized original future prediction latent vector.

[0029] The plug-and-play parametric memory-enhanced prediction method for time series-based models according to the present invention obtains memory-corrected features as follows:

[0030] The normalized memory-enhanced future prediction latent vector and the normalized original future prediction latent vector are concatenated and then input into a lightweight Transformer encoder to extract global association features for each prediction sequence.

[0031] The weights of each future branch and the base model are calculated as follows:

[0032] ,

[0033] In the formula , For the k-th weight, For the global association features of the k-th predicted latent vector, It is a single-layer weighted grid. For temperature coefficient, ;

[0034] Memory correction features are represented as :

[0035] .

[0036] According to the plug-and-play parametric memory-enhanced prediction method for time series fundamental models of the present invention, the residual update term is expressed as follows: :

[0037] ,

[0038] In the formula It is a two-layer sensor. This is a learnable scaling factor.

[0039] According to the plug-and-play parametric memory-enhanced prediction method for time series fundamental models of the present invention, the globally corrected prediction result is expressed as... :

[0040] ,

[0041] In the formula This is the original future prediction sequence. This is a global gating signal. , This is element-wise multiplication.

[0042] According to the plug-and-play parametric memory-enhanced prediction method for time series-based models of the present invention, a weighted loss function is used in the adaptive memory fusion process. :

[0043] ,

[0044] In the formula For mathematical expectation, For sample weights, The actual value;

[0045] ,

[0046] In the formula For hyperparameters, It is a very small positive hyperparameter.

[0047] The beneficial effects of this invention are as follows: This invention proposes a plug-and-play parameterized memory augmentation framework, MEMTS, whose core is a lightweight memory knowledge augmenter. This augmenter actively learns mutational and drifting patterns from domain history through offline pre-training, patterns that are easily missed by the basic model, and encodes them into compact, pluggable parameterized memories. This invention offers multiple augmentation capabilities: it can be seamlessly integrated into the original zero-sample model, instantly injecting it with awareness of key pattern mutations; it can also collaborate with fine-tuned models to compensate for anomalous pattern recognition not fully covered in their training data. The entire process requires no modification to the host model parameters and incurs no online retrieval overhead. It learns mutational and drifting patterns in domain history offline using parameterized memory modules and combines with different time-series base models in a plug-and-play manner, thus exhibiting significant advantages in efficiency, stability, and deployability.

[0048] The method of this invention possesses high flexibility: it can be directly combined with a frozen zero-sample time series foundation model (TSFM) to instantly inject the model with the ability to perceive key anomalous patterns; it can also work in conjunction with a fine-tuned model to supplement rare structures not fully covered in the training data, thereby alleviating overfitting and forgetting problems. Because the memory exists in a parameterized form, the method of this invention outperforms retrieval-based methods in terms of efficiency, interpretability, and system deployment.

[0049] Extensive experiments on multiple real-world benchmark datasets in finance, healthcare, and industry demonstrate that introducing only a single MEMTS module with approximately 0.5B parameters can achieve stable and significant performance improvements on various time series baseline models, with a maximum relative improvement of 6.84%. Simultaneously, it significantly enhances the model's ability to identify historical abrupt changes and peak patterns. These results validate the effectiveness and universality of parameterized memory as a bridge connecting general time series baseline models with domain knowledge.

[0050] The method of this invention corresponds to a plug-and-play parameterized memory augmentation framework MEMTS for time series foundation models. MEMTS achieves efficient domain adaptation while completely freezing the host TSFM parameters, and can be seamlessly integrated into time series foundation models of different architectures and sizes.

[0051] This invention is the first to systematically replace non-parametric retrieval mechanisms with compact parameterized memory modules in time series prediction tasks. It injects key domain patterns into the model representation space in an internalized manner, thereby eliminating the online retrieval overhead in the inference stage and exhibiting stronger robust generalization ability in scenarios where the distribution offset is similar to but not exactly a match with the pattern.

[0052] The method of this invention can uniformly enhance the performance of zero-shot prediction and fine-tuned models: MEMTS can serve as an external memory supplement module for zero-shot TSFM, significantly improving the model's ability to perceive mutable and drifting patterns; it can also work in conjunction with fine-tuned models to supplement rare or long-tailed structures in training data, effectively alleviating overfitting and catastrophic forgetting problems.

[0053] This invention achieves significant performance gains and interpretable modeling capabilities through lightweight parameter design. It captures key mutation points and anomalous patterns with a parameter count far smaller than that of the host model and provides interpretable analysis of domain-critical memory activations. Attached Figure Description

[0054] Figure 1 This is a flowchart of the plug-and-play parametric memory-enhanced prediction method for time series basic models described in this invention.

[0055] Figure 2 This is a flowchart of an existing knowledge-based online retrieval enhancement prediction method;

[0056] Figure 3 This is a flowchart of the knowledge base construction process;

[0057] Figure 4 This is a schematic diagram of the memory decoder architecture;

[0058] Figure 5 This is a schematic diagram of the architecture of the adaptive memory fusion module that implements the adaptive memory fusion step. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] Specific Implementation Method 1: Combination Figures 1 to 5 As shown, this invention provides a plug-and-play parametric memory-enhanced prediction method for time series basic models, including:

[0061] Building a knowledge base: The input raw time series is divided into multiple segments using a sliding window with a fixed step size. Each segment is divided into a historical context segment and a future segment. The historical context segment is used as the key and the future segment as the value. The key is encoded into a key embedding vector and paired with the corresponding value to store as a historical pattern, thus obtaining the knowledge base. An index is built for all key embedding vectors to retrieve historical patterns as supervision signals during the offline training phase of the memory decoder.

[0062] Constructing a memory decoder: A Transformer encoder is used to map the key embedding vector to a shared latent space; multiple independent future branches are configured in the shared latent space for future prediction, and each future branch divides the prediction horizon into multiple consecutive time blocks. Each time block is temporally modeled by a block Transformer encoder; the future prediction results of multiple future branches are passed through a linear layer to obtain multiple memory-enhanced future prediction sequences, forming an unordered prediction set; during the training process, the memory decoder uses a permutation-invariant loss function to calculate the loss based on the memory-enhanced future prediction sequences and the corresponding supervision signals, and adjusts the model parameters accordingly.

[0063] Base model prediction: Keep the base model parameters frozen and obtain the original future prediction sequence based on the original time series;

[0064] Adaptive Memory Fusion: All memory-enhanced future prediction sequences and the original future prediction sequences are dimensionally aligned through projection, discarding, and positional encoding. All aligned prediction sequences are concatenated, and then a lightweight Transformer encoder is used to extract the global association features of each prediction sequence based on the original future prediction sequence. The weights of each future branch and the base model are calculated based on all global association features and weighted pooling is performed to obtain memory-corrected features. The memory-corrected features are used as residual update terms through a gating mechanism and superimposed on the original future prediction sequence to obtain the globally corrected prediction result.

[0065] This implementation describes a plug-and-play parameterized memory augmentation framework, MEMTS (Memory-Enhanced Module for Time Series), designed to improve the adaptability of Time Series Foundation Models (TSFM) to domain mutations and distribution shifts. Unlike traditional retrieval augmentation methods, MEMTS does not rely on online retrieval. Instead, it encodes key historical patterns into lightweight, parameterizable memory modules through offline pre-training, thereby achieving zero-latency knowledge injection during the inference phase.

[0066] like Figure 2As shown, the traditional knowledge base-based retrieval augmentation process involves two paths: given an input sequence, one path feeds into the base model to generate an initial prediction; the other path uses an online retrieval module to query a large external knowledge base in real time, searching for historical fragments semantically similar to the current input. These retrieved fragments are then fed into an adaptive fusion model, integrated with the output of the base model, and finally generate the augmented prediction. While this method is intuitive, it has two inherent drawbacks: first, the online retrieval process introduces significant latency, limiting its application in real-time scenarios; second, knowledge injection remains only at the input level, failing to internalize key patterns into the model's own representational capabilities.

[0067] Figure 1 As shown, the input sequence is only fed into the base model, while the memory component, as an independent parameterized module, does not require any online retrieval operation. Figure 1 The memory component corresponds to the memory decoder, and the adaptive fusion model is used to achieve adaptive memory fusion.

[0068] This implementation replaces the online retrieval module with a lightweight memory component, which is an offline pre-trained neural network. During pre-training, this network learns and encodes crucial predictive mutation and drift patterns from a knowledge base. This component directly outputs a set of parameterized future prediction candidates. These candidates, along with the base model's original output, enter the adaptive fusion model to complete the final prediction. Throughout the process, there is no online retrieval; all knowledge is embedded within the model's parameters, achieving zero-latency, high-efficiency, plug-and-play enhancement.

[0069] This implementation proposes an offline parameterized memory augmentation paradigm that successfully eliminates the bottleneck of online retrieval without sacrificing performance, and improves the deployability and stability of the model.

[0070] like Figure 3 As shown, this implementation constructs a knowledge base that is an offline, structured memory storage system, used to provide high-quality, unbiased historical pattern samples for subsequent memory decoders. The entire construction process is divided into three stages: data slicing, feature encoding, and index construction. While ensuring retrieval efficiency, the focus is on avoiding data leakage problems introduced by adjacent time windows.

[0071] During the data slicing stage, the original time series is divided into a series of consecutive segments by a sliding window. Each segment is split into two parts: the historical context as the key, and the immediately following future segment as the value. This ensures that each memory unit contains a complete "context-prediction" relationship, providing semantically rich input for the subsequent memory decoder.

[0072] Figure 3In this process, the original time series is sliced ​​using a sliding window to generate key-value pairs. The key windows are then converted into key-embedded vectors by an encoder. The top k most relevant value windows from all candidate values ​​are searched and stored together in the knowledge base and indexed. These k value windows serve as external information for memory enhancement.

[0073] Specifically, let the i-th time series be represented as:

[0074] ,

[0075] in This represents the sequence length. A sliding window strategy is used to extract memory units from each time series. For any valid time starting point t, the key window and its corresponding value window are defined as follows:

[0076]

[0077] Where K and V represent the historical context length and prediction step size, respectively. The sliding window advances in steps s, and requires... This is to ensure that the key value window is intact and valid.

[0078] Furthermore, in the process of constructing the knowledge base, the original time series is divided into training set intervals, validation set intervals, and test set intervals; a segment that falls within the training set interval, validation set interval, or test set interval is selected as the data segment of the corresponding interval.

[0079] The index is organized based on an approximate nearest neighbor method of inverted file structure.

[0080] To prevent time leakage between different dataset partitions, each time series is partitioned independently based on time.

[0081] For length of The sequence is defined with two time points:

[0082] ,

[0083] in and These represent the proportions of the training set and the validation set, respectively. This results in three non-overlapping time intervals:

[0084] ;

[0085] A sliding window is assigned to the dataset partition only if its complete key-value time span falls entirely within the corresponding interval, thus strictly avoiding time overlap across sets.

[0086] Subsequently, in the feature encoding stage, each key window It will be mapped to a vector representation:

[0087]

[0088] in This represents the time-series encoder. Ultimately, during the index building phase, each memory unit in the knowledge base can be represented as... All key vectors The indexes are uniformly constructed and organized based on the inverted file structure (IVF) and the approximate nearest neighbor method to support efficient similarity retrieval on large-scale knowledge bases.

[0089] The method for selecting the supervisory signal is as follows:

[0090] Set the time exclusion radius so that its length is equal to the length of the segment; combine the current key and its corresponding current value of the memory decoder into a query window; retrieve the preset number of segments in the knowledge base that have the highest similarity to the current key of the query window, and filter out segments that are in the same original time series as the query window and whose time interval is less than the time exclusion radius; use the value of the filtered segment as a supervision signal.

[0091] During the retrieval phase, if k-nearest neighbor search is directly performed, the model often prioritizes samples that are highly temporally adjacent to the query window. However, these samples typically have a large number of overlapping observations, which can easily lead to information leakage. Therefore, drawing on the temporal masking concept of RAFT, a temporal exclusion constraint is introduced for candidate neighbors within the same time series. Specifically, for the query window... If candidate samples satisfy:

[0092]

[0093] Then that candidate will be directly eliminated. To exclude the radius of time, set it to This ensures that candidate samples do not overlap with the query window in terms of either key or value over time. Under this constraint, the sample with the highest similarity is selected. The system identifies several neighbors and uses their corresponding value fragments as external information for subsequent memory enhancement predictions.

[0094] Figure 4As shown, the memory decoder is the core component of the MEMTS framework for implementing parameterized knowledge injection. Its design goal is to learn and generate multiple possible future evolution paths from a pre-trained knowledge base without relying on online retrieval, thereby explicitly modeling the uncertainty and multimodality in time series forecasting. The entire decoding process consists of three stages: context encoding, parallel branch generation, and block-level sequence modeling, ultimately outputting an unordered set of future predictions.

[0095] Figure 4 The input is the key-embedded query window, which is first mapped to a shared latent space by a Transformer encoder. Then, the model generates K independent future branches in parallel. Each branch performs temporal modeling internally using a block-level Transformer encoder, and finally outputs the complete future sequence through a linear layer. The entire process involves no cross-branch information exchange, ensuring the independence of each branch.

[0096] After completing the knowledge base construction and nearest neighbor retrieval, the model needs to model multiple future fragments retrieved based on historical context representation and generate a set of possible future evolution results. Unlike traditional single future prediction, this implementation model the problem as an unordered multi-future prediction task, that is, the model needs to predict multiple reasonable future candidates from historical information without pre-setting the output order.

[0097] Let the history key vector corresponding to a given query window be represented as follows: in This representation is extracted from the history window by an arbitrary time-series encoder. First, it is mapped to the latent space to obtain a shared global representation:

[0098]

[0099] in This represents a coding function composed of multiple nonlinear mappings. It serves as a common condition variable for all subsequent future forecast branches.

[0100] To characterize the uncertainty and multimodal features of the future, parallel generation There are 10 future prediction branches, each corresponding to a possible future evolution pattern. Considering the length of the future prediction interval is... Divide it into There are 3 consecutive time blocks (chunks), each time block having a length of 1. ,satisfy This is to enhance the stability of long sequence prediction.

[0101] For the The first future branch, its first The initial representation corresponding to each time block is defined as follows:

[0102]

[0103] in Learnable query vectors specific to each branch, used to distinguish semantic features of different branches. This design embeds time block positions, allowing different future branches to share historical semantics while possessing independent generative trajectories.

[0104] Subsequently, only within each future branch, the time block sequence is analyzed. Apply sequence modeling operations:

[0105]

[0106] in , The time modeling module (Transformer Encoder) prevents information exchange between different future branches, thus avoiding mutual interference between prediction results.

[0107] A linear mapping is performed on the implicit representation of each time block to obtain the block-based prediction results:

[0108]

[0109] By splicing together, a complete future prediction sequence can be restored:

[0110] ,

[0111] Ultimately, the model outputs an unordered set of future predictions. .

[0112] Since the future sequences output by the model do not have a fixed order, and the supervision signals also constitute an unordered set, directly calculating the loss based on index alignment would introduce an artificial order constraint. Therefore, a permutation-invariant set matching loss function is used for training. Let the prediction set be... The corresponding set of supervised targets is Then the permutation invariance loss function is expressed as: :

[0113] ,

[0114] In the formula To enhance the future prediction sequence of the k-th memory, For the first The k-th monitoring signal in the permutation, Let K be the set of all permutations of the K monitoring signals.

[0115] Permutation invariance loss function By selecting the match with the smallest error among all possible matches, the model is ensured to focus only on the numerical matching degree between the predicted results and the true future set, rather than being forced to learn an arbitrary fixed output order. This avoids the mean collapse problem in multi-future prediction tasks and encourages the model to generate diverse and complementary future evolution patterns.

[0116] Adaptive memory fusion mechanism:

[0117] After retrieving relevant historical patterns and generating multiple candidate future predictions, an adaptive mechanism is needed to fuse this external information with the original predictions of the base model. Traditional RAG methods typically concatenate context at the input level, but this strategy significantly increases inference latency and makes it difficult to make targeted modifications to the internal representation of the base model. This implementation proposes a lightweight Adaptive Memory Fusion (AMF) module, which injects external memory knowledge directly into the prediction output layer in a parameterized manner, avoiding online retrieval overhead while maintaining a sensitive response to key pattern mutations.

[0118] like Figure 5 As shown, the AMF module receives Input sequences: A memory-enhanced future prediction sequence Original predictions from the base model The entire fusion process can be divided into three stages: representation alignment, adaptive weighted fusion, and gated residual update. After projection, discarding, position encoding, and Transformer encoder processing, the attention weights and gated signals are calculated by the weight network and gated network, respectively, and finally the final prediction is generated through residual connections.

[0119] Furthermore, methods for dimensional alignment of all memory-enhanced future prediction sequences and original future prediction sequences include:

[0120] All memory-enhanced future prediction sequences and original future prediction sequences are mapped to the latent space through a parameter-shared projection network to obtain memory-enhanced future prediction latent vectors and original future prediction latent vectors of the same dimension.

[0121] The memory-enhanced future prediction latent vector and the original future prediction latent vector are randomly discarded by applying Bernoulli masks. Learnable positional codes are added to the memory-enhanced future prediction latent vector and the original future prediction latent vector after the information is randomly discarded, and then layer normalization is performed to obtain dimension-aligned normalized memory-enhanced future prediction latent vector and normalized original future prediction latent vector.

[0122] Alignment phase: First, all input sequences are mapped to a unified latent space through a parameter-shared projection network.

[0123]

[0124] in and It is a two-layer MLP network that maps the input to a nonlinear transformation. Hidden space.

[0125] To enhance training stability, random discarding is applied to both memory tokens and base tokens during the training phase:

[0126]

[0127] in and Let be a Bernoulli mask, and the drop probabilities are respectively and .

[0128] Subsequently, to preserve temporal relative positional information, learnable positional encodings are added to all hidden representations:

[0129]

[0130] in This is the position embedding vector.

[0131] The method for obtaining memory correction features is as follows:

[0132] The normalized memory-enhanced future prediction latent vector and the normalized original future prediction latent vector are concatenated and then input into a lightweight Transformer encoder to extract global association features for each prediction sequence.

[0133] The aligned concatenation is represented as a sequence. Enter a The Transformer encoder of the layer performs context modeling:

[0134]

[0135] The encoder captures the dependencies between prediction branches through a self-attention mechanism. To control the contribution of each branch, a learnable weight network is introduced.

[0136] The weights of each future branch and the base model are calculated as follows:

[0137] ,

[0138] In the formula , For the k-th weight, For the global association features of the k-th predicted latent vector, It is a single-layer weighted grid. This is a temperature coefficient used to adjust the sharpness of the weight distribution. ;

[0139] Finally, the memory-corrected features obtained by weighted fusion of the encoder output are represented as follows: :

[0140] .

[0141] Gated residual update stage: The fused representation generates a residual update term through a prediction head, and the residual update term is represented as follows: :

[0142] ,

[0143] In the formula It is a two-layer sensor. This is a learnable scaling factor. The tanh function limits the residual magnitude.

[0144] To finely control the flow of information, two gating mechanisms are introduced:

[0145] Step-gating: Generates an independent gating signal for each time step. This allows the model to selectively accept memory corrections at different time points.

[0146] Global gating: Generates a single gating signal for the entire sequence. , as a simplified variant.

[0147] The global correction prediction result is expressed as :

[0148] ,

[0149] In the formula This is the original future prediction sequence. This is a global gating signal. , This is an element-wise multiplication. This design ensures that the base predictions remain dominant even when memories are irrelevant, enhancing the model's robustness.

[0150] During training, the AMF module minimizes the mean squared error between the prediction and the true value:

[0151] .

[0152] It is worth noting that, to improve sensitivity to mutation patterns, a loss-weighted strategy is further introduced:

[0153] A weighted loss function is used in the adaptive memory fusion process. :

[0154] ,

[0155] In the formula For mathematical expectation, For sample weights, The actual value;

[0156] ,

[0157] In the formula For hyperparameters, It is a very small positive hyperparameter.

[0158] This strategy assigns higher weights to samples with larger prediction errors, making the model pay more attention to sudden changes that are difficult for the basic model to capture.

[0159] The AMF module achieves three key objectives: (1) encodes external memory knowledge into a lightweight network in a parametric form; (2) adaptively fuses multi-source information to generate differentiated corrections for different time points; and (3) ensures the dominance of the base prediction through residual connections while providing critical corrections when needed. This mechanism enables MEMTS to respond quickly to historical abrupt change patterns without online retrieval, significantly improving the robustness and adaptability of the time series base model in the face of distribution shifts.

[0160] Experiments: MEMTS was evaluated on six publicly available time series benchmark datasets: ECL, AustraliaRainfall, METR-LA, PEMS04, PEMS08, and Solar. These datasets cover multiple domains, including electricity consumption, meteorology, traffic flow, and energy production, and have different temporal characteristics, sampling frequencies, and prediction challenges.

[0161] Comparisons are made with current state-of-the-art temporal baseline models, including Chronos-Small, Chronos-Base, Sundial, MOMENT, and Moirai. All baseline methods use their official implementations and pre-trained weights.

[0162] The evaluation metrics used were mean squared error (MSE) and mean absolute error (MAE). To verify the effectiveness of the MEMTS method at different time periods, the prediction lengths were 96, 192, 336, and 720 steps, respectively.

[0163] Main experimental results:

[0164] The zero-shot prediction performance of the MEMTS method of this invention was compared with that of various baseline methods on six publicly available time series benchmark datasets. The method of this invention achieved the best performance on all datasets, demonstrating the effective ability of MEMTS to capture key pattern abrupt changes.

[0165] The superiority of MEMTS is not only evident in zero-shot scenarios but also in fine-tuning scenarios. Further, a set of post-fine-tuning enhancement experiments were designed: first, each base model was fine-tuned on the target dataset, and then the trained MEMTS module was integrated into the fine-tuned model. Experimental results show that even when the model has been optimized for a specific dataset, MEMTS still delivers additional performance gains. This indicates that MEMTS does not simply compensate for the generalization deficiencies of the base model but can identify and inject rare or sporadic patterns that are difficult to learn even after fine-tuning.

[0166] Experiments show that MEMTS significantly improves prediction performance regardless of whether Chronos-Small, Chronos-Base, Sundial, MOMENT, or Moirai is used as the pedestal. This result fully validates the plug-and-play nature of MEMTS, which does not rely on specific architectures or pre-training strategies, but provides a lightweight, transferable knowledge enhancement pathway on top of any base model. The improvement is particularly pronounced on weaker pedestal models, indicating that MEMTS is especially adept at empowering resource-constrained or capability-limited base models.

[0167] Furthermore, MEMTS exhibits stability across varying task complexities, and its adaptability stems from its core mechanism: the memory decoder generates multiple future candidates, and the fusion module dynamically selects the most relevant patterns through gating weights. Therefore, when the dataset itself exhibits strong periodicity and regularity, MEMTS tends to retain the output of the base model; while when mutations, drifts, or anomalies exist in the data, it automatically amplifies the memory enhancement signal to achieve precise correction.

[0168] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.

Claims

1. A plug-and-play parametric memory-enhanced prediction method for time series basic models, characterized in that... include, Building a knowledge base: The input raw time series is divided into multiple segments using a sliding window with a fixed step size. Each segment is divided into a historical context segment and a future segment. The historical context segment is used as the key and the future segment is used as the value. The key is encoded into a key embedding vector and paired with the corresponding value to store as a historical pattern, thus obtaining the knowledge base. An index is built for all key embedding vectors to retrieve historical patterns as supervision signals during the offline training phase of the memory decoder. Constructing a memory decoder: A Transformer encoder is used to map the key embedding vector to a shared latent space; the shared latent space is configured with multiple independent future branches for future prediction, each future branch divides the prediction horizon into multiple consecutive time blocks, and each time block is temporally modeled by a block Transformer encoder; the future prediction results of multiple future branches are passed through a linear layer to obtain multiple memory-enhanced future prediction sequences, which constitute an unordered prediction set; During training, the memory decoder uses a permutation-invariant loss function to calculate the loss based on the memory-enhanced future prediction sequence and the corresponding supervision signal, and adjusts the model parameters accordingly. Base model prediction: Keep the base model parameters frozen and obtain the original future prediction sequence based on the original time series; Adaptive memory fusion: All memory-enhanced future prediction sequences and original future prediction sequences are dimensionally aligned through projection, discarding, and positional encoding; all aligned prediction sequences are concatenated, and then a lightweight Transformer encoder is used to extract the global association features of each prediction sequence based on the original future prediction sequence. The weights of each future branch and the base model are calculated based on all global association features and weighted pooling is performed to obtain memory-corrected features. By using a gating mechanism, the memory correction features are used as residual update terms and superimposed onto the original future prediction sequence to obtain a global correction prediction result.

2. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 1, characterized in that, In the process of building the knowledge base, the original time series is divided into training set interval, validation set interval and test set interval; a segment in the training set interval, validation set interval or test set interval is selected as the data segment of the corresponding interval.

3. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 1, characterized in that, The index is organized based on an approximate nearest neighbor method of inverted file structure.

4. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 1, characterized in that, The method for selecting the supervisory signal is as follows: Set the time exclusion radius so that its length is equal to the length of the segment; combine the current key and its corresponding current value of the memory decoder into a query window; retrieve the preset number of segments in the knowledge base that have the highest similarity to the current key of the query window, and filter out segments that are in the same original time series as the query window and whose time interval is less than the time exclusion radius; use the value of the filtered segment as a supervision signal.

5. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 1, characterized in that, The permutation invariance loss function is expressed as: : ， In the formula To enhance the future prediction sequence of the k-th memory, For the first The k-th monitoring signal in the permutation, Let K be the set of all permutations of the K monitoring signals.

6. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 1, characterized in that, Methods for dimensional alignment of all memory-enhanced future prediction sequences and original future prediction sequences include: All memory-enhanced future prediction sequences and original future prediction sequences are mapped to the latent space through a parameter-shared projection network to obtain memory-enhanced future prediction latent vectors and original future prediction latent vectors of the same dimension. The memory-enhanced future prediction latent vector and the original future prediction latent vector are randomly discarded by applying Bernoulli masks. Learnable positional codes are added to the memory-enhanced future prediction latent vector and the original future prediction latent vector after the information is randomly discarded, and then layer normalization is performed to obtain dimension-aligned normalized memory-enhanced future prediction latent vector and normalized original future prediction latent vector.

7. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 6, characterized in that, The method for obtaining memory correction features is as follows: The normalized memory-enhanced future prediction latent vector and the normalized original future prediction latent vector are concatenated and then input into a lightweight Transformer encoder to extract global association features for each prediction sequence. The weights of each future branch and the base model are calculated as follows: ， In the formula , For the k-th weight, For the global association features of the k-th predicted latent vector, It is a single-layer weighted grid. For temperature coefficient, ; Memory correction features are represented as : 。 8. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 7, characterized in that, The residual update term is represented as : ， In the formula It is a two-layer sensor. This is a learnable scaling factor.

9. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 8, characterized in that, The global correction prediction result is expressed as : ， In the formula This is the original future prediction sequence. This is a global gating signal. , This is element-wise multiplication.

10. The plug-and-play parametric memory-enhanced prediction method for time series basic models according to claim 9, characterized in that, A weighted loss function is used in the adaptive memory fusion process. : ， In the formula For mathematical expectation, For sample weights, The actual value; ， In the formula For hyperparameters, It is a very small positive hyperparameter.

Images