Cross-domain time sequence completion method integrating pre-training language model

Abstract

The invention provides a cross-domain time sequence completion method of an integrated pre-training language model. The method comprises the following steps: inputting multivariable time sequences of multiple fields; missing masking operation is carried out on the complete data, after data normalization and blocking, missing masks are generated for the time sequence data, and the original time sequence is processed according to the masks; randomly generating two groups of masks which respectively represent different missing modes; shielding a part of values of the time sequences through masks to obtain the time sequences with the masks to respectively represent two different missing scenes; a data missing mode in a real scene is simulated, so that the model can adapt to various incomplete time sequence data; model parameters are initialized, and parameters of a time sequence completion basic model obtained through pre-training on a natural language are adopted as initialized weights; the input sequence is standardized and sliced. And a missing data complementing task can be executed on any incomplete time sequence which is not seen in the training stage.

Description

Technical Field

[0001] The present invention relates to the field of new generation information technology, and in particular to a cross-domain time series completion method integrating a pre-trained language model. Background Art

[0002] Real-world time series data often have missing values. Incomplete data complicates various time series applications, such as prediction and classification, and ultimately reduces the quality of time series analysis. This makes time series completion particularly important.

[0003] Many time series completion models are designed for specific missing data patterns and domains, such as randomly missing traffic time series. For example, Hsiang-Fu et al. proposed a matrix decomposition model designed specifically for multivariate time series completion in the article "Temporal regularized matrix factorization for high-dimensional time series prediction" published at the International Conference on Advances in Neural Information Processing Systems. For datasets with specific domains and specific missing rates, this model usually requires the use of low-rank priors as the optimization objective for completion to optimize a specialized model.

[0004] Recently, deep learning technology has shown good results in solving the problem of missing data completion. Yusuke et al. proposed using diffusion models to learn the underlying distribution of incomplete time series in their paper "Csdi: Conditional score-based diffusion models for probabilistic time series imputation" published at the international conference Advances in Neural Information Processing Systems. Du et al. proposed using a deep learning architecture with a self-attention mechanism to improve the prediction accuracy of the model on missing data in their paper "Self-attention-based imputation for timeseries" published in the journal Expert Systems with Applications.

[0005] Although deep learning-based completion methods have made significant progress, they often have limitations because these models are usually trained and used on a single domain-specific dataset. For example, a model trained on time series data in the transportation domain may not work well on missing data completion in other domains such as weather and electricity. However, it is technically challenging to effectively learn a general completion model that is applicable to multiple domains. The model must be highly adaptable to cope with different domains and missing data patterns. In addition, in some applications, it also needs to have the ability to quickly specialize to a specific domain while maintaining its generality. Summary of the invention

[0006] In view of the above problems in the prior art, the present application proposes a cross-domain time series completion method integrating a pre-trained language model, a cross-domain time series completion method integrating a pre-trained language model, comprising the following steps:

[0007] Step S1, prepare data sets from different fields and input multivariate time series from multiple fields;

[0008] Step S2, perform missing masking operation on complete data. After data normalization and block division, generate missing mask for time series data, and process the original time series according to the mask; randomly generate two sets of masks, representing different missing modes respectively; shield part of the values ​​of the time series by mask, and obtain masked time series representing two different missing scenarios respectively; simulate the data missing mode in real scenarios, so that the model can adapt to various incomplete time series data;

[0009] Step S3, initializing model parameters, using the parameters of the time series completion basic model pre-trained on natural language as the initialization weights;

[0010] Step S4: standardize the input sequence and slice the sequence, normalize each variable to eliminate the distribution difference of data in different fields; then, split the time series into several non-overlapping segments, and map each segment to the latent space through linear projection;

[0011] Step S5, pre-trained language model calculation, input the embedded data into the Transformer-based pre-trained language model for feature extraction; learn complex patterns and dependencies in time series data through the Transformer structure;

[0012] Step S6, extracting the last hidden state from the PLM and generating the final time series completion result through linear mapping;

[0013] Step S7, optimizing the model weight by combining the output time series with the time series before masking with the mean square error loss;

[0014] Step S8, continue training until the model converges.

[0015] In one embodiment, in step S3, the pre-trained time series completion basic model is fine-tuned on time series data in a specific field, and the fine-tuning process includes:

[0016] Prepare domain-specific datasets: collect domain-specific datasets;

[0017] Complete data missing masking: Different from the pre-training data masking strategy, it only needs to be masked once when fine-tuning in a specific domain;

[0018] Domain-specific embedding generates prefixes: During model pre-training, domain-specific embeddings are trained together, which contain temporal distribution information of different domains. During fine-tuning in a specific domain, domain-specific embeddings participate in training and are mapped to prefixes through the domain migration neural network layer based on the linear layer. They are then inserted into the backbone of the PLM in hierarchical order.

[0019] PLM calculation: At this stage, all parameters of PLM are frozen, and only the training of domain-specific embedding and domain transfer neural network layers is retained.

[0020] In one embodiment, the pre-trained language model architecture includes an embedding layer, a contrastive learning module, a PLM module, an output layer, and an output layer.

[0021] In one embodiment, the embedding layer includes temporal embedding, domain-specific embedding, sequence statistics embedding, fragment statistics embedding, and missing embedding.

[0022] In one embodiment, the domain transfer neural network layer is a two-layer multi-layer perceptron network that maps domain-specific embeddings into continuous prompts at each layer, transferring domain-specific knowledge to each layer of the model.

[0023] In one embodiment, during the fine-tuning process, the parameters of the domain-specific embeddings are frozen and not updated; these embeddings have learned domain-specific knowledge during the pre-training phase and are passed to each layer of the model through the domain transfer layer.

[0024] In one embodiment, the step of domain transfer includes: continuous prompts generated by the domain transfer layer contain domain-specific knowledge; these continuous prompts are inserted into each layer of the model to enhance the model's representation ability in a specific domain.

[0025] In one embodiment, during fine-tuning, randomly initialized sequential hints are added to the prefix of each layer; this randomly initialized information is updated during fine-tuning.

[0026] The above technical features can be combined in various suitable ways or replaced by equivalent technical features as long as the purpose of the present invention can be achieved.

[0027] The cross-domain time series completion method integrating a pre-trained language model provided by the present invention has at least the following beneficial effects compared with the prior art:

[0028] ① The time series completion method proposed in this paper introduces a novel solution that can perform missing data completion tasks for any incomplete time series that have not been seen during the training phase. To the best of our knowledge, this is the first attempt in this field.

[0029] ② We propose a “plug-and-play” fine-tuning technique that can seamlessly transform a general model into a domain-specific model with minimal data and computational resources without modifying the original weights of the model.

[0030] ③The general time series completion model consistently outperforms the domain-specific state-of-the-art methods in almost all missing rate completion tasks. In addition, the fine-tuned Nuwa time series completion model can be extended to time series prediction, and its prediction results are comparable to or even better than existing domain-specific time series prediction models. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will be described in more detail below based on embodiments and with reference to the accompanying drawings, wherein:

[0032] Figure 1 The pre-training flow chart of the timing completion basic model of the present invention is shown;

[0033] Figure 2 A domain specific fine tuning flow chart of the present invention is shown;

[0034] Figure 3 A schematic diagram of a model of the present invention is shown;

[0035] Figure 4 A schematic diagram of the field specific fine tuning of the present invention is shown;

[0036] Figure 5 Shows a schematic diagram of the division of multivariate time series by variable dimension;

[0037] Figure 6 A schematic diagram showing the partitioning of multivariate time series by time dimension is shown. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the accompanying drawings.

[0039] The present invention provides a cross-domain time series completion method integrating a pre-trained language model. Figure 1As shown in the figure, prepare datasets from different fields: The process starts with inputting a relatively complete time series dataset, which usually contains multivariate time series from multiple fields. The input data may come from different fields, and the distribution and scale may vary greatly. Collect more than 100 million time points.

[0040] Perform missing masking operation on complete data: After data normalization and segmentation, generate missing masks for time series data, and process the original time series according to the masks. Randomly generate two sets of masks, each representing a different missing pattern. Mask part of the time series values, and obtain masked time series representing two different missing scenarios. This can simulate the data missing pattern in real scenarios, allowing the model to adapt to various incomplete time series data. At the same time, generate different positive samples (the same time series with different masks) for the contrastive learning module, thereby enhancing the model's robustness to missing patterns.

[0041] Initialize model parameters: This process uses the parameters of the GPT2 model pre-trained on natural language as the initial weights. The advantage of this is that the intermediate parameters have a good pattern, and it is easy to get better performance by fine-tuning the model on the time series.

[0042] Standardize the input sequence and slice the sequence: Use the reversible instance normalization method to normalize each variable (set missing values ​​to 0) to eliminate the distribution differences of data in different fields. Then, split the time series into several non-overlapping patches, and map each patch to the latent space through a learnable linear projection. The purpose is to enhance the model's perception of domain information and reduce the impact of distribution differences.

[0043] Embedding Generation: There are three parallel embedding generations:

[0044] 1. Generation of time series segments: encoding the divided time series through a simple linear layer.

[0045] 2. Statistical embedding: Generate global and local statistics (such as minimum, median, maximum, trend) and map them to the latent space through linear projection.

[0046] 3. Missing embedding: Combined with the fragment missing rate, it is used to capture the missing patterns of different fragments.

[0047] Add the above three embeddings together to get the input information of the pre-trained language model.

[0048] Through multiple embeddings, the representation ability of time series is enriched, and the adaptability of the model to missing patterns and domain information is improved.

[0049] Pre-trained Language Model (PLM) calculation: The embedded data is input into the Transformer-based pre-trained language model for feature extraction. The Transformer structure is used to learn complex patterns and dependencies in time series data.

[0050] Output layer: Extract the last hidden state from PLM and generate the final time series completion result through linear mapping.

[0051] The model weights are optimized by combining the output time series with the time series before masking and the mean square error (MSE) loss.

[0052] Continue training until the model converges.

[0053] In one embodiment, Figure 2 As shown in the figure, a process of fine-tuning the pre-trained time series completion base model on the time series data in a specific domain is proposed.

[0054] Prepare domain-specific datasets: Collect available domain-specific datasets.

[0055] Complete data missing masking: Different from the data masking strategy for pre-training, only one masking is required when fine-tuning on a specific domain.

[0056] Domain-specific embedding generates prefixes: During model pre-training, domain-specific embeddings are trained together, which contain temporal distribution information of different domains. During fine-tuning in a specific domain, domain-specific embeddings participate in training and are mapped to prefixes through the domain migration neural network layer based on the linear layer. And they are inserted into the backbone of the PLM in hierarchical order.

[0057] PLM calculation: At this stage, all parameters of PLM are frozen, and only the training of domain-specific embedding and domain transfer neural network layers is retained.

[0058] In one embodiment, the present application proposes a time series filling framework based on a pre-trained language model (PLM). The model introduces a specific embedding method and combines contrastive learning with missing data reconstruction tasks to fill in any incomplete time series. The model architecture is as follows Figure 3 As shown, it mainly includes the following key components:

[0059] Embedding layer:

[0060] Temporal Embedding: Split the time series into non-overlapping segments and embed these segments into a latent space via a shared learnable linear projection.

[0061] Domain-specific embeddings: In cases where domain-specific adaptation is required, domain-specific embeddings are introduced to capture domain-specific knowledge.

[0062] Sequence Statistics Embedding: Generate statistics of the entire variable (such as minimum, median, maximum, and trend) and embed them into the latent space.

[0063] Fragment Statistics Embedding: Statistics are generated for each fragment and embedded into the latent space.

[0064] Missing embedding: Design a learnable parameter to capture the missing rate of each segment and add it to the corresponding embedded segment by multiplying it by the corresponding segment mask ratio.

[0065] Comparative Learning Module:

[0066] To improve the model's adaptability to different missing patterns, a contrastive learning module is introduced. For each input time series, two random masks with different mask ratios are generated and the masked time series are input into the PLM. The contrastive learning module ensures that the model learns similar representations for the same segment under different masks, while treating representations from other segments and sequences as negative samples.

[0067] PLM Module:

[0068] The model parameters are initialized as the weights of PLM. Considering the limitation of computing resources, only the parameters of the first six layers are used. The PLM module generates a representation of the time series by combining the above embedding information.

[0069] Output layer:

[0070] After passing through the PLM module, the domain-specific embeddings and variable statistical embeddings are discarded, leaving only the fragment representations as the input to the final layer. These representations are flattened and linearly mapped back to the original dimensions to generate the output of the model.

[0071] Domain-specific fine-tuning:

[0072] To achieve domain-specific models, the P-tuning v2 strategy is adopted. During fine-tuning, domain-specific information is inserted into each layer of the PLM without modifying the weights of the pre-trained model. Specifically, a two-layer MLP network is used to map domain-specific embeddings to continuous prompts in each layer, thereby enhancing the representation power of the model.

[0073] Output embedding:

[0074] The final output embedding is generated by passing the hidden layer representation of PLM to the output layer to fill in the missing time series data.

[0075] The deep learning model proposed in this application achieves the filling of any incomplete time series by combining the semantic information, missing patterns and statistical information of the time series. Through contrastive learning and domain-specific fine-tuning, the model performs well in cross-variable and cross-domain generalization, and can effectively fill time series data in different fields and missing patterns.

[0076] In one embodiment, Figure 4 As shown in the figure, the model already has the ability to generalize across variables and domains during the pre-training stage. In order to further adapt to the needs of specific domains, the model introduces a domain-specific fine-tuning mechanism. This mechanism inserts domain-specific prefix information into each layer of the pre-trained model to achieve adaptation to specific domains while maintaining the model's cross-domain generalization ability.

[0077] Domain transfer layer: This is a two-layer multi-layer perceptron network that maps domain-specific embeddings to sequential cues at each layer; effectively transferring domain-specific knowledge to each layer of the model.

[0078] Prefix information

[0079] Layer 1 prefix: Prefix information inserted in the first layer of the model, containing domain-specific knowledge.

[0080] Layer 2 prefix: Prefix information inserted in the second layer of the model, which also contains domain-specific knowledge.

[0081] Nth-layer prefix: Prefix information inserted in the Nth layer of the model, containing domain-specific knowledge.

[0082] Base model: the pre-trained language model (PLM). During fine-tuning, the model weights are frozen and not updated. The base model enhances its representation ability in a specific domain by combining domain-specific prefix information.

[0083] Output layer: After being processed by the base model and prefix information, the final output embedding is generated to fill in the time series data in a specific field.

[0084] Frozen domain-specific embeddings: During fine-tuning, the parameters of the domain-specific embeddings are frozen and not updated. These embeddings have learned domain-specific knowledge during the pre-training phase and are passed to each layer of the model through the domain transfer layer.

[0085] Transferred prefix information: Continuous hints generated by the domain transfer layer contain domain-specific knowledge. These hints are inserted into each layer of the model to enhance the model's representation ability in a specific domain.

[0086] Randomly initialized information: During fine-tuning, randomly initialized continuous cues are added to the prefix of each layer. These randomly initialized information are updated during fine-tuning to better fit the data of the specific domain.

[0087] The domain-specific fine-tuning mechanism proposed in this application achieves adaptation to a specific domain by inserting domain-specific prefix information into each layer of the pre-trained model. This mechanism not only maintains the cross-domain generalization ability of the model, but also significantly improves the performance of the model in a specific domain. Through the domain transfer layer and randomly initialized information, the model can effectively capture and utilize domain-specific knowledge, thereby performing well in time series completion tasks in specific domains.

[0088] In one embodiment, Figure 5-6 As shown in Figure 1, in time series data analysis, the traditional division of training, validation, and test sets is usually based on the time dimension. However, this division method has limitations in evaluating the model's cross-variable and cross-domain generalization ability. In order to more rigorously evaluate the model's generalization ability on new variables and new domains, we propose a new division method, namely division by variable dimension.

[0089] Multivariate Time Series: Suppose we have a multivariate time series dataset which contains multiple variables (e.g. sensor data, meteorological data, etc.).

[0090] Partition by variable dimension: Partition the multivariate time series dataset by variable dimension instead of time dimension. Specifically, assign different variables to the training set, validation set, and test set.

[0091] Training set: contains time series data of some variables, which is used to train the model. The data of these variables are relatively complete and the missing rate is low.

[0092] Validation set: contains time series data of another part of variables, which is used to verify the performance of the model. The data of these variables may have a higher missing rate and are used to evaluate the performance of the model under different missing patterns.

[0093] Test set: contains the time series data of the remaining variables, which is used to finally evaluate the performance of the model. The data of these variables may come from different fields and are used to evaluate the cross-domain generalization ability of the model.

[0094] Evaluate cross-variable generalization ability: By dividing the dataset by variable dimension, the model's generalization ability on new variables can be more rigorously evaluated. The knowledge learned by the model on the training set needs to be verified and tested on new variables.

[0095] Evaluate cross-domain generalization capabilities: The variables in the test set may come from different fields, which requires the model to generalize to new fields. This division method can more realistically simulate actual application scenarios, that is, the model may only be exposed to some variables and fields during training, but needs to handle completely new variables and fields during deployment.

[0096] Dividing multivariate time series datasets by variable dimension is a new evaluation method that can more rigorously evaluate the generalization ability of the model on new variables and new domains. By assigning different variables to the training set, validation set, and test set, the knowledge learned by the model during training needs to be verified and tested on new variables and new domains, thereby more realistically simulating actual application scenarios.

[0097] In one embodiment, the present application proposes a general time series interpolation framework based on a pre-trained language model (PLM), aiming to address the limitations of existing models in terms of generalization. Its core technical solutions include the following aspects:

[0098] 1. Model architecture: (such as Figure 1 (shown)

[0099] Patch segmentation: Split the time series into non-overlapping patches and embed the patches into a latent space using shared linear projections to better model time series data from different domains.

[0100] Embedded Design:

[0101] Statistical embedding: Generate statistics (such as mean, median, maximum, etc.) for each segment and the entire sequence and embed them into the latent space to better align with the time series data.

[0102] Missing embedding: A learnable parameter is designed to capture the missing rate of each segment and multiplied by the segment’s mask ratio to better account for missing data in the target time series.

[0103] Domain-specific embeddings: In cases where a domain-specific model is needed, a domain-specific embedding is introduced and inserted before the fragment embedding to enhance the representation power of the model.

[0104] Contrastive Learning: Through the contrastive learning module, the model learns to generate similar representations on the same segments with different mask ratios and regards them as positive samples, while representations from other segments and sequences are regarded as negative samples to enhance the model's adaptability to different missing patterns.

[0105] Output layer: After PLM, the N fragment representations are retained as the input of the final output layer, and a linear mapping is used to map them back to the original dimension to obtain the model output.

[0106] 2. Domain-specific fine-tuning: (e.g. Figure 2 (shown)

[0107] Domain transfer layer: A two-layer MLP network is used to map domain-specific embeddings into prefix information, which is inserted into each layer of the PLM and combined with randomly initialized hint information as the key and value of each layer.

[0108] Fine-tuning process: Freeze all parameters of PLM and only fine-tune the randomly initialized hint information and domain transfer layer to achieve a lightweight domain-specific model.

[0109] 3. Evaluation method: (e.g. Figure 5 (shown)

[0110] Variable dimension division: Divide the multivariate time series data along the variable dimension, assign different variables to training, validation, and test sets, simulate real-world scenarios, and evaluate the model's cross-variable and cross-domain generalization capabilities.

[0111] 4. Application:

[0112] Interpolation: The proposed time series completion method outperforms existing domain-specific models on almost all datasets at various missing rates, demonstrating its superior performance in interpolation tasks.

[0113] Prediction: By combining the time series completion method proposed in the present invention with an existing prediction model (such as TimesNet), the performance of the prediction model can be effectively improved, especially when dealing with the prediction problem of missing data.

[0114] Domain transfer zero-sample learning: The time series completion method proposed in this paper has excellent zero-sample learning ability and can interpolate time series data in other fields without additional training.

[0115] The time series completion method proposed in the present invention realizes universal time series interpolation across variables and fields through its unique design and evaluation method, providing an effective solution to the problem of missing data in the real world.

[0116] Although the present invention is described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the present invention. It should therefore be understood that many modifications may be made to the exemplary embodiments and that other arrangements may be devised without departing from the spirit and scope of the present invention as defined by the appended claims. It should be understood that the various dependent claims and features described herein may be combined in a manner different from that described in the original claims. It should also be understood that features described in conjunction with individual embodiments may be used in other described embodiments.

Claims

1. A cross-domain time series completion method integrating a pre-trained language model, characterized in that: The following steps are involved: Step S1, prepare data sets from different fields and input multivariate time series from multiple fields; Step S2, perform missing masking operation on complete data. After data normalization and block division, generate missing mask for time series data, and process the original time series according to the mask; randomly generate two sets of masks, representing different missing modes respectively; shield part of the values ​​of the time series by mask, and obtain masked time series representing two different missing scenarios respectively; simulate the data missing mode in real scenarios, so that the model can adapt to various incomplete time series data; Step S3, initializing model parameters, using the parameters of the time series completion basic model pre-trained on natural language as the initialization weights; Step S4: standardize the input sequence and slice the sequence, normalize each variable to eliminate the distribution difference of data in different fields; then, split the time series into several non-overlapping segments, and map each segment to the latent space through linear projection; Step S5, pre-trained language model calculation, inputting the embedded data into the Transformer-based pre-trained language model for feature extraction; Learn complex patterns and dependencies in time series data through the Transformer structure; Step S6, extracting the last hidden state from the PLM and generating the final time series completion result through linear mapping; Step S7, optimizing the model weight by combining the output time series with the time series before masking with the mean square error loss; Step S8, continue training until the model converges.

2. The cross-domain time series completion method integrating pre-trained language models according to claim 1 is characterized in that: In step S3, the pre-trained time series completion basic model is fine-tuned on the time series data in a specific field. The fine-tuning process includes: Prepare domain-specific datasets: collect domain-specific datasets; Complete data missing masking: Different from the pre-training data masking strategy, it only needs to be masked once when fine-tuning in a specific domain; Domain-specific embedding generates prefixes: During model pre-training, domain-specific embeddings are trained together, which contain temporal distribution information of different domains. During fine-tuning in a specific domain, domain-specific embeddings participate in training and are mapped to prefixes through the domain migration neural network layer based on the linear layer. They are then inserted into the backbone of the PLM in hierarchical order. PLM calculation: At this stage, all parameters of PLM are frozen, and only the training of domain-specific embedding and domain transfer neural network layers is retained.

3. The cross-domain time series completion method integrating pre-trained language models according to claim 2 is characterized in that: The pre-trained language model architecture includes embedding layer, contrastive learning module, PLM module, output layer, and output layer.

4. The cross-domain time series completion method integrating pre-trained language models according to claim 3 is characterized in that: The embedding layer includes temporal embedding, domain-specific embedding, sequence statistics embedding, fragment statistics embedding, and missing embedding.

5. The cross-domain time series completion method integrating pre-trained language models according to claim 2, characterized in that: The domain transfer neural network layer is a two-layer multi-layer perceptron network that maps domain-specific embeddings to continuous prompts at each layer, transferring domain-specific knowledge to each layer of the model.

6. The cross-domain time series completion method integrating pre-trained language models according to claim 2, characterized in that: During fine-tuning, the parameters of the domain-specific embeddings are frozen and not updated; these embeddings have learned domain-specific knowledge during the pre-training phase and are passed to each layer of the model through the domain transfer layer.

7. The cross-domain time series completion method integrating pre-trained language models according to claim 2, characterized in that: The steps of domain transfer include: continuous prompts generated by the domain transfer layer contain domain-specific knowledge; these continuous prompts are inserted into each layer of the model to enhance the model's representation ability in a specific domain.

8. The cross-domain time series completion method integrating pre-trained language models according to claim 2, It is characterized in that During fine-tuning, a randomly initialized continuous cue is added to the prefix of each layer; These randomly initialized information are updated during the fine-tuning process.

Images