Time series prediction method based on multi-modal contrast learning technique

Abstract

The application provides a time series prediction method based on a multi-modal contrast learning technology, original multi-variable time series data are converted into structured visual representation and language representation respectively, without relying on external natural language or real image data, an internally consistent multi-modal representation can be constructed, the deep semantic understanding ability of a model for a complex operation state of rail transit is effectively enhanced, a multi-modal contrast learning mechanism is introduced, visual and text modal representations are aligned in a shared embedding space, positive sample consistency is maximized and negative sample interference is inhibited through an InfoNCE loss, the robustness and generalization ability of time series features are significantly improved, and the importance of each variable for a prediction task is dynamically evaluated by using the aligned multi-modal representation, key variables are automatically screened, not only the interference of redundant information is reduced, but also the prediction precision and calculation efficiency of the model in a high-dimensional multi-variable scene are improved.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of intelligent operation and maintenance of rail transit and artificial intelligence. Specifically, it relates to a time series prediction method based on multimodal comparative learning technology, which is applicable to the fusion analysis, status monitoring and operation trend prediction of multi-source heterogeneous data in rail transit systems. Background Technology

[0002] In rail transit systems, time series data are widely used in multiple key scenarios such as train operation status monitoring, track health assessment, passenger flow forecasting, and energy consumption management. These data are typically generated continuously in a high-frequency, multi-dimensional manner. Accurate modeling and prediction of these data are of great significance for ensuring operational safety, improving scheduling efficiency, and optimizing resource allocation. Traditional time series forecasting methods mainly rely on statistical models (such as ARIMA and SARIMA) or shallow machine learning models (such as support vector regression and random forest). These methods perform well when dealing with linear and stationary sequences, but they often fail to capture deep dynamic patterns when faced with the complex characteristics of nonlinearity, non-stationarity, high noise, and multivariate coupling that are common in rail transit systems.

[0003] In recent years, deep learning technologies (such as recurrent neural networks (RNN), long short-term memory networks (LSTM), gated recurrent units (GRU), and the Transformer architecture) have made significant progress in time series modeling, enabling automatic extraction of temporal dependencies and end-to-end prediction. However, these methods are still mainly limited to single-modal numerical inputs, that is, directly using raw sensor readings or historical observations as model inputs. They lack a deep understanding of the semantic structure of the data and are difficult to effectively integrate complementary information from different sources or different forms of expression, thus limiting the model's generalization ability and interpretability in complex scenarios.

[0004] With the development of artificial intelligence technology, researchers have begun to explore combining time series with other modal information to improve modeling performance. One emerging trend is to use large language models (LLM) to "translate" time series into natural language descriptions, such as describing temperature change trends as "a continuous rise followed by a sharp drop", or summarizing train delay patterns as "frequent delays during the morning rush hour". These methods attempt to enhance the model's understanding of time series patterns through the semantic abstraction capabilities of language. However, such conversions usually rely on manually designed rules or pre-trained language models, which not only carries the risk of information loss, but also the generated text tags are essentially discrete symbol sequences, making it difficult to retain the continuity and fine-grained dynamic features of the original time series. More importantly, natural language itself lacks the intuitive ability to express visual patterns in time series (such as periodic fluctuations, abrupt changes, trend inflection points, etc.), while human experts often rely on visual forms such as charts and waveforms to make quick judgments when analyzing time series. Therefore, conversion methods that rely solely on text modality cannot fully simulate the multi-dimensional perception mechanism of human time series, resulting in significant limitations in the model's ability to capture complex spatiotemporal patterns. Furthermore, most existing methods do not consider the differences in correlation between variables in multivariate time series. When making predictions, they often treat all input variables the same, ignoring the fact that some variables may contain redundant or noisy information, thus affecting prediction accuracy and computational efficiency.

[0005] To address the aforementioned issues, recent research has attempted to introduce multimodal learning frameworks to enhance time series understanding by fusing various information formats such as vision, text, and even audio. However, these methods typically rely on external modal data (such as surveillance videos, maintenance logs, or manual annotations). In specialized fields like rail transit, such multimodal data is often difficult to obtain, annotation is costly, and alignment with core time series data presents technical challenges. More importantly, existing methods lack a mechanism for automatically generating structured multimodal representations from within the original numerical sequences, resulting in superficial multimodal fusion that fails to truly achieve semantic complementarity and synergy. Furthermore, although contrastive learning has shown promise in computer vision and natural language processing... It exhibits strong representation learning capabilities, enabling it to learn invariant features by bringing positive sample pairs closer together and pushing negative sample pairs apart. However, its application in time series multimodal modeling is still in the initial exploratory stage, and a systematic framework suitable for rail transit scenarios has not yet been formed. In particular, in terms of variable selection, existing models mostly adopt fixed weights or simple attention mechanisms, which are difficult to dynamically identify the most discriminative variable combinations under specific prediction tasks. Therefore, there is an urgent need for a new modeling method that can automatically generate structured visual and textual modalities from the original time series, achieve semantic alignment through contrastive learning, and combine variable selection mechanisms for efficient multivariate prediction, in order to break through the current technical bottlenecks and improve the intelligent operation and maintenance and decision support capabilities of rail transit systems. Summary of the Invention

[0006] To overcome the shortcomings of the existing technologies, the present invention aims to propose a time series prediction method based on multimodal contrastive learning technology in the field of rail transit. The core technical idea is to construct a multimodal contrastive learning framework. The overall framework can not only enhance the model's ability to represent complex rail transit operation data, such as train operation status and equipment monitoring signals, but also improve the accuracy and generalization of multi-step prediction. It is suitable for typical application scenarios such as fault early warning, passenger flow prediction, and energy consumption optimization.

[0007] To achieve the above objectives, this invention provides a time series prediction method based on multimodal contrastive learning technology, comprising the following steps:

[0008] Step S1: Map the original multivariate time series data into structured visual modalities and text modalities respectively;

[0009] Step S2: Convert the visual modal mapped from the original multivariate time series data into a visual representation;

[0010] Step S3: Convert the text modality mapped from the original multivariate time series data into a language representation;

[0011] Step S4: Design a contrastive learning mechanism to align visual and linguistic representations in a shared embedding space. By maximizing the consistency of positive sample pairs and minimizing the similarity of negative sample pairs, learn temporal features and calculate their contrastive loss.

[0012] Step S5: Dynamically evaluate the importance of each variable to the prediction task using the aligned multimodal representation, and automatically select the most discriminative input variables for subsequent modeling;

[0013] Step S6: Generate the predicted loss, combine the predicted loss and the contrastive loss to obtain the overall training loss and perform method prediction.

[0014] Furthermore, the visual modality includes temporal images and heatmaps; the textual modality includes symbolic sequences and semantic descriptions.

[0015] Furthermore, the visual representation transformation step in step S2 is as follows:

[0016] Step S2.1: Given historical multivariate time series observations of length T ,in , N Representing different recorded variables, where T is the known historical time step, the goal is to predict the future of the time series. S time step ;

[0017] Step S2.2: Fix the window sizeL window The process involves normalizing each variable in the original multivariate time series data and plotting each time series data variable with a specific color to help distinguish the variable type and align it with the linguistic representation of the time series in the multimodal feature space. This process can be expressed as the following formula:

[0018] ;

[0019] Step S2.3: Convert the time series into a series of images Image sequences of time series X Img Input to a pre-trained and frozen multimodal visual encoder E v In this process, a series of feature maps are generated. f 1 ，f 2 , . . . , f N}, where each feature map f i Depend on generate;

[0020] Step S2.4: Since the multimodal visual backbone network remains frozen during training, an additional mapping layer is introduced. Proj img To align the visual features of the time series with the multimodal feature space, the above visual representation is denoted as... ;

[0021] Step S2.5: For multivariate time series data of length T X i 1:T Define its visual representation as .

[0022] Furthermore, the language representation conversion step in step S3 is as follows:

[0023] Step S3.1: Introduce a block-segmentation strategy to divide the time series into subsequence-level blocks, and use these blocks as input blocks for the Transformer. The specific definition is shown in the following formula:

[0024]

[0025]

[0026]

[0027] in, PLThis indicates the patch length, while Indicates the number of fragments. S Indicates stride length;

[0028] Step S3.2: A randomly initialized, learnable linear layer is introduced as the embedding layer for the time series data, also known as the Tokenizer.

[0029] ;

[0030] Step S3.3: Add a learnable embedding before the word segmentation sequence of each variable. ; This is called 'category segmentation', and then a pre-trained multimodal language model is used. E text Learning language representations by embedding time series data into a multimodal language representation space:

[0031]

[0032] Where [., .] indicates the concatenation of time series segmentation and [class] segmentation. e pos To represent learnable positional embeddings, this invention uses the same [class] tokenization for each given variable. v patch ;

[0033] Step S3.4: Obtain the linguistic representation of the time series as shown in the following formula:

[0034]

[0035] Will The first segment in the sequence is considered a [category] segment, denoted as To align with visual representation, the language classification representation is defined. .

[0036] Furthermore, the specific steps of step S4 are as follows:

[0037] Step S4.1: Steps S2 and S3 can process multi-element time series of graphs of length T. X 1:T To obtain visual representation and language classification representation ,in , N Indicates the number of variables. D It is the feature dimension;

[0038] Step S4.2: Use a batch of data to calculate the loss and obtain the visual representation. and language classification representation ,in B Indicates batch size;

[0039] Step S4.3: Incorporate the multimodal contrast alignment of time series data into the standard contrast framework, i.e., based on the InfoNCE loss function, the specific definition of which is shown below:

[0040]

[0041]

[0042] Where τ is a learnable temperature parameter, φ(·, ·) represents the cosine similarity, and each multimodal pair represents a sequence of modes of length τ. T time series N One variable, positive sample pairs in and The positive samples are formed from the same sample, while the negative samples are formed from different samples in the batch.

[0043] Step S4.4: Calculate the contrast loss bidirectionally as follows:

[0044] .

[0045] Furthermore, the specific steps of step S5 are as follows:

[0046] Step S5.1: For a length of T Multivariate time series X 1:T Each variable v i ∈R 1×T Each word is treated as a separate word and is not processed into blocks. Therefore... X 1:T It can be represented as X ∈R N×T ,in N Indicates the number of variables. T Indicating the time step, after word segmentation, the time series is converted into... ,in D It is a hidden dimension;

[0047] Step S5.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Input into a pre-trained multimodal language encoder Generate variant hierarchical time series representations The process is defined as follows:

[0048]

[0049]

[0050] Among them, the variant-level time series is represented as H and Standardization needs to be performed using layer normalization.

[0051] Step S5.3: Select the most relevant variable features by employing a cross-attention layer in the Transformer decoder, where As a query H can Simultaneously serving as both key and value, the above process can be formally defined as: W q , W k and W v These are the weight matrices for query, key, and value projection in the cross-attention mechanism. d k This represents the dimensionality of each head in multi-head attention;

[0052] Step S5.4: Finally, An improved Transformer decoder is formed by processing a LayerNorm and a two-layer feedforward neural network FFN, together with the cross-attention layer of the Transformer decoder described in step S5.3.

[0053] Furthermore, the specific steps of step S6 are as follows:

[0054] Step S6.1: After obtaining Next, for each variable in step S5, these features are combined with the multimodal language representation of the time series, rather than simply appending them to the end of the feature sequence. The last word in the multimodal language representation is replaced, and this replacement is driven by the padding strategy used.

[0055] Step S6.2: Apply a flattened layer, then connect it to a linear header to generate the prediction results. ;

[0056] Step S6.3: For all prediction tasks, the mean squared error (MSE) loss is used for evaluation. However, for the open-source M4 dataset, the symmetric mean absolute percentage error loss is used.

[0057] Conduct evaluation;

[0058] Step S6.4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require LMSE and L SMAPE Both are referred to as prediction loss. L gen Therefore, the overall training loss combines the prediction loss and the contrastive loss term: , where λ1 and λ2 are weighting coefficients.

[0059] The beneficial effects of this invention are as follows: The "time series prediction method based on multimodal contrastive learning technology" proposed in this invention has significant advantages. First, by transforming the original multivariate time series data into structured visual and linguistic representations (such as semantic descriptions), it can construct an inherently consistent multimodal representation without relying on external natural language or real image data, effectively enhancing the model's deep semantic understanding of complex operational states of rail transit (such as train operation parameters, equipment monitoring signals, etc.). Second, by introducing a multimodal contrastive learning mechanism, it aligns visual and textual modal representations (linguistic representations) in a shared embedding space, maximizing positive sample consistency through InfoNCE loss. Suppressing negative sample interference significantly improves the robustness and generalization ability of time series features. Furthermore, by utilizing aligned multimodal representations to dynamically evaluate the importance of each variable to the prediction task and automatically selecting key variables, not only is redundant information interference reduced, but the prediction accuracy and computational efficiency of the model in high-dimensional, multivariate scenarios are also improved. In addition, the overall framework has strong compatibility and is applicable to various typical rail transit application scenarios such as fault early warning, passenger flow prediction, and energy consumption optimization, and demonstrates superior accuracy and stability compared to existing methods in multi-step prediction tasks. In summary, this invention enhances the expressive power of time series modeling while also considering practicality and scalability, providing reliable technical support for intelligent operation and maintenance and decision-making in rail transit. Attached Figure Description

[0060] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0061] The general multimodal time series contrastive learning framework proposed in this invention mainly includes two components: a visual component and a language component. Specifically, in order to realize the general application of the method proposed in this invention, this invention further designs a model framework, which consists of the following step modules: (i) a multimodal visual step module, (ii) a multimodal language step module, and (iii) a variable selection step module.

[0062] The multimodal vision module comprises three parts: visualization preprocessing for time series, a pre-trained and frozen multimodal vision backbone network for extracting time series image features, and a trainable multimodal projection layer. First, the visualization preprocessing converts the original time series signal into an image through a designed normalization step. Then, the original values ​​are normalized and visualized in different colors within the image. By converting the original time series into an image, this invention can extract time series features from a visual perspective using existing multimodal vision encoders. Furthermore, following existing common methods, this invention inputs visual features into a learnable multimodal mapping layer to transform and represent the visual features of the time series, placing them into a multimodal time series feature space. Therefore, this multimodal vision module plays a crucial role in the proposed multimodal framework, enabling the conversion of numerical data from the time series into a visual representation space for alignment with the linguistic representation space.

[0063] In the variable selection step module, this invention divides each variable of the time series data into blocks and uses a learnable time series word segmenter to convert the block-based time series data signal into language words. Then, the category words of each variable are concatenated and input into a pre-trained language encoder to obtain the language representation of the time series data. In addition, the category word segmentation introduced in this invention is used for multimodal alignment. Specifically, the most significant relevant variable features are selected from the multivariate time series data using classification information in cross-modal space. Finally, the aligned language representation is input into the generator to generate the final prediction.

[0064] The framework first maps the original multivariate time series data into structured visual modalities (such as time series images, heatmaps, etc.) and textual modalities (such as symbolic sequences, semantic descriptions, etc.). Both modalities are generated directly from the numerical sequences without relying on external natural language or real image data. Subsequently, a contrastive learning mechanism is designed to align the representations of visual and textual modalities in a shared embedding space. By maximizing the consistency of positive sample pairs and minimizing the similarity of negative sample pairs, the model learns more robust, complementary, and semantically rich temporal features. To further improve prediction performance, this invention also uses the aligned multimodal representations to dynamically evaluate the importance of each variable to the prediction task and automatically selects the most discriminative input variables for subsequent modeling.

[0065] Based on the above-described overall framework, this invention proposes a time series prediction method for the rail transit field based on multimodal contrastive learning technology. The specific method steps are as follows:

[0066] Step S1: Map the original multivariate time series data into structured visual modalities and text modalities respectively;

[0067] Step S2: Convert the visual modal mapped from the original multivariate time series data into a visual representation;

[0068] Step S3: Convert the text modality mapped from the original multivariate time series data into a language representation;

[0069] Step S4: Design a contrastive learning mechanism to align visual and linguistic representations in a shared embedding space. By maximizing the consistency of positive sample pairs and minimizing the similarity of negative sample pairs, learn temporal features and calculate their contrastive loss.

[0070] Step S5: Dynamically evaluate the importance of each variable to the prediction task using the aligned multimodal representation, and automatically select the most discriminative input variables for subsequent modeling;

[0071] Step S6: Generate the predicted loss, combine the predicted loss and the contrastive loss to obtain the overall training loss and perform method prediction.

[0072] Furthermore, the visual modality includes temporal images and heatmaps; the textual modality includes symbolic sequences and semantic descriptions.

[0073] Furthermore, the specific steps in step S2 are as follows: Step S2.1: In the time series prediction task, given historical multivariate time series observations of length T... ,in N represents different recorded variables, and T is a known historical time step. Therefore, this invention aims to predict the next S time steps of a time series. ;

[0074] Step S2.2: Since the original time series data contains multivariate numerical signals, and the numerical ranges of each variable differ to varying degrees, in order to visualize these data in the graph with a reasonable numerical range, this invention uses a fixed window size. L window Instead of normalizing all time series data or mini-batch data, this invention normalizes each variable individually. This reduces the impact of outliers (maximum or minimum values) compared to normalizing values ​​in mini-batch or all data. Next, each time series variable is plotted with a specific color to help distinguish variable types and align them with the linguistic representation of the time series in the multimodal feature space. This process can be expressed as the following formula:

[0075] ;

[0076] Step S2.3: Therefore, the present invention converts the time series into a series of images. Inspired by current research, this invention uses time-series image sequences. XImg Input to a pre-trained and frozen multimodal visual encoder E v In this process, a series of feature maps are generated. f 1 ，f 2 , . . . , f N}, where each feature map f i Depend on generate;

[0077] Step S2.4: Since the multimodal visual backbone network remains frozen during training, this invention introduces an additional mapping layer. Proj img To align the visual features of the time series with the multimodal feature space, similar to existing research methods, this invention denotes the above visual representation as... ;

[0078] Step S2.5: Finally, for multivariate time series data of length T... X i 1:T The present invention defines its visual representation as .

[0079] Furthermore, the specific steps in step S3 are as follows:

[0080] Step S3.1: Inspired by the ViT and BERT models, this invention introduces a block-segmentation strategy to divide the time series into subsequence-level blocks, which are then used as input blocks for the Transformer. The specific definition is shown in the following formula:

[0081]

[0082]

[0083]

[0084] in, PL This indicates the patch length, while Indicates the number of fragments. S Indicates stride length;

[0085] Step S3.2: Unlike previous Transformer-based time series prediction models (such as PatchTST), this invention employs a multimodal pre-trained model instead of training a Transformer from scratch. In previous studies, words had to be segmented using a pre-trained tokenizer before being input into the language encoder. However, due to the significant domain differences between time series data and natural language, this invention cannot use pre-trained language tokenizers such as CLIPTokenizer. To address this issue, this invention introduces a randomly initialized learnable linear layer as the embedding layer for the time series data, also known as the Tokenizer.

[0086]

[0087] ;

[0088] Step S3.3: Subsequently, inspired by existing models, this invention, in order to apply a multimodal language encoder to the time series domain, adds a learnable embedding before the word segmentation sequence of each variable. This invention refers to it as 'category segmentation'. Then, this invention utilizes a pre-trained multimodal language model... E text Time series data are embedded into a multimodal language representation space to learn language representations.

[0089]

[0090] Where [., .] indicates the concatenation of time series segmentation and [class] segmentation. e pos To represent learnable positional embeddings, this invention uses the same [class] tokenization for each given variable. v patch ;

[0091] Step S3.4: Therefore, the present invention obtains the following linguistic representation of the time series:

[0092]

[0093] Following the contrastive learning strategy in existing methods, this invention will The first segment in the sequence is considered a [category] segment, denoted as To align with visual representation, the language classification representation is defined. .

[0094] Furthermore, the specific steps of step S4 are as follows:

[0095] Step S4.1: Steps S2 and S3 can process multi-element time series of graphs of length T. X 1:T To obtain visual representation and language classification representation ,in , N Indicates the number of variables. D It is the feature dimension;

[0096] Step S4.2: Then, the present invention uses a batch of data to calculate the loss and obtain the visual representation. and language classification representation , where B represents the batch size;

[0097] Step S4.3: Following this, the present invention incorporates multimodal contrast alignment of time series into the standard contrast framework, namely, based on the InfoNCE loss function, the specific definition of which is as follows:

[0098]

[0099]

[0100] Where τ is a learnable temperature parameter, φ(·, ·) represents the cosine similarity, and each multimodal pair represents a sequence of modes of length τ. T time series N One variable, positive sample pairs in and The positive samples are formed from the same sample, while the negative samples are formed from different samples in the batch.

[0101] Step S4.4: Thus, the bidirectional comparison loss of the present invention is calculated as follows:

[0102] .

[0103] This invention designs a novel variable selection step module to effectively utilize variable information from different perspectives in the original time series data. Inspired by existing research, this invention uses the [class] word segmentation in step S3 as the "query" and identifies the most relevant time series variable-level word segmentation through a cross-attention mechanism. Time series word segmentation is generated by segmenting the time series data and then encoding it using the shared multimodal language model designed in the previous section.

[0104] Furthermore, the specific steps of step S5 are as follows:

[0105] Step S5.1: For a multivariate time series of length T X 1:TThis step differs from the block-based strategy used in step S3; each variable... v i ∈R 1×T Each word is treated as a separate word and is not processed into blocks. Therefore... X 1:T It can be represented as X ∈R N×T ,in N Indicates the number of variables. T Indicating the time step, after word segmentation, the time series is converted into... ,in D It is a hidden dimension;

[0106] Step S5.2: Then, the present invention will Input into a pre-trained multimodal language encoder Generate variant hierarchical time series representations The process is defined as follows:

[0107]

[0108]

[0109] Among them, the variant-level time series is represented as H and Standardization needs to be performed using layer normalization.

[0110] Step S5.3: Subsequently, the present invention selects the most relevant variable features by employing a cross-attention layer of the Transformer decoder, wherein... As a query, H can serve as both a key and a value. Formally, the above process can be defined as follows: Wq, Wk, and Wv are the weight matrices for the query, key, and value projections in the cross-attention mechanism, respectively. d k This represents the dimensionality of each head in multi-head attention;

[0111] Step S5.4: Finally, An improved Transformer decoder is formed by processing a LayerNorm and a two-layer feedforward neural network FFN, together with the cross-attention layer of the Transformer decoder described in step S5.3.

[0112] Furthermore, the specific steps of step S6 are as follows:

[0113] Step S6.1: After obtaining Subsequently, for each variable in step S5, this invention combines these features with a multimodal language representation of the time series. Unlike simply appending them to the end of the feature sequence, this invention uses... The last word in the multimodal language representation is replaced, and this replacement is driven by the padding strategy used.

[0114] Step S6.2: Following the approach of existing models, this invention applies a flattened layer followed by a linear header to generate the prediction results. ;

[0115] Step S6.3: For all prediction tasks, the mean squared error (MSE) loss is used for evaluation. However, for the open-source M4 dataset, this invention follows previous research and uses the symmetric mean absolute percentage error loss: Conduct evaluation;

[0116] Step S6.4: Therefore, the present invention will L MSE and L SMAPE Both are referred to as prediction loss. L gen Therefore, the overall training loss combines the prediction loss and the contrastive loss term: , where λ1 and λ2 are weighting coefficients.

[0117] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the time series prediction method based on multimodal contrastive learning technology.

[0118] A computer-readable storage medium storing a computer program and instructions thereon, characterized in that the computer program and instructions, when executed by a processor, implement the steps of the time series prediction method based on multimodal contrastive learning technology.

[0119] A computer program product includes a computer program and instructions, characterized in that, when the computer program and instructions are executed by a processor, they implement the steps of the time series prediction method based on multimodal contrastive learning technology.

Claims

1. A time series prediction method based on multimodal contrastive learning technology, characterized in that, Includes the following steps: Step S1: Map the original multivariate time series data into structured visual modalities and text modalities respectively; Step S2: Convert the visual modal mapped from the original multivariate time series data into a visual representation; Step S3: Convert the text modality mapped from the original multivariate time series data into a language representation; Step S4: Design a contrastive learning mechanism to align visual and linguistic representations in a shared embedding space. By maximizing the consistency of positive sample pairs and minimizing the similarity of negative sample pairs, learn temporal features and calculate their contrastive loss. Step S5: Dynamically evaluate the importance of each variable to the prediction task using the aligned multimodal representation, and automatically select the most discriminative input variables for subsequent modeling; Step S6: Generate the predicted loss, combine the predicted loss and the contrastive loss to obtain the overall training loss and perform method prediction; The visual modality includes time-series images and heatmaps; the textual modality includes symbolic sequences and semantic descriptions. The specific steps of step S2 are as follows: Step S2.1: Given historical multivariate time series observations of length T ,in , N Representing different recorded variables, where T is the known historical time step, the goal is to predict the future of the time series. S time step ; Step S2.2: Fix the window size L window The process involves normalizing each variable in the original multivariate time series data and plotting each time series data variable with a specific color to help distinguish the variable type and align it with the linguistic representation of the time series in the multimodal feature space. This process can be expressed as the following formula: ； Step S2.3: Convert the time series into a series of images Image sequences of time series X Img Input to a pre-trained and frozen multimodal visual encoder E v In this process, a series of feature maps are generated. f 1 ， f 2 , . . . , f N }, where each feature map f i Depend on generate; Step S2.4: Since the multimodal visual backbone network remains frozen during training, an additional mapping layer is introduced. Proj img To align the visual features of the time series with the multimodal feature space, the above visual representation is denoted as... ; Step S2.5: For multivariate time series data of length T Define its visual representation as .

2. The time series prediction method based on multimodal contrastive learning technology according to claim 1, characterized in that, The specific steps of step S3 are as follows: Step S3.1: Introduce a block-segmentation strategy to divide the time series into subsequence-level blocks, and use these blocks as input blocks for the Transformer. The specific definition is shown in the following formula: in, PL This indicates the patch length, while Indicates the number of fragments. S Indicates stride length; Step S3.2: A randomly initialized, learnable linear layer is introduced as the embedding layer for the time series data, also known as the Tokenizer. ； Step S3.3: Add a learnable embedding before the word segmentation sequence of each variable. ; This is called 'category segmentation', and then a pre-trained multimodal language model is used. E text Learning language representations by embedding time series data into a multimodal language representation space: Where [., .] indicates the concatenation of time series segmentation and [class] segmentation. e pos To represent learnable positional embeddings, this invention uses the same [class] tokenization for each given variable. v patch ; Step S3.4: Obtain the linguistic representation of the time series as shown in the following formula: Will The first segment in the sequence is considered a [category] segment, denoted as To align with visual representation, the language classification representation is defined. .

3. The time series prediction method based on multimodal contrastive learning technology according to claim 2, characterized in that, The specific steps of step S4 are as follows: Step S4.1: Steps S2 and S3 can process multi-element time series of graphs of length T. X 1:T To obtain visual representation and language classification representation ,in , N Indicates the number of variables. D It is the feature dimension; Step S4.2: Use a batch of data to calculate the loss and obtain the visual representation. and language classification representation ,in B Indicates batch size; Step S4.3: Incorporate the multimodal contrast alignment of time series data into the standard contrast framework, i.e., based on the InfoNCE loss function, the specific definition of which is shown below: Where τ is a learnable temperature parameter, φ(·, ·) represents the cosine similarity, and each multimodal pair represents a sequence of modes of length τ. T time series N One variable, positive sample pairs in and The positive samples are formed from the same sample, while the negative samples are formed from different samples in the batch. Step S4.4: Calculate the contrast loss bidirectionally as follows: 。 4. The time series prediction method based on multimodal contrastive learning technology according to claim 3, characterized in that, The specific steps of step S5 are as follows: Step S5.1: For a length of T Multivariate time series X 1:T Each variable v i ∈R 1×T Each word is treated as a separate word and is not processed into blocks. Therefore... X 1:T It can be represented as X ∈R N×T ,in N Indicates the number of variables. T Indicating the time step, after word segmentation, the time series is converted into... ,in D It is a hidden dimension; Step S5.2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require Input into a pre-trained multimodal language encoder Generate variant hierarchical time series representations The process is defined as follows: Among them, the variant-level time series is represented as H and Standardization needs to be performed using layer normalization. Step S5.3: Select the most relevant variable features by employing a cross-attention layer in the Transformer decoder, where As a query H can Simultaneously serving as both key and value, the above process can be formally defined as: W q , W k and W v These are the weight matrices for query, key, and value projection in the cross-attention mechanism. d k This represents the dimensionality of each head in multi-head attention; Step S5.4: Finally, An improved Transformer decoder is formed by processing a LayerNorm and a two-layer feedforward neural network FFN, together with the cross-attention layer of the Transformer decoder described in step S5.

3.

5. A time series prediction method based on multimodal contrastive learning technology according to claim 4, characterized in that, The specific steps of step S6 are as follows: Step S6.1: After obtaining Next, for each variable in step S5, these features are combined with the multimodal language representation of the time series, rather than simply concatenating them to the end of the feature sequence. The last word in the multimodal language representation is replaced, and this replacement is driven by the padding strategy used. Step S6.2: Apply a flattened layer, then connect it to a linear header to generate the prediction results. ; Step S6.3: For all prediction tasks, the mean squared error (MSE) loss is used for evaluation. However, for the open-source M4 dataset, the symmetric mean absolute percentage error loss is used. Conduct evaluation; Step S6.4: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] L MSE and L SMAPE Both are referred to as prediction loss. L gen Therefore, the overall training loss combines the prediction loss and the contrastive loss term: , where λ1 and λ2 are weighting coefficients.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-5.

7. A computer-readable storage medium storing computer programs and instructions thereon, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-5.

8. A computer program product, comprising a computer program and instructions, characterized in that, When the computer program or instructions are executed by the processor, they implement the steps of the method described in any one of claims 1-5.

Images