Automatic generation of text time series descriptions

Abstract

Embodiments of the present disclosure relate to automatic generation of text time series descriptions. A method (100) for automatically generating a time series text description for monitoring or predicting a process variable, the method comprising the steps of: training (110) a deep learning model on a cross-modal automatic encoding module (10) having an architecture consisting of a time series encoder (11), a text decoder (12) and a time series decoder (13); acquiring (120) time series data by invoking a display of readings of the temporal process variable; encoding (130) the time series data on the basis of a deep learning model by means of a time series encoder (11) and generating an embedding as an input to a text decoder (12); transmitting (140) the generated embedding from the time series encoder (11) to the text decoder (12); and generating (150), by means of the text decoder (12), a time series text description based on the embedding of the encoded time series data from the time series encoder (11) based on a deep learning model.

Description

Technical Field

[0001] This disclosure relates to industrial process automation. Specifically, this disclosure relates to a method, computer program, and data processing system for automatically generating time-series text descriptions to monitor or predict process variables. Background Technology

[0002] Recently, the automatic generation of text descriptions for time series data has become a rapidly developing field combining Natural Language Processing (NLP) and time series analysis. With the increasing volume of continuous data from sources such as IoT and sensing systems, the demand for tools capable of transforming complex digital data into human-readable insights is also growing. Traditional data analysis often relies on charts and statistical measures, which require expert interpretation. However, automated text descriptions can democratize data understanding, allowing even non-experts to effectively extract insights from time series data.

[0003] In particular, factory operators need to monitor numerous processes to ensure safe and efficient production. In addition to the latest trends of given process variables, modern distributed computing (DCS) systems also need to be able to display predictions of signals of interest, as well as sensor readings that interpret predictions of faults or anomalies from machine learning (ML) models. However, displaying this additional information can be problematic due to existing user interface (UI) limitations and screen space constraints.

[0004] Typically, to gain a deeper understanding of a plant's status, sensor readings can be invoked to assess certain time-varying variables of interest. Such displays can also be triggered by the system itself, for example, if an alarm threshold is exceeded. When a DCS is combined with machine learning (ML) technology, it can display predictions for a given process variable, or predict alarms or faults, and show abnormal behavior signals as interpretations of the predicted events. However, screen space may be limited. Furthermore, the user interface needs to be rethought to accommodate these emerging characteristics. Field operators may only have a very small screen, or even none at all.

[0005] Despite the recent surge in developments of large language models (LLMs), advancements in natural language generation (NLG) have not yet extended to the problem of time series description due to the lack of large training datasets. In particular, it is difficult to train a model so that the text it produces is not only human-like but also accurate relative to the time series it describes. Summary of the Invention

[0006] One object of the present invention is to provide a more flexible and convenient alternative or supplementary system for displaying signal diagrams in an efficient and reliable automatic generation system.

[0007] This objective is achieved through the subject matter of the independent claims. The dependent claims, the following description, and the accompanying drawings illustrate embodiments of the invention.

[0008] According to a first aspect of this disclosure, a method is provided for automatically generating time-series textual descriptions for monitoring or predicting process variables. The method includes the following steps: - Train a deep learning model on a cross-modal autoencoder module with an architecture consisting of a time series encoder, a text decoder, and a time series decoder; - Obtain time series data by calling the readings of time process variables; - Time series data is encoded using a time series encoder and based on a deep learning model, and embeddings are generated as input for a text decoder; - The embedding of the generated encoded time series data transmitted from the time series encoder to the text decoder; - By leveraging a text decoder and based on a deep learning model, time series text descriptions are generated based on embeddings of encoded time series data from a time series encoder.

[0009] Time series data can be understood as a series of continuous data points collected or recorded at specific time intervals. Time series data can capture the temporal dependence and patterns of variables over time, and is commonly used for forecasting, trend analysis, and anomaly detection in fields such as plant operations, sensing technologies, and / or process monitoring.

[0010] Time series text descriptions can be understood as narratives summarizing key patterns, trends, and anomalies within series data. They can translate complex numerical insights into natural language, for example, helping plant operators understand changes, peaks, or recurring behaviors over time, and are valuable for data interpretation in the field of plant operations.

[0011] Cross-modal autocoding modules, or cross-modal auto-encoding / decoding methods, can be understood as a model training technique that learns to reconstruct data from one modality via another modality in a cross-modal setting. This means that an explicit representation of an object or process in one modality (e.g., a sequence of images constituting a video, or a time series representing a temperature measurement) is encoded into a latent representation and then decoded into another modality (e.g., an audio track of a video changing over time or a textual description of a temperature measurement). The latent representation of this second modality is then decoded back into the original modality. This approach enables models to capture relationships between different data modalities, facilitating tasks such as image-to-text generation or audio-to-image synthesis. Cross-modal autocoding is particularly useful in applications requiring multimodal understanding, such as converting visual data into textual descriptions or generating audio based on text. Therefore, cross-modal autocoding can be performed as a novel training technique for conditional NLG (Natural Language Generation) on small datasets, allowing deep learning models to be trained during pre-training to generate time-series textual descriptions and reconstruct time series based on the generated text.

[0012] Using the modeling system trained with the above method utilizing a cross-modal autoencoder module with a time-series encoder, the text decoder and time-series decoder can achieve flexible multimodal user experiences by automatically generating time-series text descriptions that can be delivered as concise text alert messages or as audio messages using existing text-to-speech technologies.

[0013] A time series encoder can be designed to produce one or more embeddings for a text decoder by transforming sequential numerical data into a fixed-dimensional representation or embedding. In other words, it can capture temporal patterns and dependencies. A time series encoder can process sequential time series data and compress it into a context vector or embedding. This embedding represents the temporal structure and key features of the input sequence. For example, a time series encoder can be, but is not limited to, an LSTM (Long Short-Term Memory), a GRU (Gated Recurrent Unit), or a transformer.

[0014] The embeddings can then be passed to a text decoder in the form of a sequence-to-sequence model, such as a transformer-based or RNN-based decoder. The text decoder can then interpret the encoded information and generate corresponding text sequences one token at a time based on patterns learned or trained from the time series data. Using the context provided by the embeddings, the text decoder can output text consistent with the temporal insights of the original time series data.

[0015] According to one embodiment, the training steps of a deep learning model may include sub-steps of training or activating a time-series decoder and enabling a cross-modal autoencoder module.

[0016] According to another embodiment, the steps of training the time-series decoder and enabling the cross-modal autoencoder module may include the following sub-steps: - Transferring time-series text descriptions from the text decoder to the time-series decoder; - Reconstruct time series data by using a time series decoder and generating a second time series data based on the generated text description using a deep learning model, as the reconstructed time series data;

[0017] Therefore, the steps of training the deep learning model are performed using a cross-modal autoencoder module based on the reconstructed time series data as the second time series data.

[0018] This method can involve reconstructing the input time series based on the embeddings of the descriptions generated by the text encoder during the training phase, as a form of pre-training, and optionally discarding the time series decoder. This technique can be designed to guide the model to better learn the correlation between the time series and the descriptions using small datasets.

[0019] The text embedding used as input to the time series decoder can be a concatenation of the output of the penultimate layer of the text decoder, or it can be a different type of hidden representation extracted from the text decoder depending on its architecture.

[0020] By reconstructing time series based on their textual descriptions generated by a text decoder, the transfer learning method enhanced by cross-modal autoencoding disclosed herein advantageously allows for training sufficiently accurate text generation models by relying on small training datasets, without the expensive data annotations and computational costs required for large models, and only requires small language models (LMs), thus saving resources.

[0021] According to another embodiment, the loss used to reconstruct the time series data can be scaled by a coefficient α used for balancing the training, such that the combined loss L can be calculated using the following function: . Where L txt It is the text generation loss, L tx It is the time series reconstruction loss.

[0022] According to another embodiment, the loss function used to reconstruct time series data can be regression loss.

[0023] According to another embodiment, the steps of training the time series decoder and enabling the cross-modal autoencoder module may include the following sub-steps: initializing the time series encoder and / or text decoder and / or time series decoder by means of a pre-trained model.

[0024] According to another embodiment, the step of initializing the time series encoder and / or text decoder and / or time series decoder may include the following sub-steps: initializing the time series encoder and text decoder and time series decoder using pre-trained model weights.

[0025] Initializing time-series encoders, text decoders, and time-series decoders with pre-trained model weights can offer advantages such as enhanced performance, reduced training time, and improved robustness. In particular, pre-trained weights can capture generalized patterns and features from relatively large datasets, enabling the model to learn more effectively and improving the accuracy of generating relevant textual descriptions from time-series data. Deep learning models initialized with pre-trained weights can converge faster, reducing the amount of time-series data and the time required to reach optimal performance. This is especially useful when data resources are limited.

[0026] Furthermore, pre-trained models can draw knowledge from a wider range of data sources, enabling them to adapt to specific domains with minimal fine-tuning. Additionally, pre-training enhances a model's resistance to overfitting and improves its generalization ability, ensuring accurate and coherent textual descriptions even for complex or subtle patterns in time-series data.

[0027] According to another embodiment, the step of training a deep learning model may further include the following sub-steps: pre-training a deep learning model in a self-supervised manner on an unlabeled time series dataset that includes unlabeled time series data and / or an unlabeled second time series dataset that includes unlabeled second time series data and / or on a corpus of time series descriptions.

[0028] Typically, time series descriptions have two main requirements: fidelity to the data (accurately describing the relevant properties of the time series window) and readability (grammatically correct and stylistically appropriate). Initializing a model's text decoder can aim to satisfy the latter, i.e., readability, while the first requirement of fidelity to the data can still be challenging due to the lack of large-scale, well-curated datasets of parallel time series samples and their descriptions. Pre-training a time series encoder on unlabeled time series in a self-supervised manner can help improve the model's representation learning capabilities. However, to reinforce the learning of the correlation between patterns in the time series and the text description, the method disclosed in this disclosure uses a cross-modal autoencoding module, which results in a general trend where time series samples are reconstructed based on their text descriptions.

[0029] According to another embodiment, the step of training a deep learning model may further include a sub-step of discarding the time-series decoder.

[0030] According to another embodiment, the step of training a deep learning model may further include the following sub-steps: expanding the time series dataset used to train the deep learning model, which may further include the following sub-steps: increasing the annotated time series dataset by adding noise to the annotated time series dataset; and / or increasing the text description by using an open-source large language model LLM to interpret the corresponding text descriptions of the annotated time series dataset.

[0031] For example, open-source LLMs can be, but are not limited to, GPT-based models such as OpenAI's GPT-Neo, GPT-J, GPT-2, and GPT-3, BERT and its variants such as RoBERTa and DistilBERT, Mitral:7b, Gemma, Llama-2, or T5 as text-to-text transfer transducers. These models are typically built on a transducer architecture and can excel at tasks such as paraphrasing, summarizing, and text generation. This advantageously allows for training sufficiently accurate text generation models by relying on small training datasets as the initial training foundation.

[0032] According to another embodiment, the step of reconstructing time series data by means of a cross-modal autoencoder module is performed as a task of full reconstruction, partial masked time series reconstruction, or scrambled time series reordering.

[0033] According to another embodiment, the step of enabling the cross-modal autoencoding module can be used in parallel, or it can be used before the final training or fine-tuning of the main task of generating time series text descriptions.

[0034] According to another embodiment, the cross-modal autocoding module can be executed as a contrastive learning method.

[0035] According to another embodiment, the backbone of the time series encoder or text decoder or time series decoder can be a transformer or a long short-term memory, such that the hidden dimensions of each of the time series encoder, text decoder and time series decoder can be compatible with each other or can be adjusted with additional linear layers.

[0036] According to another embodiment, the time series encoder and the time series decoder can share the same structure and weights until the encoder output layer of the time series encoder generates an embedding for the text decoder and the decoder output layer of the time series decoder generates reconstructed second time series data.

[0037] The output layer can be understood as the last layer in a neural network, responsible for generating predictions from the model based on features learned from previous layers. It transforms the network's output into the desired format, such as classification labels, text tags, or regression values.

[0038] According to another embodiment, the method may further include the step of outputting a time-series text description generated by a text decoder by displaying a text warning message or by outputting an audio message using a text-to-speech (TTS) method.

[0039] For example, TTS can be, but is not limited to, cascaded TTS, parametric TTS, deep learning TTS (such as Tacotron), WaveNet, or end-to-end TTS (such as FastSpeech).

[0040] In view of the above, the method for automatically generating time series descriptions disclosed herein may include: constructing a deep learning model including a time series encoder, a text decoder, and a time series decoder; and optionally initializing each of these components with weights from a pre-trained model, and optionally pre-training each of these components using a self-supervised method. The deep learning model may also be pre-trained on a cross-modal autoencoding task, which involves reconstructing an input time series based on embeddings of the input time series description generated by the text encoder, to enhance the model's ability to learn the correlation between patterns in the time series and the text, such as an upward trend in a signal and words like 'rise', 'increase', or 'ascend' in the description. Finally, the time series decoder component may be discarded, and the remainder of the model may be used as intended for the main task of text generation.

[0041] For a long time, the automatic generation of time series descriptions has been on the periphery of general NLG research and involves highly complex rule-based systems. Even after the advent of fairly powerful LMs, surprisingly few studies have attempted to use them in the time series domain. A recent example first learns a set of predefined patterns that identify the input time series, and then trains an LSTM-based network to generate descriptions based on the predicted patterns. To accomplish this task, a set of 5700 time series descriptions has been crowdsourced. However, given the surge in popularity of LLMs, many studies have attempted to apply them to time series. Numerous approaches exist for using LLMs to handle time series-related tasks, namely: cues (e.g., using time series data directly as raw text to cue LLMs), quantization (e.g., discretizing time series into intervals), alignment (e.g., learning time series embeddings aligned with language), vision as a bridge (e.g., plotting time series and using visual language models), and tool integration (e.g., using LLMs to output specialized tools). Most studies involve tasks such as prediction or classification and achieve performance generally comparable to existing models, which are often more compact and efficient. Few studies address time series descriptions, and may not provide informative evaluations. For example, some studies aim to create a 'baseline model' for time series data, excluding descriptive tasks. For instance, a transformer model can be pre-trained on many datasets (finance, healthcare, transportation, etc.), but it cannot be trained or tested on industrial data.

[0042] According to a second aspect of this disclosure, one or more computer program products including instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to implement the method of the first aspect of this disclosure.

[0043] A computer program product can be one or more computer programs, which means a computer program consisting of or including program code to be executed by a data processing device (in particular a computer).

[0044] Alternatively, the computer program products may be one or more products, such as data storage devices, particularly computer-readable data storage media, on which the computer programs may be temporarily or permanently stored.

[0045] According to a third aspect of this disclosure, a data processing system is provided, which is configured to perform the method according to a first aspect of this disclosure.

[0046] It should be noted that the above aspects, examples, and features can be combined with each other, regardless of the aspects involved.

[0047] These and other aspects will become clear and illustrated by referring to the embodiments in the accompanying drawings described below. Attached Figure Description

[0048] Examples of this disclosure will now be described with reference to the accompanying drawings.

[0049] Figure 1 A flowchart illustrating a method for automatically generating time series text descriptions according to the present disclosure is shown schematically;

[0050] Figure 2 An example of a cross-modal autocoding module according to this disclosure is illustrated schematically;

[0051] Figure 3 A flowchart illustrating the steps of a method for training a deep learning model on a cross-modal autoencoder module according to the present disclosure is shown schematically.

[0052] Figure 4a Examples of BERT embeddings of common verbs describing time series trends are shown;

[0053] Figure 4b An example of an embedding generated using a deep learning model trained according to the method of this disclosure is shown;

[0054] Figure 5 Examples of model scores compared to deep learning models trained using a cross-modal autoencoder module according to the method of this disclosure are shown; and

[0055] Figure 6 a) to Figure 6 e) Examples of time series with corresponding descriptions are shown, including those written by human annotators, generated by off-the-shelf methods, and generated by deep learning models trained using cross-modal autoencoder modules according to the methods of this disclosure.

[0056] These figures are merely illustrative and are intended only to illustrate examples of this disclosure. In principle, identical or equivalent elements have the same reference numerals. Detailed Implementation

[0057] Figure 1 A flowchart is schematically shown illustrating a method 100 for automatically generating time series text descriptions to monitor or predict process variables. Method 100 includes multiple steps.

[0058] In step 110 of method 100, the deep learning model can be trained or pre-trained on or using the cross-modal autoencoder module 10, such as Figure 2 As shown, the module has an architecture consisting of a time series encoder 11, a text decoder 12, and a time series decoder 13.

[0059] In step 120 of method 100, time-series data can be obtained by invoking readings of time process variables, for example, from sensors in a plant monitoring system, which can be selected or determined by the plant operator, to gain insights into the plant's status. For example, a display can be triggered by the plant monitoring system itself in the event of an alarm threshold exceeding a certain threshold; this can be used to predict alarms or malfunctions, or to display abnormal behavior signals as an interpretation of the predicted event.

[0060] In step 130 of method 100, the time-series data can be encoded based on a pre-trained deep learning model using a time-series encoder 11, and an embedding can be generated as input to a text decoder 12. Alternatively or additionally, the text embedding used as input to the time-series decoder can be a concatenation of the outputs of the penultimate layer of the text decoder, or it can be a different type of hidden representation extracted from the text decoder according to its architecture.

[0061] Subsequently, in step 140 of method 100, the embedding generated from time series encoder 11 can be transmitted, passed, or fed to text decoder, followed by step 150 of method 100, in which time series text description can be generated by means of text decoder 12 based on a pre-trained deep learning model and based on the embedding of encoded time series data generated from time series decoder 11.

[0062] Optionally, method 100 may further include the step of outputting a time-series text description of the generated text decoder 12 by displaying a text warning message or by outputting an audio message using a text-to-speech method. The text warning may be triggered by a variable of interest in the plant monitoring system that exceeds a preset alarm threshold indicating a system malfunction and / or abnormal behavior signal, as an interpretation of a predicted harmful event.

[0063] For example, TTS can be, but is not limited to, cascaded TTS, parametric TTS, deep learning TTS (such as Tacotron), WaveNet, or end-to-end TTS (such as FastSpeech).

[0064] Figure 3The method step 110 for training a deep learning model may include sub-step 160, wherein a time series decoder 13 and a cross-modal autoencoder module 10 may be used to train the deep learning model. Furthermore, or in parallel, the training step 110 may include sub-step 111, wherein the deep learning model may be pre-trained in a self-supervised manner or method on an unlabeled time series dataset including unlabeled time series data and an unlabeled second time series dataset including unlabeled second time series data, and / or on a corpus of time series descriptions. Pre-training the time series encoder in a self-supervised manner on unlabeled time series can help improve the model's representation learning ability.

[0065] The training sub-step 160 of the training step 100 for the cross-modal autoencoder module 10 may further include multiple sub-steps, particularly sub-step 161 as an optional transfer learning method, wherein the time series encoder 11 and / or text decoder 12 and / or time series decoder 13 can be initialized for each cross-modal autoencoder module 10 using a pre-trained model. Optionally, the initialization sub-step 161 may further include sub-step 1611, wherein the time series encoder 11, text decoder 12, and time series decoder 13 can be initialized using pre-trained model weights. Initializing the time series encoder, text decoder, and time series decoder using pre-trained model weights can provide advantages such as enhanced performance, reduced training time, and improved robustness. In particular, pre-trained weights can capture generalized patterns and features from relatively large datasets, enabling the model to learn more effectively and improving the accuracy of generating relevant text descriptions from time series datasets. Deep learning models initialized with pre-trained weights can converge faster, reducing the amount of time series data and the time required to reach optimal performance. This is particularly useful when data resources are limited.

[0066] Furthermore, after initializing the time series encoder 11, text decoder 12, and time series decoder 13 for each cross-modal autoencoder module 10 in sub-step 161, a subsequent step 162 can be provided, in which the time series text description can be transferred from the text decoder 12 to the time series decoder 13.

[0067] In sub-step 163, the time series data can be reconstructed by generating second time series data based on the text description generated by the text decoder 12 using the time series decoder 13 and a deep learning model. Therefore, step 110 of training the deep learning model can be performed using the cross-modal autoencoding module 10 based on the reconstructed second time series data.

[0068] In summary, the method 100 for automatically generating text time series descriptions and the corresponding system of the cross-modal autoencoding module 10 having a time series encoder 11, a text decoder 12 and a time series decoder 13 in the architecture can provide training techniques for cross-modal autoencoding, thereby enabling deep learning models to generate time series text descriptions and reconstruct time series based on the generated or produced text, minimizing the loss of both tasks.

[0069] The backbone of time series encoder 11, text decoder 12, or time series decoder 13 can be a transformer or a Long Short-Term Memory (LSTM), allowing the hidden dimensions of each of the time series encoder 11, text decoder 12, and time series decoder 13 to be compatible with each other, or to be tuned using additional linear layers. Furthermore, each component (i.e., time series encoder 11, text decoder 12, and time series decoder 13) can be initialized using an existing pre-trained model. Additionally, time series encoder 11 and time series decoder 13 can share the same structure and weights until the encoder output layer of time series encoder 11 can generate embeddings for text decoder 12, and the decoder output layer of time series decoder 13 can generate reconstructed second time series data.

[0070] The training of the deep learning model can be enhanced through cross-modal autoencoding, thereby enabling the text decoder 12 to generate time-series text descriptions, and the time-series decoder 13 to reconstruct the time series based on the embeddings of the generated text descriptions, while minimizing the losses of both tasks, such as the text classification cross-entropy and the time-series mean squared error (MSE). For balanced training, the loss of time-series reconstruction can be scaled by the balanced training coefficient α, such that the combined loss L can be calculated using the following function: , Where L txt It can be text generation loss, L tx It can be the time series reconstruction loss.

[0071] For inference purposes, in sub-step 164, the time series decoder 13 may optionally be discarded.

[0072] The cross-modal autocoding method and system of Method 10 can be implemented in the following variations.

[0073] For example, the cross-modal autoencoder module 10 can be used in parallel or before the final training or fine-tuning of the main task of generating 150 time-series text descriptions.

[0074] Alternatively or additionally, the text embedding used as input to the time series decoder 13 may be a cascade of the output of the penultimate layer of the text decoder 12, or a different type of hidden representation extracted from the text decoder 12 depending on its architecture.

[0075] For example, the loss function used to reconstruct the 163 time series data could be a regression loss. For instance, the loss function for time series reconstruction could be any suitable regression loss, such as MSE (mean squared error), MAE (mean absolute error), and RMSE (root mean squared error).

[0076] Cross-modal autocoding can be used for a variety of tasks. In other words, the reconstruction 163 of the time series data of the cross-modal autocoding module 10 can be performed as a task of full reconstruction, partial masking time series reconstruction, or scrambled time series reordering.

[0077] Furthermore, the cross-modal autoencoding method of Method 100 can be implemented as a contrastive learning method.

[0078] Figure 3 Furthermore, in addition to or in parallel with optional transfer learning methods, an augmentation method may be optionally provided for training the deep learning model 110, wherein the text decoder 12 may be initialized using weights from an open-source (L)LM, or pre-trained on a text corpus including time-series descriptions or similar text, and / or in addition to or in parallel with optional self-supervised pre-training methods, wherein the time-series encoder 11 and the time-series decoder 13 may be pre-trained in a self-supervised manner on a time-series dataset without any labels or annotations.

[0079] In substep 112 of method 110, the time series can be augmented for training the deep learning model 110. Substep 112 may further include: substep 1121, wherein the annotated time series dataset is augmented by adding noise to the annotated time series dataset; and / or substep 1122, wherein the description is augmented using an open-source LLM by interpreting the corresponding textual description of the annotated time series dataset.

[0080] For example, open-source LLMs can be, but are not limited to, GPT-based models such as OpenAI's GPT-Neo, GPT-J, GPT-2, and GPT-3, BERT and its variants such as RoBERTa and DistilBERT, Mitral:7b, Gemma, Llama-2, or T5 as text-to-text transfer transducers. These models are typically built on a transducer architecture and can excel at tasks such as paraphrasing, summarizing, and text generation. This advantageously allows for training sufficiently accurate text generation models by relying on small training datasets as the initial training foundation.

[0081] In addition to cross-modal autoencoding methods, the method 100 of this disclosure can advantageously allow training of sufficiently accurate text generation models that faithfully describe time series without expensive data annotation—through a combination of techniques that rely on small datasets and apply to enhance the model’s representation learning capabilities, including novel cross-modal autoencoding.

[0082] For a long time, the automatic generation of time series descriptions has been on the periphery of general NLG research and involves highly complex rule-based systems. Even after the advent of fairly powerful LMs, surprisingly few studies have attempted to use them in the time series domain. One recent example first learns a set of predefined patterns that identify the input time series, and then trains an LSTM-based network to generate descriptions based on the predicted patterns. To accomplish this task, a set of 5700 time series descriptions has been crowdsourced. However, given the surge in popularity of LLMs, many studies have attempted to apply them to time series. Numerous approaches exist for using LLMs to handle time series-related tasks, namely: cues (e.g., using time series data directly as raw text to cue LLMs), quantization (e.g., discretizing time series into intervals), alignment (e.g., learning time series embeddings aligned with language), vision as a bridge (e.g., plotting time series and using visual language models), and tool integration (e.g., using LLMs to output specialized tools). Most studies involve tasks such as prediction or classification and achieve performance generally comparable to existing models, which are often more compact and efficient. Few studies address time series descriptions, and therefore may not provide informative evaluations. For example, some research aims to create a 'baseline model' for time series data, such as excluding descriptive tasks. For instance, a transformer model can be pre-trained on many datasets (finance, healthcare, transportation, etc.), but it cannot be trained or tested on industrial data.

[0083] Figure 4a and Figure 4b Examples of plotted embeddings describing common verbs of time series trends are shown, using BERT embeddings and embeddings generated by a deep learning model trained using the method 100 of this disclosure. Two t-SNE (t-distributed random neighborhood embedding) components are plotted on the x and y axes, respectively, as a dimensionality reduction technique to visualize complex data distributions by embedding them into a low-dimensional 2D space while preserving the structure of data clusters and understanding feature embeddings.

[0084] A series of preliminary experiments have been conducted, comparing the results of Method 100 with several common existing methods: LLM hints using the raw time series directly, rounded values, and SAX-transformed values ​​(interval-based or 'quantized'). For evaluation purposes, the experimenters avoided using classic metrics such as BLEU, ROUGE, BERT scores, etc., because they do not reflect the truth of the description (e.g., scores for 'value is rising' and 'value is falling' are either equal or negligible). The reason is that these scores are either based on the n-gram overlap between the underlying truth and the prediction, which is not a useful indicator of accuracy in this case, or on the similarity of word embeddings in the latent space, where 'increase' and 'decrease' and their synonyms can be very close due to the way word embeddings are learned in (L)LM, making such metrics ineffective in the current use case.

[0085] Figure 4a The diagram shows BERT (Bidirectional Encoder Representation from Transformer) embeddings of several common verbs describing time series trends.

[0086] For example, in Figure 4a In this context, while words like 'stable' and 'stationary' are close in the embedding space and relatively far from their corresponding words, antonym pairs such as 'descent' and 'climb', 'increase' and 'descrese', and 'downward' and 'rise' are very close, rather than grouping words synonymous with 'increase' together and separating words synonymous with 'decrease'.

[0087] Figure 4b A visualization of embeddings generated by a deep learning model trained using the method 100 of this disclosure is shown. These embeddings describe several common verbs representing time-series trends extracted from the deep learning model after training. Repetition is due to the fact that the generative model may produce different embeddings for the same words depending on the context compared to discriminative models such as BERT. It can be observed that 'increase', 'decrease', and their changes are clearly separated.

[0088] To evaluate the deep learning model of Method 100, Figure 5 The model comparison using F1 scores based on manually labeled classes is shown, describing the main time series properties 'increase', 'decrease', 'oscillation / noise', 'stable', 'increase first, then decrease', and 'decrease first, then increase'. Figure 5 The curves in the figure show the scores obtained by different deep learning models that incorporate the methods of this disclosure. A higher F1 score indicates a higher overall score.

[0089] Baseline methods compared to Method 100 include off-the-shelf LLMs (locally deployed GPT-2, Mistral:7b, Gemma, and Llama-2), which use either direct hints (the original time series as strings) or discrete time series hints (rounded to the nearest integer or approximated using symbolic aggregation or converted to a discrete symbolic representation using SAX). These models are referred to in the figure as...<LLM_name> _<time_series_dtype> .

[0090] All deep learning models in Method 100 should be referenced using the following format: pg_ <attention> _ <pretraining>, where 'pg' represents 'PatchTST+GPT2' (the model used for time series encoder 11, time series decoder 13, and text decoder 12), 'attention' represents cross-attention and self-attention, and 'pretraining' represents the pretraining applied: no pretraining (na), prediction (f), auto-encoding (a), and cross-modal auto-encoding (ar).

[0091] like Figure 5 As shown, the deep learning model trained using method 100 with cross-modal autoencoding module 10 as a pre-training strategy performs best on the unseen test set. This is followed by another deep learning model of method 100 of this disclosure, but with a different pre-training strategy. The results obtained by off-the-shelf LLM are significantly less accurate. It can be assumed that the cross-modal autoencoding strategy of this disclosure is key to learning word embeddings, where words describing time series trends ('increase' and 'decrease', 'oscillation', etc.) correspond to the corresponding patterns in the time series, allowing the deep learning model to learn to generate time series descriptions faithful to the input.

[0092] It is also worth noting that deep learning models using cross-modal autoencoding can be trained on word prediction rather than classification, thus eliminating the need for corresponding labels. However, the high classification scores are due to the proposed data-efficient pre-training technique.

[0093] In addition, the following table details Figure 5 The model size, training and inference time, and F1 score were compared. It can be observed that off-the-shelf LLMs not only have significantly lower accuracy in time series descriptions, but also require an order of magnitude more GPU space (e.g., 30 to 40 times more parameters) and significantly longer inference time. Furthermore, using LLMs for time series description requires generating cue text, which is time-consuming and not robust enough.

[0094] Therefore, the results of deep learning models using the methods of this disclosure demonstrate that the proposed model architecture and pre-training strategy can generate descriptions that faithfully capture the properties of the input time series, outperforming existing LLM hint methods in terms of accuracy, model size, and inference time.

[0095] Figure 6 a) to Figure 6 e) shows an example of a time series. Below is the corresponding description, annotated by human commentator M. HA Compiled by using M according to this disclosure CMA Method 100 generates a deep learning model trained with a cross-modal autoencoder module 10, and is derived from an off-the-shelf method M. OTS generate.

[0096] The time series in the graph represents sensor readings for temperature, level, etc. Accordingly, the y-axis represents the range of values, while the x-axis represents the number of minutes within a given time window. To make the solution generalizable, all specific process variables have been replaced with the placeholder word 'variable,' which can be replaced with the name of any specific process variable or sensor if necessary.

[0097] Figure 6 a): M HA The variable initially begins to decrease, reaches its lowest point, then gradually increases, and later begins to decrease again. M CMA The variable gradually decreases and eventually returns to a decreasing state; M OTS The variable exhibits a generally irregular pattern, with some fluctuations between decreasing and increasing values.

[0098] Figure 6 b): M HA The variable showed slight decreases and increases, before finally declining rapidly. M CMA The variable rises, then falls slowly, then rises again gradually, but eventually falls sharply. M OTS The variable exhibits a fluctuating trend, with some values ​​increasing significantly while others decrease significantly.

[0099] Figure 6 c): M HA After a long period of slow growth, the variable drops sharply, then initially rises sharply again, and then rises slowly near the end. M CMA The variable first rises slowly, then falls sharply, then rises again, and finally begins to rise again. M OTS The variable generally shows an upward trend, with some fluctuations and occasional sharp declines.

[0100] Figure 6 d): M HA The variable rises sharply from zero during the slow increase phase and before finally stabilizing. M CMA The variable rises sharply from zero and then stabilizes at a high level. M OTS The variable generally shows an upward trend, with some fluctuations around it.

[0101] Figure 6 e): M HA After a slight decline and a subsequent drop, the variable exhibits a meaningful but stable decline, but once it reaches a low point, it begins to rebound. M CMA The variable behaves like a modified sine wave; M OTS The variable exhibits a roughly irregular pattern, fluctuating between approximately 15 and 18, with some values ​​decreasing slightly and others increasing slightly.

[0102] These examples demonstrate that descriptions generated by deep learning models trained using cross-modal autoencoding tend to faithfully depict the plotted time series, closely resembling human descriptions. In contrast, off-the-shelf LLMs produce very vague descriptions that are not specific to a given time series.

[0103] Although the invention has been shown and described in detail in the accompanying drawings and the foregoing description, such illustrations and descriptions should be considered illustrative or exemplary, not restrictive; the invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art and those practicing the claimed invention through a study of the drawings, this disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements, and the indefinite articles "a" or "an" do not exclude a plural. A single processor or controller or other unit can perform the functions of several items described in the claims. The fact that certain measures are referenced in mutually different dependent claims does not mean that a combination of these measures cannot be advantageous. Any reference signs in the claims should not be construed as limiting the scope. List of reference numerals < / pretraining> < / attention>

Claims

1. A method (100) for automatically generating time series text descriptions for monitoring or predicting process variables, comprising the following steps: A deep learning model is trained (110) on a cross-modal autoencoder module (10), which has an architecture consisting of a time series encoder (11), a text decoder (12), and a time series decoder (13); (120) time series data is obtained by calling the readings of the time process variables; The time series data is encoded (130) using the time series encoder (11) and based on the deep learning model, and an embedding is generated as input to the text decoder (12); The embedding of the encoded time series data generated by the transmission (140) from the time series encoder (11) to the text decoder (12); The time series text description is generated (150) by means of the text decoder (12) and based on the deep learning model, based on the embedding of the time series data encoded from the time series encoder (11).

2. The method (100) according to claim 1, wherein training (110) the deep learning model comprises the following sub-steps: The time series decoder (13) is trained (160) and the cross-modal autoencoder module (10) is enabled.

3. The method (100) according to claim 2. The training (160) of the time-series decoder (13) and the enabling of the cross-modal autoencoder module (10) include the following sub-steps: The time-series text description is transmitted (162) from the text decoder (12) to the time-series decoder (13); The time series data is reconstructed (163) by means of the time series decoder (13) and by generating second time series data based on the generated text description using the deep learning model; The training (110) of the deep learning model is performed using the cross-modal autoencoder module (10) based on the reconstructed second time series data.

4. The method (100) according to claim 3. The loss used for reconstructing the time series data (163) is scaled by a coefficient α that balances the training, such that the combined loss L is calculated by the following function. ， Where L txt It is the text generation loss, L tx It is the time series reconstruction loss.

5. The method (100) according to any one of the preceding claims. The loss function used for the reconstruction (163) of the time series data is a regression loss.

6. The method (100) according to any one of claims 2 to 5. The training (160) of the time-series decoder (13) and the enabling of the cross-modal autoencoder module (10) include the following sub-steps: The time series encoder (11) and / or the text decoder (12) and / or the time series decoder (13) are initialized (161) with the aid of a pre-trained model.

7. The method (100) according to any one of the preceding claims. The initialization (161) of the time series encoder (11) and / or the text decoder (12) and / or the time series decoder (13) includes the following sub-steps: The time series encoder (11), the text decoder (12), and the time series decoder (13) are initialized (1611) using pre-trained model weights.

8. The method (100) according to any one of the preceding claims, wherein training (110) the deep learning model further comprises the following sub-steps: The deep learning model described in (111) is pre-trained in a self-supervised manner on an unlabeled time series dataset and / or an unlabeled second time series dataset and / or on a corpus of time series descriptions.

9. The method (100) according to any one of the preceding claims. The training (110) of the deep learning model further includes the sub-step of discarding (165) the time series decoder.

10. The method (100) according to any one of the preceding claims, wherein training (110) of the deep learning model further comprises the following sub-steps: Expanding (112) the time series dataset used for training (110) the deep learning model includes the following sub-steps: The annotated time series dataset (1121) is augmented by adding noise to it; and / or The text description (1122) is augmented by using an open-source LLM to interpret the corresponding text description of the annotated time series dataset.

11. The method (100) according to any one of claims 3 to 10. The time series data of the cross-modal autoencoder module (10) described in the reconstruction (163) are performed as a task of full reconstruction, partial masking time series reconstruction or scrambled time series reordering.

12. The method (100) according to any one of the preceding claims. The cross-modal autoencoding module (10) is used in parallel or before the final training or fine-tuning of the main task of generating (150) the time series text description.

13. The method (100) according to any one of the preceding claims. The cross-modal autoencoder module (10) is executed as a contrastive learning method.

14. The method (100) according to any one of the preceding claims. The backbone of the time series encoder (11), the text decoder (12), or the time series decoder (13) is a transformer or long short-term memory, such that the hidden dimensions of each of the time series encoder (11), the text decoder (12), and the time series decoder (13) can be compatible with each other or can be adjusted using an additional linear layer.

15. The method (100) according to any one of the preceding claims. The time series encoder (11) and the time series decoder (13) share the same structure and weights until the encoder output layer of the time series encoder (11) generates the embedding for the text decoder (12) and the decoder output layer of the time series decoder (13) generates the reconstructed second time series data.

16. The method (100) according to any one of the preceding claims further includes the following step: The time-series text description generated by the text decoder (12) is output by displaying a text warning message or by outputting an audio message using a text-to-speech method.

17. A computer program comprising machine-readable instructions that, when executed by one or more data processing devices, cause the one or more data processing devices to perform the method (100) according to any one of claims 1 to 16.

18. A data processing system configured to perform the method (100) according to any one of claims 1 to 16.

Images