A system and method using a machine learning framework for system forensics

Abstract

This invention provides a method and system for performing tasks in the time-series domain using various modalities of time-series information. [Solution] The method involves a time series tokenizer receiving time series data containing numerical data for a set of channels, generating a digital image displaying a set of graphic representations of the time series data, generating time series embeddings using the time series data in a channel-independent manner to store local semantic information, a plot tokenizer generating image embeddings using the digital image to capture the global context of the time series data and inter-channel dependencies, a pre-trained large-scale language model (LLM) generating semantic embeddings based on at least the time series embeddings and image embeddings, and a task module performing a task and generating output data for the task using the semantic embeddings.

Description

TECHNICAL FIELD

[0001] Technical Field The present disclosure relates generally to system forensics via time series analysis, computer vision, and natural language processing, and more specifically to performing specific tasks via a machine learning framework that utilizes different modalities of time series information.

BACKGROUND ART

[0002] Background Following the success of pre-trained models in natural language processing (NLP) and computer vision (CV), there has been growing interest in adapting pre-trained large language models (LLMs) for time series analysis. However, unlike NLP and CV, time series analysis still relies heavily on specialized approaches for each specific task such as classification, anomaly detection, prediction, and few-shot learning.

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0003] While elaborate preprocessing of time series data and specially designed prompts can improve the zero-shot performance of LLMs on time series tasks, there exists a semantic gap between natural language and numerical time series data, which still poses a considerable challenge regarding the use of LLMs for time series tasks. To address this modality gap, recent approaches have focused on fine-tuning specific components of LLMs, or on training additional modules to preprocess the time series data itself before feeding it into the LLM. However, these approaches may have difficulty capturing the global perspective of time series data.

MEANS FOR SOLVING THE PROBLEMS

[0004] overview The following is an overview of specific embodiments described in detail below. The embodiments described are presented solely to provide the reader with a brief overview of these specific embodiments, and the descriptions of these embodiments are not intended to limit the scope of this disclosure. In fact, this disclosure may encompass a variety of other embodiments that are not expressly described below.

[0005] According to at least one embodiment, a computer-implemented method relates to performing a task in the time series domain. The method includes receiving time series data, including numerical data. The method includes generating a digital image that displays a set of graphic representations of the time series data. The method includes generating time series embeddings using the time series data via a tokenizer. The method includes generating image embeddings using pixels of the digital image via an image encoder. The method includes generating semantic embeddings based on at least the time series embeddings and image embeddings via a pre-trained large-scale language model (LLM). The method includes performing the task via at least one task head and generating output data for the task using the semantic embeddings.

[0006] According to at least one embodiment, the system comprises one or more processors and one or more non-temporary computer memories. One or more processors communicate data with one or more non-temporary computer memories. One or more non-temporary computer memories store computer-readable data. The computer-readable data includes instructions for one or more processors to perform the Method relating to performing a task in the time-series domain, when executed by one or more processors. The Method includes receiving time-series data, which includes numerical data. The Method includes generating a digital image that displays a set of graphic representations of the time-series data. The Method includes generating time-series embeddings using the time-series data via a tokenizer. The Method includes generating image embeddings using pixels of the digital image via an image encoder. The Method includes generating semantic embeddings based on at least the time-series embeddings and image embeddings via a pre-trained large-scale language model (LLM). The Method includes performing a task via at least one task head and generating output data for the task using the semantic embeddings.

[0007] The above and other features, aspects and advantages of the present invention are described in the following detailed description with reference to the accompanying drawings, and similar letters throughout the accompanying drawings represent similar or equivalent parts. Furthermore, the drawings are not necessarily to scale, because some features may be exaggerated or minimized to show details of certain components. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an overview of an example of a multimodal large language model for time series (MLLM4TS) system according to an exemplary embodiment of the present disclosure. [Figure 2]This figure shows an example of a time-series tokenizer according to an exemplary embodiment of the present disclosure. [Figure 3] This figure shows an example of a plot tokenizer according to an exemplary embodiment of the present disclosure. [Figure 4] This figure shows an example of a text tokenizer according to an exemplary embodiment of the present disclosure. [Figure 5] This figure shows a non-limiting example of a digital image that displays a set of graphic representations of multivariate time-series data in a 2x2 grid format, according to exemplary embodiments of the present disclosure. [Figure 6] This figure shows a non-limiting example of a digital image that displays a set of graphic representations of multivariate time-series data in the form of a vertical stack, according to exemplary embodiments of the present disclosure. [Figure 7] This figure shows a non-limiting example of a digital image that displays a set of graphic representations of multivariate time-series data in the form of a horizontal stack, according to exemplary embodiments of the present disclosure. [Figure 8] This figure shows a non-limiting example of a digital image displaying a set of graphic representations of multivariate time series data in a 2x2 grid format according to an exemplary embodiment of the present disclosure, in which each graphic representation is color-coded and plotted using a different color. [Figure 9] This figure shows an example of a system including an MLLM4TS system according to at least one exemplary embodiment of the present disclosure. [Figure 10] This figure shows an example of interaction between a computer-controlled machine and a control system according to at least one exemplary embodiment of the present disclosure. [Figure 11] Figure 10 shows a control system configured to control at least partially or fully autonomous mobile machinery according to at least one exemplary embodiment of the present disclosure. [Figure 12] Figure 10 shows a control system configured to control a manufacturing machine in a manufacturing system, such as part of a production line, according to at least one exemplary embodiment of the present disclosure. [Figure 13] Figure 10 is a schematic diagram showing a control system configured to control a monitoring system according to at least one exemplary embodiment of the present disclosure. [Figure 14] Figure 10 is a schematic diagram showing a control system configured to control a medical imaging system according to at least one exemplary embodiment of the present disclosure. [Modes for carrying out the invention]

[0009] Detailed explanation Many of the embodiments and their advantages described herein, illustrated and described as examples, can be understood from the foregoing description and will be apparent to you that various modifications can be made to the form, configuration and arrangement of the components without departing from the subject matter disclosed or sacrificing one or more of its advantages. In fact, the forms described in these embodiments are for illustrative purposes only. Various modifications and alternative forms are envisioned for these embodiments, and the appended claims are intended to encompass and include such modifications and are not limited to any particular form disclosed, but rather to cover all modifications, equivalents and alternative forms that fall within the spirit and scope of this disclosure.

[0010] Figure 1 shows an overview of a Multimodal Large-Scale Language Model for Time Series (MLLM4TS) system 100 that effectively leverages foundational models of natural language processing (NLP) and computer vision (CV) for time series analysis. This integrated framework is configured to handle multivariate time series data (e.g., multi-channel time series data) and perform one or more time series tasks (e.g., classification, anomaly detection, prediction, fusion learning, missing value imputation, clustering, etc.). The MLLM4TS system 100 includes two or more coding branches (e.g., a time series tokenizer 110, a plot tokenizer 120, and a text tokenizer 130), a pre-trained large-scale language model (LLM) 140, and a task module 150 having at least one output layer (e.g., at least one task head).

[0011] The MLLM4TS system 100 is trained using supervised learning for at least one specific task (e.g., classification, anomaly detection, etc.). In particular, the MLLM4TS system 100 includes several components (e.g., time series tokenizer 110, plot projector 126, and task module 150) whose parameters can be updated during this training process, as indicated by the flame icon, while also including several components (e.g., image encoder 124 and text encoder 132) whose parameters remain frozen during this training process, as indicated by the snowflake icon. Furthermore, as shown in Figure 1, the pre-trained LLM 140 is efficiently fine-tuned and is therefore indicated by both the flame icon and the snowflake icon. In this regard, the pre-trained LLM 140 updates at least some of its parameters through an efficient fine-tuning technique. According to efficient fine-tuning, the pre-trained LLM 140 benefits from not having to perform a complete update of all of its parameters, which could be costly, while still obtaining high-quality task-specific results. For example, an example of efficient fine-tuning of a pre-trained LLM140 involves fine-tuning the position embedding and normalization layers while keeping the self-attention layer and feedforward neural network (FFN) frozen. Specifically, during this training process, the MLLM4TS system 100 is trained using at least one applicable supervised loss function related to a particular task (e.g., cross-entropy loss for classification, reconstruction loss for anomaly detection, etc.) in which the parameters of a given component are updated.

[0012] As shown in Figure 1, the MLLM4TS system 100 includes multiple different coding branches for encoding time-series information of different modalities. Each coding branch encodes time-series information captured in a specific format (e.g., numerical data, digital images, text data, etc.). Specifically, the MLLM4TS system 100 is configured to include at least a time-series coding branch and a visual coding branch. For example, in Figure 1, the MLLM4TS system 100 includes at least these two coding branches (e.g., a time-series coding branch and a visual coding branch) and at least one additional coding branch (e.g., a language coding branch). That is, in the example shown in Figure 1, the MLLM4TS system 100 includes three coding branches: a time-series coding branch, a visual coding branch, and a language coding branch. In this regard, the MLLM4TS system 100 is configured to handle time-series information across these three different modalities. Figures 2, 3, and 4 show examples of each of these different coding branches. The resulting embeddings from these three coding branches are then sent to a pre-trained LLM140, which serves as the central hub of the MLLM4TS system 100.

[0013] FIG. 2 illustrates an example of a time series encoding branch according to an exemplary embodiment. By encoding a time series as a string of numbers separated by commas, an LLM can achieve reasonable zero-shot prediction performance. However, to bridge the modality gap between time series data and natural language, more sophisticated embedding techniques are often required to obtain comparable performance. Recognizing this challenge, the MLLM4TS system 100 includes a time series encoding branch that comprises a time series tokenizer 110, which has three components: a normalization component 112, a data patching component 114, and an embedding component 116. The time series tokenizer is co-trained with the MLLM4TS system 100 during the training process using supervised learning for at least one task (e.g., classification, anomaly detection, etc.) specified and selected based on the application scenario.

[0014] As shown in FIG. 2, the normalization component 112 receives multivariate time series data. The normalization component 112 normalizes the multivariate time series data. As a non-limiting example, the normalization component 112 includes (i) a standardization process that transforms the data to have a mean of 0 and a variance of 1, (ii) Min-Max normalization that transforms data features into the range [0,1], or (iii) any applicable normalization module. For example, referring to FIG. 1, as an example, the normalization component 112 normalizes the time series data of each channel. In FIG. 1, for example, the normalization component 112 is configured to (i) normalize "Speed Y" data, which is the first speed data of the first channel, (ii) normalize "Inter-vehicle distance" data of the second channel, and (iii) normalize "Speed X" data, which is the second speed data of the third channel. Next, to extract local semantic information, the time series tokenizer 110 normalizes the time series data for each respective channel, and then employs patching via the data patching component 114.

[0015] The data patching component 114 receives normalized time series data and generates data patches (e.g., segments) of the normalized time series data. For example, the data patching component 114 generates data patches by segmenting the normalized time series data based on a given patch size and a given stride parameter. For example, the data patching component 114 is configured to generate a set of data patches for multivariate time series data. For convenience of explanation, as an unrestricted example, the data patching component 114 receives normalized multivariate time series data such that the received normalized multivariate time series data includes (i) a sample of normalized time series data containing [y1, y2, y3, y4, y5, y6, y7, y8, y9] for a first channel, where y t (ii) represents normalized numerical data (e.g., the speed of the first vehicle) y at timestamp t, and further includes a sample of normalized time series data for the second channel, including [d1, d2, d3, d4, d5, d6, d7, d8, d9], where d t (iii) represents normalized numerical data (e.g., the distance between the first vehicle and the second vehicle) d at timestamp t, and further includes a sample of normalized time series data for the third channel, including [x1, x2, x3, x4, x5, x6, x7, x8, x9], where x tx represents normalized numerical data (e.g., the speed of the second vehicle) x at the timestamp t. In this example, given a predetermined patch size and a predetermined stride (e.g., patch size = 6, stride = 3), the data patching component 114 generates a data patch that includes (i) a first data patch containing [y1, y2, y3, y4, y5, y6], [d1, d2, d3, d4, d5, d6], and [x1, x2, x3, x4, x5, x6], and (ii) a second data patch containing [y4, y5, y6, y7, y8, y9], [d4, d5, d6, d7, d8, d9], and [x4, x5, x6, x7, x8, x9]. This patching technique allows a pre-trained LLM140 to analyze longer historical sequences of time-series data using data patches, while reducing information redundancy, given equivalent training time and processing unit memory (e.g., graphics processing unit (GPU) memory).

[0016] Further, the time series tokenizer 110 includes an embedding component 116 configured to generate time series embeddings using data patches (i.e., tokens of time series data). The embedding component 116 includes an embedding process that bridges the domain gap between numerical time series data and text language data. As an example, the time series tokenizer 110 includes a one-dimensional (1D) convolutional layer as the embedding component 116 for its simplicity and effectiveness. Specifically, the embedding component 116 is configured to convert normalized multivariate time series data associated with a plurality of data patches into a single univariate time series, and to generate time series embeddings that meet the input requirements of the pre-trained LLM 140. For example, as a non-limiting example, for convenience of description, the embedding component 116 receives a first data patch having a normalized multivariate time series including [y1,y2,y3,y4,y5,y6], [d1,d2,d3,d4,d5,d6], [x1,x2,x3,x4,x5,x6], and is configured to generate a single univariate time series including [y1,y2,y3,y4,y5,y6,d1,d2,d3,d4,d5,d6,x1,x2,x3,x4,x5,x6] via concatenation. As another example, the embedding component 116 receives a second data patch having a normalized multivariate time series including [y4,y5,y6,y7,y8,y9], [d4,d5,d6,d7,d8,d9], [x4,x5,x6,x7,x8,x9], and is configured to generate a single univariate time series including [y4,y5,y6,y7,y8,y9,d4,d5,d6,d7,d8,d9,x4,x5,x6,x7,x8,x9] via concatenation. Each univariate time series is then processed by a 1D convolutional layer, which generates a time series embedding and projects this time series embedding into the dimensional space required by the pre-trained LLM 140.

[0017] As described above, time-series coding branching enables a pre-trained LLM140 to process time-series data containing numerical data (e.g., numerical values ​​such as diagnostic data values ​​shown in Figure 1). However, time-series coding branching does not explicitly capture inter-channel dependencies. As a technical solution to this challenge, the MLLM4TS system 100 is configured to generate, acquire, and / or receive multiple visual representations of multivariate time-series data and provide this information to the pre-trained LLM140 in order to capture inter-channel dependencies. In this regard, the multivariate time-series data is generated into a set of visual representations, such as a set of graphic representations, so that inter-channel dependencies can be visually captured and provided to the pre-trained LLM140. For example, in Figure 1, the MLLM4TS system 100 is configured to generate a line plot for each channel of the multivariate time-series data. As shown in Figure 1, the set of line plots (e.g., three line plots) is combined into a single digital image, which is then input to the visual coding branching. Visual coding branching allows a pre-trained LLM140 to extract the global context and inter-channel dependencies of multivariate time series data, because these trends and inter-channel relationships are often more apparent in the visual representation of multivariate time series data. By providing this complementary visual information for multivariate time series data, a pre-trained LLM140 can better process and learn the underlying structures and relationships between multiple different channels of time series data.

[0018] Figure 3 shows an example of a visual coding branch according to an exemplary embodiment. In this example, the visual coding branch includes an image patching component 122, a pre-trained image encoder 124, and a plot projector 126. Alternatively, the visual coding branch may include the pre-trained image encoder 124 and the plot projector 126. More specifically, the image patching component 122 is configured to receive a single digital image on which a collection of line plots is displayed. The image patching component 122 is configured to generate a predetermined number of image patches using the digital image. Each image patch points to another different part of the digital image (e.g., another different pixel region). As a non-limiting example, a particular patch may include at least a particular portion of a particular line plot.

[0019] The pre-trained image encoder 124 is configured to receive either the digital image itself or an image patch of the digital image, depending on whether the image patching component 122 is used. If the image patching component 122 is not used, the pre-trained image encoder 124 is configured to generate an image embedding using pixels of the digital image having a set of graphic representations. Alternatively, if the image patching component 122 is used, the pre-trained image encoder 124 is configured to generate an image embedding using pixels of an image patch of the digital image having a set of graphic representations.

[0020] The image encoder 124 includes a pre-trained visual-language model (VLM) image coding network, or an applicable pre-trained image encoder. For example, the image encoder 124 includes an image encoder for a contrastive language-image pre-training (CLIP) model (e.g., the ViT-L / 14CLIP model). In this regard, since the CLIP model is not pre-trained with time-series images and is frozen during the training process (as indicated by the snowflake icon in Figure 3), the MLLM4TS system 100 incorporates efficient tuning techniques such as BitFit, LoRA, or similar tuning techniques into CLIP, thereby enabling the CLIP model to better adapt to the domain-specific characteristics of time-series data while maintaining training efficiency.

[0021] The visual coding branch also includes a plot projector 126. For example, in Figure 3, the plot projector 126 includes a multilayer perceptron (MLP), which is trained during the training process, as indicated by the flame icon illustrated on the plot projector 126 in Figure 3. The MLP functions as a visual tokenizer configured to map image features of image embeddings into a language embedding space. The plot projector 126 is configured to at least (i) receive image embeddings of a predetermined number of output dimensions from the image encoder 124, and (ii) use the image embeddings of a predetermined number of output dimensions from the image encoder 124 to generate image embeddings of a number of dimensions suitable for a pre-trained LLM 140.

[0022] As described above, in addition to the time-series coding branch and the visual coding branch, the MLLM4TS system 100 is configured to include one or more additional branches. One or more additional branches are configured to reinforce and enhance the multivariate time-series information (e.g., time-series embeddings and image embeddings) input to the pre-trained LLM 140. One or more additional branches may include language coding branches, acoustic coding branches, coding branches of any type applicable for downstream applications, or any number and combination thereof.

[0023] Referring to Figure 1, as an example, the MLLM4TS system 100 includes a language coding branch as an additional coding branch to the time-series coding branch and the visual coding branch. The language coding branch is configured to enhance multivariate time-series data using relevant contextual information. For example, the language coding branch is configured to receive text data. The text data is associated with or corresponds to the multivariate time-series data. With respect to the multivariate time-series data, the text data can provide background information, contextual information, relevant information, or any number and combination thereof. By including text data via the language coding branch, the pre-trained LLM 140 is guided to perform specific tasks more effectively.

[0024] Text data may include annotation data for multivariate time series data, descriptions of multivariate time series data, at least one task instruction relating to multivariate time series data, key statistics for multivariate time series data in linguistic prompt format, other data related to the time series data, or any number and combination thereof. Text data may describe or relate to one or more visual representations of the multivariate time series data (e.g., graphic representations such as line plots). Text data may relate to one, more, or all channels of the multivariate time series data. Additionally or alternatively, as an example, text data may incorporate statistics into prompts. By incorporating easily computable statistics into prompts, the pre-trained LLM140 can conserve its ability to capture more complex patterns and relationships within the multivariate time series data. In this regard, prompts are initialized using a text tokenizer 130, so that the parameters of the text tokenizer 130 are frozen during training and use (as indicated by the snowflake icon in Figure 4). This approach allows for an improvement in the capabilities of the text encoder 132 without compromising efficiency, thereby enabling the pre-trained LLM 140 to adapt more effectively to subtle differences in text data.

[0025] Figure 4 shows an example of a language coding branch according to an exemplary embodiment. In this example, the language coding branch includes a text tokenizer 130 configured to receive text data as input. For example, the text tokenizer 130 includes a text encoder 132. The text encoder 132 is configured to receive text data. The text encoder 132 is configured to use the text data to generate text embeddings. For example, the text encoder 132 may include a GPT (generative pre-trained transformer), BERT (bidirectional encoder representations from transformers), LLAMA (large language model meta AI), or an applicable text coding network. The text embeddings are then sent to a pre-trained LLM 140.

[0026] Referring to Figure 1, the pre-trained LLM140 functions as a central hub across multiple different modalities. In this example, the pre-trained LLM140 handles multiple different embeddings across different modalities. For example, the pre-trained LLM140 may include GPT2, LLAMA3, or an applicable pre-trained language model. For example, in Figure 1, if the MLLM4TS system 100 includes time-series coding branches, visual coding branches, and language coding branches, the pre-trained LLM140 is configured to generate semantic embeddings using time-series embeddings, image embeddings, and text embeddings. Alternatively, in other embodiments, if the MLLM4TS system 100 includes time-series coding branches and visual coding branches, the pre-trained LLM140 is configured to generate semantic embeddings using time-series embeddings and image embeddings. In each of these examples, the pre-trained LLM140 is configured to receive concatenated data as input, which is a combination of multiple different embeddings (e.g., time-series embeddings, image embeddings, text embeddings, etc.) received from different coding branches.

[0027] As mentioned above, the pre-trained LLM140 generates semantic embeddings, which are high-dimensional vector representations that capture the underlying meaning and relationships within the data. The semantic embeddings are generated by the pre-trained LLM140 from multiple different coding branches, thereby encoding temporal dependencies and structural patterns so that the task module 150 (e.g., one or more task heads) can efficiently process and interpret them. These semantic embeddings function as compact and informative representations of the input data (e.g., various modalities of time-series information), making them useful for at least one task (e.g., classification, anomaly detection, clustering, etc.). The MLLM4TS system 100 leverages the pre-trained LLM140 to generate semantic embeddings that encode contextual and relational information from various modalities of multivariate time-series data (e.g., multi-channel time-series data), thereby facilitating improvements in generalization and transfer learning. Semantic embeddings are provided within a structured feature space where more similar patterns or sequences are mapped closer to each other, and more dissimilar patterns are mapped further apart, enabling a pre-trained LLM140 to recognize trends, anomalies, and correlations across multiple different time-series data sets.

[0028] Furthermore, considering the large number of semantic embeddings (or output tokens) generated by the pre-trained LLM140, the MLLM4TS system 100 would require an extremely large prediction head to directly flatten all of the pre-trained LLM140 semantic embeddings (or output tokens) to make predictions relevant to a particular task. Recognizing this technical challenge and to improve efficiency, the MLLM4TS system 100 applies max pooling or average pooling across the semantic embeddings (or output tokens) output from the pre-trained LLM140 to consolidate these semantic embeddings into a single token, which is then linearly projected onto each task head of the task module 150 to generate output data or final predictions for a given task.

[0029] Furthermore, as shown in Figure 1, the MLLM4TS system 100 includes a task module 150 having at least one output layer (e.g., at least one task head). Each task head may include one or more artificial neural network layers. The task head is configured to perform a specific task in the time-series domain using semantic embeddings and to generate output data based on that specific task. The task module 150 is configured to include one or more task heads based on one or more specific tasks for a given application. For example, the task module 150 may include a classification head that performs a classification task. The task module 150 may include a prediction head that performs a prediction task. Also, fusion shot learning or zero shot learning can be applied to task heads (e.g., classification head, prediction head, etc.). The task module 150 may include an anomaly detection head that performs an anomaly detection task. The task module 150 may include a missing value imputation task head that performs a missing value imputation task. The task module 150 may include a clustering head that performs a clustering task. In this regard, the MLLM4TS system 100 includes task modules 150 configured for one or more specific tasks of a given application.

[0030] Furthermore, for illustrative and explanatory purposes, Figure 1 further illustrates a non-limiting data example relating to the MLLM4TS system 100. The MLLM4TS system 100 processes multivariate time series data, which includes time series data in multiple channels. For example, in Figure 1, the MLLM4TS system 100 receives multivariate time series data (e.g., numerical data relating to time) which includes (i) a first channel containing speed data of a first vehicle relating to time, (ii) a second channel containing the distance between a first vehicle and a second vehicle relating to time, and (iii) a third channel containing speed data of a second vehicle relating to time. The MLLM4TS system 100 also generates a digital image including a graph (e.g., a line plot) of this multivariate time series data. For example, as shown in Figure 1, the MLLM4TS system 100 generates (i) a first line plot of first vehicle speed data over time, (ii) a second line plot of inter-vehicle distance over time, and (iii) a third line plot of second vehicle speed data. The first line plot corresponds to the first channel of the time series data. The second line plot corresponds to the second channel of the time series data. The third line plot corresponds to the third channel of the time series data. Furthermore, in the example shown in Figure 1, the digital image displays three line plots in a 2x2 grid format, having (i) a first line plot displayed in a first color and arranged in the first row and first column of a 2x2 grid, (ii) a second line plot displayed in a second color and arranged in the first row and second column of a 2x2 grid, and (iii) a third line plot displayed in a third color and arranged in the second row and first column of a 2x2 grid, and (iv) a blank area arranged in the second row and second column of a 2x2 grid. In this regard, the blank area indicates that the multivariate time series data does not include a fourth channel and / or time series data for the fourth channel. In the example shown in Figure 1, the MLLM4TS system 100 also receives text data related to the time series data.In this case, the text data includes annotations indicating that the time series data corresponds to "sudden braking at a roundabout," thereby providing contextual information related to the multivariate time series data (e.g., numerical data) and the set of graphic representations of digital images.

[0031] Furthermore, with respect to the non-restrictive data example shown in Figure 1, the time series tokenizer 110 generates time series embeddings based on the time series data. The plot tokenizer 120 generates image embeddings based on the pixels of image patches in the digital image. The text tokenizer 130 generates text embeddings based on the text data. By concatenating these multimodal embeddings (e.g., time series embeddings, image embeddings, and text embeddings), concatenated data is generated. The pre-trained LLM 140 receives the concatenated data as input and generates semantic embeddings as output based on this concatenated data. In this particular example, the task module 150 includes a classification head. The classification head is configured to generate class data as output data for a classification task based on the semantic embeddings. In this non-restrictive example, the classification head predicts that the semantic embeddings belong to the "forward / backward jerk failure" class from among several classes (e.g., various different failure classes). Additionally or alternatively, the task module 150 includes an anomaly detection head, which is configured to categorize semantic embeddings associated with various modalities of multi-channel time-series information (e.g., time-series data, digital images with graphic representations of time-series data, and text data with context for time-series data) and generate output data indicating whether the multi-channel time-series information is anomalous or not (i.e., normal). For example, in Figure 1, the anomaly detection head determines that the multi-channel time-series information is "anomalous" and generates output indicating that an anomaly has been detected. As demonstrated by this non-limiting data example, the MLLM4TS system 100 is advantageous in that it can perform at least one specific task based on multimodal time-series information, both from a local and global perspective of the multi-channel time-series data.

[0032] Figures 5, 6, 7, and 8 show other non-limiting examples of different digital images that may be generated based on other non-limiting examples of multivariate time series data. The plot tokenizer 170 is not limited to processing specific versions and layouts of the visual representations shown in Figures 1, 5, 6, 7, and 8, and may include other visual representations that are assumed to be applicable to multivariate time series data based on several factors (e.g., type of time series data, number of channels, specific use, specific task, etc.). As mentioned above, the plot tokenizer 120 is configured to receive at least one digital image that displays a visual representation (e.g., graphic representation) of the multivariate time series data. The multivariate time series data may include, or correspond to, a given number of channels for a given use. For example, in the example shown in Figure 1, the multivariate time series data includes a first set of diagnostic data for a first channel, a second set of diagnostic data for a second channel, and a third set of diagnostic data for a third channel. As other non-limiting examples, Figures 5, 6, 7, and 8 each show digital images generated as input to a visual coding branch. Each digital image contains four graphical representations of different time-series data for four channels. In each of these examples, for a given channel, the time-series data is graphically represented as a graph (e.g., a line plot). Each line plot is plotted with respect to the x-axis representing time and the y-axis representing numerical data (e.g., numerical values ​​such as speed, distance, diagnostic data, sensor data, etc.).

[0033] In each of these examples, the four line plots for the four channels are displayed together within a single digital image (or one common digital image) consisting of multiple pixels. Each example shows a different version (e.g., layout, shape, orientation, color, etc.) of how the line plots are displayed within the digital image for processing by the plot tokenizer 120. For example, Figure 5 shows an example of a digital image 500 in which the four line plots are displayed in a 2x2 grid format. Specifically, the digital image 500 includes (i) a first line plot for the time series data of the first channel, located in the first row and first column of the grid; (ii) a second line plot for the time series data of the second channel, located in the first row and second column of the grid; (iii) a third line plot for the time series data of the third channel, located in the second row and first column of the grid; and (iv) a fourth line plot for the time series data of the fourth channel, located in the second row and second column of the grid.

[0034] Furthermore, in Figure 5, the digital image 500 displays line plots along with the x-axis (e.g., time data) extending parallel to the top / bottom edges of the digital image 500, and the y-axis (e.g., numerical data) extending parallel to the right / left edges of the digital image 500. In addition, in Figure 5, the four line plots are displayed in the same color. In contrast, in Figure 5, other information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed in a different color from the four line plots. For example, in Figure 5, each line plot is displayed in one specific color (e.g., orange or another selected color), while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another selected color). However, in other examples, depending on the given application, these other pieces of information and the line plots may be displayed in the same color.

[0035] As another example, Figure 6 shows an example of a digital image 600 in which four line plots are displayed in the form of a vertical stack with respect to the vertical orientation. As shown in Figure 6, the digital image 600 displays a stack such that the top of the stack is located at the top of the digital image 600 and the bottom of the stack is located at the bottom of the digital image 600. In this case, the digital image 600 displays a vertical stack in which the first line plot for the first channel, the second line plot for the second channel, the third line plot for the third channel, and the fourth line plot for the fourth channel are arranged in this order, with the first line plot at the top of the stack and the fourth line plot at the bottom of the stack. Alternatively, the digital image can be displayed as a vertical stack, with the fourth line plot for the fourth channel, the third line plot for the third channel, the second line plot for the second channel, and the first line plot for the first channel arranged in this order, with the fourth line plot at the top of the stack and the first line plot at the bottom. In each of these cases, the x-axis (e.g., time data) extends parallel to the top / bottom edges of the digital image 600, and the y-axis (e.g., numerical data) extends parallel to the right / left edges of the digital image 600. Furthermore, in Figure 6, the four line plots are displayed in the same color. In contrast, in Figure 6, other information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed in a different color from the four line plots. For example, in Figure 6, each line plot is displayed in one specific color (e.g., orange or another selected color), while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another selected color). However, in other examples, depending on the application, these other pieces of information and the line plots may be the same color.

[0036] As an example, Figure 7 shows an example of a digital image 700 in which four line plots are displayed in a horizontal stack configuration with respect to the vertical orientation. The digital image 700 contains four line plots in a stack arrangement that is horizontally oriented when viewed vertically. Specifically, in this horizontal stack arrangement, the x-axis (e.g., time data) is parallel to the left / right side of the digital image 700, and the y-axis (e.g., numerical data) is parallel to the top / bottom side of the digital image 700. In Figure 7, the line plots are oriented so that the x-axis (e.g., time) increases from the top to the bottom of the digital image 700. Alternatively, the line plots can be shown so that the x-axis (e.g., time) increases from the bottom to the top of the page. Furthermore, in Figure 7, the line plots are oriented in a horizontal stack configuration, with the first line plot of the first channel, the second line plot of the second channel, the third line plot of the third channel, and the fourth line plot of the fourth channel stacked in this order from left to right of the digital image 700. Alternatively, the line plots can be oriented in a stack configuration, with the fourth line plot of the fourth channel, the third line plot of the third channel, the second line plot of the second channel, and the first line plot of the first channel stacked in this order from left to right of the digital image. In addition, in Figure 7, the four line plots are displayed in the same color. In contrast, in Figure 7, other information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed in a different color from the four line plots. For example, in Figure 7, each line plot is displayed in one specific color (e.g., orange or another selected color), while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another selected color). However, in other examples, depending on the application, these other pieces of information and the line plots may be the same color.

[0037] Figure 8 shows another example of digital image 800, in which four line plots are displayed in a 2x2 grid format, similar to digital image 500 in Figure 5. However, digital image 800 in Figure 8 differs from digital image 500 in that it includes color-coded line plots such that the first line plot in the first channel is a first color (e.g., blue or another selected color), the second line plot in the second channel is a second color (e.g., green or another selected color), the third line plot in the third channel is a third color (e.g., red or another selected color), and the fourth line plot in the fourth channel is a fourth color (e.g., purple or another selected color). As demonstrated by this example, each line plot can be displayed in a color-coded manner, each in its own selected color. At the same time, in Figure 8, other information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed in a different color from the four line plots. For example, in Figure 8, each line plot is displayed in a selected color (e.g., red, blue, green, and purple) according to the color coding scheme, while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another selected color). However, in other examples, there may be flexibility in the color coding of the line plots or other information, depending on the given application. For example, one or more of the line plots may be colored in one or more specific colors, while one or more of the remaining line plots may be colored in one or more other specific colors. Also, in Figure 8, the digital image 800 displays each line plot along with the x-axis (e.g., time data) extending parallel to the top / bottom edges of the digital image 800 and the y-axis (e.g., numerical data) extending parallel to the right / left edges of the digital image 800.

[0038] Figure 9 is a block diagram of an example of system 900 including the MLLM4TS system 100. System 900 includes at least a processing system 902. Processing system 902 includes at least one processing unit. For example, processing system 902 may include an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any processing technology, or any number and combination thereof. Processing system 902 is operable to provide the functions described herein.

[0039] System 900 includes at least one sensor system 904. Sensor system 904 includes one or more sensors (e.g., speed sensor, radar, infrared sensor, image sensor, GPS sensor, acoustic sensor, motion sensor, etc.). In particular, sensor system 904 includes one or more sensors that provide multivariate time-series data. Sensor system 904 is operable to communicate with one or more other components of system 900 (e.g., processing system 902 and memory system 910). For example, sensor system 904 can provide sensor data (e.g., time-series data) which is then processed by processing system 902. Sensor system 904 is local or remote with respect to one or more components of system 900, or a combination thereof (e.g., some are local and some are remote). Upon receiving sensor data (for example, time-series data from multiple channels), the processing system 902 is configured to process this sensor data (in conjunction with the application program 912, the MLLM4TS system 100, machine learning (ML) data 914, other relevant data 916, or any number and combination thereof).

[0040] System 900 includes a memory system 910 operably connected to a processing system 902. In this regard, the processing system 902 communicates data with the memory system 910. The memory system 910 includes at least one non-temporary computer-readable storage medium, which is configured to store and provide access to various types of data in order to enable the processing system 902 to perform operations and functions as disclosed herein at least. The memory system 910 includes one or more memory devices. The memory system 910 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable storage technology. For example, the memory system 910 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any number and combination thereof. The memory system 910 includes computer-readable data configured to perform at least the functions described herein when executed by the processing system 902. The computer-readable data may include instructions, code, routines, various related data, software techniques, or any number and combination thereof.

[0041] The memory system 910 includes computer-readable data for the application program 912. The application program 912 is configured to perform the functions described in this disclosure with respect to the MLLM4TS system 100. For example, the application program 912 can associate processes associated with the MLLM4TS system 100 with respect to training, tuning, testing, validation, deployment, adoption, or any number and combination thereof. The application program 912 may be configured to apply output data from the MLLM4TS system 100 (e.g., class data such as a “forward / backward jerk failure” class or an “abnormal” label) to a given application and / or at least one downstream technical system.

[0042] Furthermore, the memory system 910 includes computer-readable data for the MLLM4TS system 100, which is configured to perform operations and functions as described, for example, with respect to Figures 1, 2, 3, and 4 of this disclosure. The memory system 910 also includes ML data 914, which includes various data (e.g., parameters, sensor data, time-series data, digital images, text data, loss data, training data, channel data, etc.) related to various aspects of one or more parts of the MLLM4TS system 100 (e.g., training, tuning, fine-tuning, testing, validation, deployment, adoption, etc.) as described in this disclosure. The memory system 910 also includes computer-readable data for other related data 916, which provides various data (e.g., operating system, etc.) so that the system 900 and / or processing system 902 can perform functions as described herein. Furthermore, the system 900 may include one or more input / output devices 906 (e.g., display devices, microphones, speakers, etc.) for receiving inputs and / or providing outputs to one or more components of the system 900.

[0043] Furthermore, System 900 includes other functional modules 908, such as any suitable hardware, software, or combination thereof, that support or contribute to the functions of System 900 and MLLM4TS System 100. For example, the other functional modules 908 include communication technologies (e.g., wired communication technologies, wireless communication technologies, or combinations thereof) that enable the components of System 900 to communicate with each other and / or with one or more other computing devices via at least one communication network. One or more other computing devices (not shown) may include mobile communication devices, smartphones, laptops, tablets, servers, cloud computing systems, etc.

[0044] Figure 10 shows a schematic diagram of the interaction between a computer-controlled machine 1000 and a control system 1002 according to another exemplary embodiment. The computer-controlled machine 1000 includes actuators 1004 and sensors 1006. Actuators 1004 may include one or more actuators, and sensors 1006 may include one or more sensors. Sensors 1006 are configured to detect the status of the computer-controlled machine 1000. Sensors 1006 may be configured to encode the detected status into a sensor signal 1008 and transmit the sensor signal 1008 to the control system 1002. Non-limiting examples of sensors 1006 include video, radar, LiDAR, ultrasonic sensors, image sensors, acoustic sensors, and motion sensors.

[0045] The control system 1002 is configured to receive a sensor signal 1008 from the computer-controlled machine 1000. As described below, the control system 1002 may be further configured to calculate an actuator control command 1010 depending on the sensor signal and to transmit the actuator control command 1010 to the actuator 1004 of the computer-controlled machine 1000.

[0046] As shown in Figure 10, the control system 1002 includes a receiving unit 1012. The receiving unit 1012 may be configured to receive a sensor signal 1008 from the sensor 1006 and convert the sensor signal 1008 into an input signal x. In an alternative embodiment, the sensor signal 1008 is received directly as an input signal x without using the receiving unit 1012. Each input signal x may be a part of each sensor signal 1008. The receiving unit 1012 may be configured to process each sensor signal 1008 to generate each input signal x. The input signals x may include data corresponding to a digital image recorded by the sensor 1006.

[0047] The control system 1002 includes a classifier 1014. In this example, the classifier 1014 includes an MLLM4TS system 100 having a task module 150 which includes at least a classifier head as a task head. The classifier 1014 may be configured to classify an input signal x into one or more labels using an ML algorithm via the MLLM4TS system 100. The classifier 1014 is configured to be parameterized by a parameter θ. The parameter θ may be stored in a non-volatile storage 1016, or may be provided by the non-volatile storage 1016. The classifier 1014 is configured to determine an output signal y from an input signal x. Each output signal y contains information to assign one or more labels to each input signal x. The classifier 1014 can transmit the output signals y to a conversion unit 1018. The conversion unit 1018 is configured to convert the output signals y into actuator control commands 1010. The control system 1002 is configured to transmit actuator control commands 1010 to the actuator 1004, and the actuator 1004 is configured to operate the computer-controlled machine 1000 in response to the actuator control commands 1010. In some embodiments, the actuator 1004 is configured to operate the computer-controlled machine 1000 directly based on an output signal y.

[0048] When actuator 1004 receives an actuator control command 1010, actuator 1004 is configured to perform an action corresponding to the associated actuator control command 1010. Actuator 1004 may include control logic configured to translate actuator control command 1010 into a second actuator control command used to control actuator 1004. In one or more embodiments, the actuator control command 1010 can be used to control a display instead of, or in addition to, the actuator.

[0049] In some embodiments, the control system 1002 includes sensor 1006 in place of or in addition to computer-controlled machine 1000, which includes sensor 1006. The control system 1002 may also include actuator 1004 in place of or in addition to computer-controlled machine 1000, which includes actuator 1004. As shown in Figure 10, the control system 1002 also includes processor 1020 and memory 1022. Processor 1020 may include one or more processors. Memory 1022 may include one or more non-temporary memory devices. In one or more embodiments, classifier 1014 may be implemented by a control system 1002 which includes non-volatile storage 1016, processor 1020, and memory 1022.

[0050] Non-volatile storage 1016 may include one or more persistent data storage devices, such as hard drives, optical drives, tape drives, non-volatile solid-state devices, cloud storage, or any other devices capable of persistently storing information. Processor 1020 may include one or more devices selected from the following, which process (analog or digital) signals based on computer executable instructions residing in memory 1022: namely, high-performance computing (HPC) systems including high-performance cores, graphics processing units, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices. Memory 1022 may include, but is not limited to, one or more memory devices, including RAM, ROM, volatile memory, non-volatile memory, static random-access memory (SRAM), dynamic random-access memory (DRAM), flash memory, cache memory, or any other devices capable of storing information.

[0051] The processor 1020 resides in the non-volatile storage 1016 and is configured to load and execute computer executable instructions that embody one or more ML algorithms and / or methodologies of one or more embodiments into memory 1022. The non-volatile storage 1016 may contain one or more operating systems and applications. The non-volatile storage 1016 can store compiled and / or interpreted computer programs created using a variety of programming languages ​​and / or technologies, including, but not limited to, Java, C, C++, C#, Objective-C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, either alone or in combination.

[0052] When executed by processor 1020, the computer-executable instructions of non-volatile storage 1016 can cause the control system 1002 to implement one or more ML algorithms and / or methodologies for employing a classifier 1014 as disclosed herein. Non-volatile storage 1016 may also contain ML data (including model parameters) that support the functions, features and processes of one or more embodiments described herein.

[0053] Program code embodying the algorithms and / or methodologies described herein may be distributed individually or collectively as various different forms of program products. The program code may be distributed using computer-readable storage media containing computer-readable program instructions for causing a processor to execute an aspect of one or more embodiments. Computer-readable storage media that are essentially non-transient may include volatile and non-volatile, removable and non-removable tangible media implemented in any way or technique for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable compact disk read-only memory (CD-ROM), or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store information of a desired information and can be read by a computer. Computer-readable program instructions may be downloaded from a computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or they may be downloaded via a network to an external computer or external storage device.

[0054] Computer-readable program instructions stored on a computer-readable medium may be used to cause a computer, other types of programmable data processing devices, or other devices to function in a particular manner so that the instructions stored on the computer-readable medium produce a product containing instructions that perform functions, behaviors, and / or actions specified in a flowchart or diagram. In certain alternative embodiments, multiple functions, behaviors, and / or actions specified in a flowchart and diagram may be rearranged, processed serially, and / or processed simultaneously according to one or more embodiments. Furthermore, both the flowchart and / or diagram may contain more or fewer nodes or blocks than those illustrated according to one or more embodiments. Furthermore, the process, method, or algorithm may be embodied in whole or in part using appropriate hardware components, such as ASICs, FPGAs, state machines, controllers, or other hardware components or hardware devices, or combinations of hardware components, software components, and firmware components.

[0055] Figure 11 shows a schematic diagram of a control system 1002 configured to control a vehicle 1100, which may be at least partially autonomous or partially autonomous robot. The vehicle 1100 includes actuators 1004 and sensors 1006. Sensors 1006 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., Global Positioning System). One or more of the one or more specific sensors may be incorporated into the vehicle 1100. In place of or in addition to the one or more specific sensors identified above, sensor 1006 may include a software module configured to determine the state of actuator 1004 when executed. A non-limiting example of a software module includes a weather information software module configured to determine the current or future state of the weather near or elsewhere than the vehicle 1100.

[0056] The classifier 1014 of the vehicle 1100's control system 1002 may be configured to classify objects in the vehicle 1100's vicinity depending on an input signal x. In such an embodiment, the output signal y may contain information classifying or characterizing objects in the vehicle 1100's vicinity. According to this information, an actuator control command 1010 can be identified. The actuator control command 1010 can be used to navigate the vehicle 1100 and avoid collisions based on the classification provided by the classifier 1014.

[0057] In some embodiments, the vehicle 1100 is at least partially autonomous or fully autonomous. Actuators 1004 can be implemented in the brakes, propulsion system, engine, drivetrain, steering, etc., of the vehicle 1100. Actuator control commands 1010 can be identified so that the actuators 1004 are controlled to allow the vehicle 1100 to avoid collisions with detected objects. Detected objects can also be identified and classified according to what the classifier 1014 considers most likely to be, such as pedestrians, trees, or any appropriate label. Actuator control commands 1010 can be identified depending on the classification of objects from digital images generated via sensor 1006.

[0058] In some embodiments where the vehicle 1100 is at least partially autonomous, the vehicle 1100 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, walking, or other locomotion actions. The mobile robot may be at least partially autonomous lawnmowers or at least partially autonomous cleaning robots. In such embodiments, actuator control commands 1010 can be specified to control the propulsion unit, steering unit and / or brake unit of the mobile robot so that it can navigate according to classifications provided by the classifier 1014 and / or avoid collisions with objects.

[0059] In some embodiments, the vehicle 1100 is at least partially autonomous robot in the form of a horticultural robot. In such embodiments, the vehicle 1100 can use an optical sensor as a sensor 1006 to determine the state of plants in the environment near the vehicle 1100. The actuator 1004 may be a nozzle configured to spray a chemical. Depending on the identified species of plant and / or the identified state of the plant via a classifier 1014, an actuator control command 1010 can be specified to cause the actuator 1004 to spray an appropriate amount of appropriate chemical onto the plant.

[0060] Figure 12 shows a schematic diagram of a control system 1002 configured to control a system 1200 (e.g., a manufacturing machine) which may include punch cutters, cutters, gun drills, manufacturing tools, or similar equipment in a manufacturing system 1202, for example, a part of a manufacturing line. The control system 1002 may be configured to control an actuator 1004 configured to control the system 1200 (e.g., a manufacturing machine).

[0061] Sensor 1006 of system 1200 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more objects associated with the manufacture of product 1204. Classifier 1014 may be configured to make decisions from one or more of the captured properties. Actuator 1004 may be configured to control system 1200 (e.g., a manufacturing machine) in a specified state regarding the manufacture of product 1204 for subsequent manufacturing steps that manufacture product 1204. Actuator 1004 may be configured to control the function of system 1200 (e.g., a manufacturing machine) in a subsequent state of product 1206 in a specified state of product 1204.

[0062] Figure 13 shows a control system 1002 configured to control a surveillance system 1300 (e.g., a security system). The surveillance system 1300 may be configured to physically control access through the door 1302. A sensor 1006 may be configured to detect scenes relevant to determining whether access is permitted. The sensor 1006 may be an optical sensor configured to generate and transmit image and / or video data. Such image and / or video data can be used by the control system 1002 to detect and classify objects (e.g., people, animals, bicycles, weapons, trash cans, etc.) that may be present within the detection area of ​​the sensor 1006 near the door 1302.

[0063] Furthermore, the control system 1002 may be configured to generate actuator control commands 1010 in response to the classification of one or more objects in the image and / or video data via the classifier 1014. The control system 1002 is configured to transmit the actuator control commands 1010 to the actuator 1004. In this embodiment, the actuator 1004 is configured to lock or unlock the door 1302 in response to the actuator control commands 1010. In some embodiments, non-physical and logical access control is also possible.

[0064] The monitoring system 1300 may be a security system. In such an embodiment, the sensor 1006 includes at least an image sensor or camera configured to detect scenes under security, and the control system 1002 is configured to control the display 1304. The classifier 1014 is configured to classify scenes, for example, to determine whether a scene detected by the sensor 1006 is suspicious. The control system 1002 is configured to send actuator control commands 1010 to the display 1304 in response to the classification. The display 1304 may be configured to adjust what is displayed in response to the actuator control commands 1010. For example, the display 1304 may highlight objects that the classifier 1014 has deemed suspicious.

[0065] Figure 14 shows a schematic diagram of a control system 1002 configured to control an imaging system 1400, such as a magnetic resonance imaging (MRI) device, an X-ray imaging device, or an ultrasound device. Sensor 1006 may be, for example, an imaging sensor. Classifier 1014 may be configured to determine the classification of all or part of a detected image. Actuator control commands 1010 are selected based on the classification obtained from classifier 1014. For example, classifier 1014 may interpret a region of a digital image as either abnormal or non-abnormal. In this case, actuator control commands 1010 may be selected to display the digital image on display 1402 and highlight the potentially abnormal region through the classification provided by classifier 1014.

[0066] As described in this disclosure, each embodiment includes numerous advantageous features and benefits. For example, each embodiment includes an MLLM4TS system 100 configured to perform specific tasks related to system forensics through a machine learning framework that uses various modalities of time-series information. Furthermore, the MLLM4TS system 100 is configured to handle multi-channel time-series data when performing at least one specific task. Furthermore, the MLLM4TS system 100 provides an integrated framework that leverages a multimodal foundational model for time-series analysis. In an exemplary embodiment, the MLLM4TS system 100 includes three coding branches, e.g., a time-series coding branch, a visual coding branch, and a language coding branch. Specifically, the MLLM4TS system 100 is configured to segment numerical time-series data into patches and generate time-series embeddings using a channel-independent method to preserve local semantic information. The MLLM4TS system 100 also transforms the time-series data into a graphical representation (e.g., a line plot) to capture the global context and inter-channel dependencies. The MLLM4TS system 100 is configured to incorporate graphic representations of time-series data from multiple channels into a single digital image. The digital image can display the graphic representations using a color coding scheme, thereby providing additional visual information that consequently improves performance on a given task. The MLLM4TS system 100 includes a visual base model (e.g., image encoder 124) for generating image embeddings based on at least the pixels of its single digital image. The MLLM4TS system 100 is also configured to enrich the time-series data with relevant descriptions, task instructions, key statistics in language prompt format, arbitrary relevant text data, or any number or combination thereof. Furthermore, the MLLM4TS system 100 employs a pre-trained LLM 140 as a central hub for processing concatenated data of embeddings resulting from these various modalities for a particular task.The MLLM4TS system 100 has an advantage in that it is an integrated framework that can handle multimodal time-series information from multiple channels and can be configured to perform one or more diverse tasks in the time-series domain.

[0067] Furthermore, the above description is intended to be illustrative rather than restrictive, and is provided in relation to specific uses and their requirements. Those skilled in the art will understand from the above description that the invention can be implemented in various forms, and that various embodiments can be implemented individually or in combination. Therefore, while embodiments of the invention have been described in relation to specific examples, the general principles defined herein can be applied to other embodiments and uses without departing from the spirit and scope of the embodiments described, and the true scope of the embodiments and / or methods of the invention is not limited to the illustrated and described embodiments, as various modifications become apparent to those skilled in the art by examining the drawings, specification and the following claims. Additionally or alternatively, components and functions may be separated or combined in ways different from those of the various embodiments described, and may be described using different terminology. The above and other variations, modifications, additions and improvements may be included within the scope of this disclosure as defined in the following claims.

Claims

1. A computer-implemented method for performing tasks in the time-series domain, The computer-implemented method described above is Receiving time-series data including numerical data, To generate a digital image that displays a set of graphic representations of the aforementioned time-series data, The tokenizer generates time series embeddings using the aforementioned time series data, To generate an image embedding using the pixels of the digital image via an image encoder, To generate semantic embeddings based on at least the time-series embeddings and image embeddings via a pre-trained large-scale language model (LLM), Performing the task through at least one task head and generating output data for the task using the semantic embedding, Computerized methods, including those mentioned above.

2. The computer-implemented method described above is Receiving text data that provides context for the aforementioned time-series data, Using a pre-trained text encoder, generate text embeddings using the text data, The process involves generating concatenated data by concatenating the aforementioned time-series embedding, the aforementioned image embedding, and the aforementioned text embedding. It further includes, The pre-trained LLM uses the concatenated data to generate the semantic embeddings. The computer-implemented method according to claim 1.

3. The aforementioned time-series data includes a set of channels having diagnostic data. The set of graphic representations includes at least one line plot for each channel of the diagnostic data. The computer-implemented method according to claim 1.

4. The computer-implemented method described above is To generate an image patch of the aforementioned digital image. It further includes, The image encoder receives the image patch as input and uses the image patch to generate the image embedding. The computer-implemented method according to claim 1.

5. The computer-implemented method described above is To generate a data patch of the aforementioned time-series data. It further includes, The tokenizer uses the data patch to generate the time series embedding. The computer-implemented method according to claim 1.

6. The computer-implemented method described above is This involves generating a univariate time series by concatenating multivariate time series data associated with each data patch, The process involves generating the time series embedding using the univariate time series via a one-dimensional convolutional layer, Further including, The computer-implemented method according to claim 5.

7. The set of graphic representations includes at least a first line plot and a second line plot, The set of graphic representations is color-coded such that the first line plot is displayed in a different color from the second line plot. The aforementioned image embedding is generated based on the colors of the graphic representation set. The computer-implemented method according to claim 1.

8. The set of graphic representations includes at least a first line plot and a second line plot, The digital image displays the first line plot and the second line in the same row or column. The computer-implemented method according to claim 1.

9. The aforementioned tasks include classification, anomaly detection, prediction, missing value imputation, or clustering. The computer-implemented method according to claim 1.

10. The computer-implemented method further includes controlling an actuator based on the output data. The computer-implemented method according to claim 1.

11. One or more processors, One or more computer memories that communicate data with the aforementioned one or more processors, A system equipped with, The aforementioned one or more computer memories store computer-readable data. The computer-readable data includes instructions that cause the one or more processors to perform a method for performing a task in a time-series domain, when executed by the one or more processors. The aforementioned method, Receiving time-series data including numerical data, To generate a digital image that displays a set of graphic representations of the aforementioned time-series data, The tokenizer generates time series embeddings using the aforementioned time series data, To generate an image embedding using the pixels of the digital image via an image encoder, To generate semantic embeddings based on at least the time-series embeddings and image embeddings via a pre-trained large-scale language model (LLM), Performing the task through at least one task head and generating output data for the task using the semantic embedding, A system that includes this.

12. The aforementioned method, Receiving text data that provides context for the aforementioned time-series data, Using a pre-trained text encoder, generate text embeddings using the text data, The process involves generating concatenated data by concatenating the aforementioned time-series embedding, the aforementioned image embedding, and the aforementioned text embedding. It further includes, The pre-trained LLM uses the concatenated data to generate the semantic embeddings. The system according to claim 11.

13. The aforementioned time-series data includes a set of channels having diagnostic data. The set of graphic representations includes at least one line plot for each channel of the diagnostic data. The system according to claim 11.

14. The aforementioned method, To generate an image patch of the aforementioned digital image. It further includes, The image encoder receives the image patch as input and uses the image patch to generate the image embedding. The system according to claim 11.

15. The aforementioned method, To generate a data patch of the aforementioned time-series data. It further includes, The tokenizer uses the data patch to generate the time series embedding. The system according to claim 11.

16. The aforementioned method, This involves generating a univariate time series by concatenating multivariate time series data associated with each data patch, The process involves generating the time series embedding using the univariate time series via a one-dimensional convolutional layer, Further including, The system according to claim 15.

17. The set of graphic representations includes at least a first line plot and a second line plot, The set of graphic representations is color-coded such that the first line plot is displayed in a different color from the second line plot. The aforementioned image embedding is generated based on the colors of the graphic representation set. The system according to claim 11.

18. The set of graphic representations includes at least a first line plot and a second line plot, The digital image displays the first line plot and the second line in the same row or column. The system according to claim 11.

19. The aforementioned tasks include classification, anomaly detection, prediction, missing value imputation, or clustering. The system according to claim 11.

20. The system further comprises an actuator, The actuator is controlled at least based on the output data. The system according to claim 11.

Images