Systems and methods for system forensics with machine learning framework

Abstract

A computer-implemented method and system involve performing a task in a time series domain using different modalities of time series information. Time series data is received and includes numerical data of a set of channels. A digital image is generated and displays a set of graphical representations of the time series data. A tokenizer uses the time series data to generate time series embeddings in a channel-independent manner to preserve local semantic information. An image encoder uses the digital image to generate image embeddings to capture global context and cross-channel dependencies of the time series data. A pre-trained large language model (LLM) generates semantic embeddings based on at least the time series embeddings and the image embeddings. At least one task head performs the task and generates output data for the task using the semantic embeddings. The task can include classification, anomaly detection, forecasting, imputation, clustering, etc.

Description

Technical Field

[0001] This disclosure generally relates to forensics via time series analysis, computer vision, and natural language processing, and more particularly to performing specific tasks via machine learning frameworks that utilize various modalities of time series information. Background Technology

[0002] Following the success of pre-trained models in Natural Language Processing (NLP) and Computer Vision (CV), there is growing interest in adapting pre-trained Large Language Models (LLMs) to time series analysis. However, unlike NLP and CV, time series analysis still heavily relies on specialized approaches for each specific task, such as classification, anomaly detection, prediction, and few-shot learning.

[0003] While meticulous preprocessing of time-series data and specially designed prompts can enhance the zero-shot performance of LLMs on time-series tasks, the semantic gap between natural language and numerical time-series data remains a significant challenge for applying LLMs to time-series tasks. To address this modality gap, recent methods have focused on fine-tuning specific components of the LLM or training additional modules to preprocess the time-series data itself before feeding it into the LLM. However, these methods may struggle to capture a global perspective of the time-series data. Summary of the Invention

[0004] The following is a summary of certain embodiments described in detail below. The described aspects are presented merely to provide the reader with a brief overview of these certain embodiments, and the description of these aspects is not intended to limit the scope of this disclosure. In fact, this disclosure may cover various aspects that may not be explicitly set forth below.

[0005] According to at least one aspect, a computer-implemented method relates to performing a task in a time-series domain. The method includes receiving time-series data, said time-series data including numerical data. The method includes generating a digital image displaying a set of graphical representations of the time-series data. The method includes generating time-series embeddings using the time-series data via a word segmenter. The method includes generating image embeddings using pixels of the digital image via an image encoder. The method includes generating semantic embeddings based at least on the time-series embeddings and image embeddings via a pre-trained Large Language Model (LLM). The method includes performing the task via at least one task head and generating output data for the task using the semantic embeddings.

[0006] According to at least one aspect, the system includes one or more processors and one or more non-transitory computer memories. The one or more processors communicate data with the one or more non-transitory computer memories. The one or more non-transitory computer memories have computer-readable data stored thereon. The computer-readable data includes instructions, which, when executed by the one or more processors, cause the one or more processors to perform a method relating to performing a task in a time-series domain. The method includes receiving time-series data, the time-series data including numerical data. The method includes generating a digital image displaying a set of graphical representations of the time-series data. The method includes generating time-series embeddings using the time-series data via a word segmenter. The method includes generating image embeddings using pixels of the digital image via an image encoder. The method includes generating semantic embeddings based at least on the time-series embeddings and image embeddings via a pre-trained Large Language Model (LLM). The method includes performing a task via at least one task head and generating output data for the task using the semantic embeddings.

[0007] These and other features, aspects, and advantages of the invention are discussed in the following detailed description with reference to the accompanying drawings, throughout which similar characters denote similar or analogous parts. Furthermore, the drawings are not necessarily drawn to scale, as some features may be enlarged or reduced to show details of specific components. Attached Figure Description

[0008] Figure 1 This is a diagram summarizing examples of time-series multimodal large language model (MLLM4TS) systems according to exemplary embodiments of the present disclosure.

[0009] Figure 2 An example of a time series word segmenter according to an exemplary embodiment of the present disclosure is illustrated.

[0010] Figure 3 An example of a drawing word segmenter according to an exemplary embodiment of the present disclosure is illustrated.

[0011] Figure 4 An example of a text segmenter according to an exemplary embodiment of the present disclosure is illustrated.

[0012] Figure 5 The illustration shows a non-limiting example of a digital image according to an exemplary embodiment of the present disclosure, which displays a set of graphical representations of multivariate time series data in a 2×2 grid format.

[0013] Figure 6 The illustration shows a non-limiting example of a digital image according to an exemplary embodiment of the present disclosure, which displays a set of graphical representations of multivariate time series data in a vertically stacked manner.

[0014] Figure 7 The illustration shows a non-limiting example of a digital image according to an exemplary embodiment of the present disclosure, which displays a set of graphical representations of multivariate time series data in a horizontally stacked manner.

[0015] Figure 8 The illustration shows a non-limiting example of a digital image according to an exemplary embodiment of the present disclosure, which displays a set of graphical representations of multivariate time series data in a 2×2 grid format, wherein each graphical representation is color-coded and plotted using different colors.

[0016] Figure 9 This is a diagram illustrating an example of a system including an MLLM4TS system according to at least one exemplary embodiment of the present disclosure.

[0017] Figure 10 This is a diagram illustrating an example of interaction between a computer-controlled machine and a control system according to at least one exemplary embodiment of the present disclosure.

[0018] Figure 11 This is according to at least one example embodiment of the present disclosure. Figure 10 A diagram of a control system configured to control a mobile machine that is at least partially or fully autonomous.

[0019] Figure 12 This is according to at least one example embodiment of the present disclosure. Figure 10 A diagram of a control system configured to control manufacturing machines in a manufacturing system, such as a part of a production line.

[0020] Figure 13 At least one example embodiment according to this disclosure is depicted. Figure 10 A schematic diagram of the control system, which is configured as a control and monitoring system.

[0021] Figure 14 At least one example embodiment according to this disclosure is depicted. Figure 10 A schematic diagram of the control system configured to control the medical imaging system. Detailed Implementation

[0022] From the foregoing description, one will understand the embodiments described herein (which have been shown and described by way of example) and their many advantages, and it will be apparent that various changes can be made to the form, construction, and arrangement of the components without departing from the disclosed subject matter or sacrificing one or more of its advantages. In fact, the descriptive form of these embodiments is merely illustrative. These embodiments allow for various modifications and alternative forms, and the following claims are intended to cover and include such changes, and are not limited to the specific forms disclosed, but rather cover all modifications, equivalents, and alternatives falling within the spirit and scope of this disclosure.

[0023] Figure 1 An overview of a multimodal large language model (MLLM4TS) system 100 for time series analysis is illustrated. System 100 effectively leverages NLP and CV underlying models for time series analysis. This unified framework is configured to handle multivariate time series data (e.g., multichannel time series data) and perform one or more time series tasks (e.g., classification, anomaly detection, prediction, few-shot learning, imputation, clustering, etc.). MLLM4TS system 100 includes two or more encoding branches (e.g., time series tokenizer 110, plotting tokenizer 120, and text tokenizer 130), a pre-trained large language model (LLM) 140, and task modules 150 with at least one output layer (e.g., at least one task head).

[0024] Supervised learning is used to train the MLLM4TS system 100 for at least one specific task (e.g., classification, anomaly detection, etc.). Specifically, the MLLM4TS system 100 includes several components (e.g., a time-series word segmenter 110, a plotting projector 126, and a task module 150), as indicated by the flame icon, whose parameters can be updated during the training process. It also includes several components (e.g., an image encoder 124 and a text encoder 132), as indicated by the snowflake icon, whose parameters are frozen during the training process. Furthermore, as... Figure 1As shown, the pre-trained LLM140 is efficiently fine-tuned, thus illustrated using both flame and snowflake icons. At this point, the pre-trained LLM140 updates at least a subset of its parameters via an efficient fine-tuning method. Utilizing efficient fine-tuning, the pre-trained LLM140 benefits from not having to update all its parameters (which can be expensive) while still achieving high-quality task-specific results. For example, examples of efficient fine-tuning of the pre-trained LLM 140 include fine-tuning the location embeddings and normalization layers while keeping the self-attention layers and feedforward neural networks (FFNs) frozen. Specifically, during this training process, the MLLM4TS system 100 is trained using at least one suitable supervised loss function relevant to the specific task (e.g., cross-entropy loss for classification, reconstruction loss for anomaly detection, etc.) to update the parameters of predetermined components in the specific task.

[0025] like Figure 1 As shown, the MLLM4TS system 100 includes several different coding branches for encoding time-series information of different modalities. Each coding branch encodes time-series information captured in a specific format (e.g., numerical data, digital images, text data, etc.). Specifically, the MLLM4TS system 100 is configured to include at least a time-series coding branch and a visual coding branch. For example, in Figure 1 In this system, the MLLM4TS system 100 includes at least two coding branches (e.g., a time-series coding branch and a visual coding branch), and at least one additional coding branch (e.g., a language coding branch). That is, in Figure 1 In the example shown, the MLLM4TS system 100 includes three coding branches: a time-series coding branch, a visual coding branch, and a language coding branch. In this respect, the MLLM4TS system 100 is configured to handle time-series information across these three different modalities. Furthermore, Figure 2 , Figure 3 and Figure 4 Examples of these different coding branches are illustrated. The embeddings generated from these three coding branches are then fed into a pre-trained LLM 140, which serves as the central pivot of the MLLM4TS system 100.

[0026] Figure 2An example of a time series encoding branch according to an exemplary embodiment is illustrated. Encoding time series data into comma-separated numeric strings enables LLM to achieve reasonable zero-shot prediction performance. However, filling the modal gap between time series data and natural language often requires more sophisticated embedding techniques to achieve competitive performance. Recognizing this problem, the MLLM4TS system 100 includes a time series encoding branch comprising a time series tokenizer 110 with three components: a normalization component 112, a data patching component 114, and an embedding component 116. As specified and selected based on the application, the time series tokenizer is jointly trained with the MLLM4TS system 100 during the training process using supervised learning for at least one task (e.g., classification, anomaly detection, etc.).

[0027] like Figure 2 As shown, normalization component 112 receives multivariate time series data. Normalization component 112 normalizes the multivariate time series data. As a non-limiting example, normalization component 112 includes (i) standardization, which transforms the data to have zero mean and unit variance, (ii) min-max normalization, which transforms the data features to [0, 1], or (iii) a suitable normalization module. For example, refer to... Figure 1 As an example, normalization component 112 normalizes the time-series data for each channel. For example, in Figure 1 In this process, the normalization component 112 is configured to (i) normalize the first speed data, "speed Y" data, of the first channel; (ii) normalize the "leading vehicle distance" data of the second channel; and (iii) normalize the second speed data, "speed X" data, of the third channel. Next, for local semantic information extraction, after normalizing the time-series data for each channel, the time-series segmenter 110 uses data patching components 114 to extract the patched data.

[0028] Data patching component 114 receives normalized time-series data and generates data patches (e.g., segments) of the normalized time-series data. For example, data patching component 114 generates data patches by segmenting the normalized time-series data based on a given patch size and a given stride parameter. For example, data patching component 114 is configured to generate a set of data patches for multivariate time-series data. For ease of explanation, as a non-limiting example, data patching component 114 receives normalized multivariate time-series data, which includes (i) a sample of normalized time-series data comprising [y1, y2, y3, y4, y5, y6, y7, y8, y9] of a first channel, where y t(ii) a normalized numerical data (e.g., the speed of the first vehicle) y at timestamp t, and a sample of normalized time-series data comprising [d1, d2, d3, d4, d5, d6, d7, d8, d9] from the second channel, where d t (iii) represents the normalized numerical data (e.g., the distance between the first and second vehicles) d at timestamp t, and (iii) a sample of normalized time-series data comprising [x1, x2, x3, x4, x5, x6, x7, x8, x9] of the third channel, where x t The normalized numerical data (e.g., the speed of the second vehicle) x represents the data at timestamp t. In this example, given a predetermined patch size and a predetermined stride (e.g., patch size = 6, stride = 3), the data patch component 114 generates data patches including (i) a first data patch comprising [y1, y2, y3, y4, y5, y6], [d1, d2, d3, d4, d5, d6] and [x1, x2, x3, x4, x5, x6], and (ii) a second data patch comprising [y4, y5, y6, y7, y8, y9], [d4, d5, d6, d7, d8, d9] and [x4, x5, x6, x7, x8, x9]. Given the same training time and processing unit memory (e.g., graphics processing unit (GPU) memory), this patching technique enables pre-trained LLM 140 to utilize data patches to analyze the historical order of longer time series data while reducing information redundancy.

[0029] Additionally, the time series tokenizer 110 includes an embedding component 116 configured to generate time series embeddings using data patches (i.e., tokens of the time series data). The embedding component 116 includes an embedding process that fills in the domain gaps between the numerical time series data and the textual language data. As an example, the time series tokenizer 110, for its simplicity and effectiveness, includes a one-dimensional (1D) convolutional layer as the embedding component 116. Specifically, the embedding component 116 is configured to transform normalized multivariate time series data associated with several data patches into a single univariate time series, and to generate time series embeddings compatible with the input requirements of a pre-trained LLM 140. For example, as a non-limiting example, for ease of explanation, the embedded component 116 is configured to receive a first data patch having a normalized multivariate time series including [y1, y2, y3, y4, y5, y6], [d1, d2, d3, d4, d5, d6] and [x1, x2, x3, x4, x5, x6], and via concatenation, to generate a single univariate time series including [y1, y2, y3, y4, y5, y6, d1, d2, d3, d4, d5, d6, x1, x2, x3, x4, x5, x6]. Furthermore, as another example, the embedding component 116 is configured to receive a second data patch having a normalized multivariate time series including [y4, y5, y6, y7, y8, y9], [d4, d5, d6, d7, d8, d9], and [x4, x5, x6, x7, x8, x9], and via concatenation, generate a single univariate time series including [y4, y5, y6, y7, y8, y9, d4, d5, d6, d7, d8, d9, x4, x5, x6, x7, x8, x9]. Each univariate time series is then processed by a 1D convolutional layer that generates a time series embedding and projects the time series embedding into the dimension space required by the pre-trained LLM 140.

[0030] As discussed above, the time series coding branch enables the pre-trained LLM 140 to process data including numerical data (e.g., numerical values, such as...). Figure 1The diagnostic data values ​​shown are time-series data. However, the time-series coding branch does not explicitly capture cross-channel dependencies. As a technical solution to this problem, the MLLM4TS system 100 is configured to at least generate, acquire, and / or receive several visual representations of multivariate time-series data, and provide this information to a pre-trained LLM 140 to capture cross-channel dependencies. At this point, the multivariate time-series data is generated as a set of visual representations, such as a set of graphical representations, so that cross-channel dependencies can be visually captured and provided to the pre-trained LLM140. For example, in Figure 1 In this configuration, the MLLM4TS system 100 is set to generate line plots for each channel of multivariate time series data. For example... Figure 1 As shown, the set of line plots (e.g., three line plots) is combined into a single digital image, which is then fed into the visual encoding branch. This visual encoding branch enables the pre-trained LLM 140 to extract the global context and cross-channel dependencies of multivariate time series data, as these trends and cross-channel relationships are often more readily apparent in the visual representation of multivariate time series data. By providing this supplementary visual information to the multivariate time series data, the pre-trained LLM 140 is better able to process and learn the fundamental structures and relationships across the different channels of the time series data.

[0031] Figure 3 An example of a visual coding branch according to an exemplary embodiment is illustrated. In this example, the visual coding branch includes an image patch assembly 122, a pre-trained image encoder 124, and a drawing projector 126. Alternatively, the visual coding branch may include a pre-trained image encoder 124 and a drawing projector 126. More specifically, the image patch assembly 122 is configured to receive a single digital image on which a set of line drawings is displayed. The image patch assembly 122 is configured to generate a predetermined number of image patches using the digital image. Each image patch relates to a different portion of the digital image (e.g., a different pixel region). As a non-limiting example, a particular patch may include at least one specific portion of a particular line drawing.

[0032] Depending on whether image patch component 122 is used, the pre-trained image encoder 124 is configured to receive either the digital image itself or an image patch of the digital image. If image patch component 122 is not used, the pre-trained image encoder 124 is then configured to generate an image embedding using pixels of the digital image having a set of graphical representations. Alternatively, if image patch component 122 is used, the pre-trained image encoder 124 is then configured to generate an image embedding using pixels of an image patch of the digital image having a set of graphical representations.

[0033] Image encoder 124 includes an image encoding network of a pre-trained visual language model (VLM) or a suitable pre-trained image encoder. For example, image encoder 124 includes an image encoder of a contrastive language-image pre-trained (CLIP) model (e.g., the ViT-L / 14 CLIP model). In this respect, since the CLIP model is not pre-trained on time-series images, and the CLIP model is frozen during training (as per instructions), this approach is more efficient. Figure 3 (As indicated by the snowflake icon in the image), the MLLM4TS system 100 integrates efficient tuning techniques within CLIP, such as BitFit, LoRA, or similar tuning techniques, thereby allowing CLIP models to better adapt to the domain-specific characteristics of time-series data while maintaining training efficiency.

[0034] Furthermore, the visual encoding branch includes a drawing projector 126. For example, in Figure 3 In the drawing projector 126, a multilayer perceptron (MLP) is included. The MLP is trained during the training process, such as by... Figure 3 The flame icon shown on the drawing projector 126 indicates this. The MLP is used as a visual word segmenter, which is configured to map the image features of the image embeddings into a language embedding space. The drawing projector 126 is configured to at least (i) receive the image embeddings of the output dimension from the image encoder 124, and (ii) generate image embeddings of the appropriate dimension for the pre-trained LLM140 using the image embeddings of the output dimension from the image encoder 124.

[0035] As previously described, in addition to the time-series coding branch and the vision coding branch, the MLLM4TS system 100 is configured to include one or more additional branches. These additional branches are configured to supplement and enhance the multivariate time-series information (e.g., time-series embeddings and image embeddings) input to the pre-trained LLM 140. The one or more additional branches may include a language coding branch, an audio coding branch, a coding branch of the appropriate type for downstream applications, or any number and combination of the above.

[0036] refer to Figure 1As an example, the MLLM4TS system 100 includes a language encoding branch as an additional encoding branch to the time series encoding branch and the vision encoding branch. The language encoding branch is configured to augment multivariate time series data with relevant contextual information. For example, the language encoding branch is configured to receive text data. The text data is associated with or corresponds to the multivariate time series data. Regarding the multivariate time series data, the text data can provide background information, contextual information, relevant information, or any amount and combination of the above. Through the language encoding branch, the text data helps guide the pre-trained LLM 140 to perform specific tasks more effectively.

[0037] Text data may include annotation data of multivariate time series data, descriptions of multivariate time series data, at least one task instruction regarding multivariate time series data, key statistics of multivariate time series data in a language prompt format, other data related to the time series data, or any quantity and combination of the above. Text data may describe or relate to one or more visual representations of the multivariate time series data (e.g., graphical representations such as line plots). Text data may relate to one, some, or all channels of the multivariate time series data. Additionally or alternatively, as an example, text data may incorporate statistical data into the prompt. By easily incorporating computable statistical data into the prompt, the pre-trained LLM 140 is able to retain its ability to capture more complex patterns and relationships within multivariate time series data. At this point, the prompt is initialized using a text segmenter 130, thereby freezing its parameters during training and adoption (as per...). Figure 4 (As indicated by the snowflake icon in the image). This approach allows for increased capacity of the text encoder 132 without compromising efficiency, enabling the pre-trained LLM 140 to more effectively adapt to the nuances of text data.

[0038] Figure 4 An example of a language encoding branch according to an exemplary embodiment is illustrated. In this example, the language encoding branch includes a text segmenter 130, which is configured to receive text data as input. As an example, the text segmenter 130 includes a text encoder 132. The text encoder 132 is configured to receive text data. The text encoder 132 is configured to generate text embeddings using the text data. For example, the text encoder 132 may include a generative pre-trained transform (GPT), a bidirectional encoder representation (BERT) from a transformer, a large language model meta-AI (LLAMA), or a suitable text encoding network. The text embeddings are then fed into a pre-trained LLM 140.

[0039] refer to Figure 1The pre-trained LLM 140 serves as a central pivot across different modalities. In this example, the pre-trained LLM 140 processes various embeddings across different modalities. For example, the pre-trained LLM 140 can include GPT2, LLAMA3, or a suitable pre-trained language model. Figure 1 In this embodiment, when the MLLM4TS system 100 includes a time-series coding branch, a visual coding branch, and a language coding branch, the pre-trained LLM 140 is configured to generate semantic embeddings using time-series embeddings, image embeddings, and text embeddings. Alternatively, in another embodiment, when the MLLM4TS system 100 includes a time-series coding branch and a visual coding branch, the pre-trained LLM 140 is configured to generate semantic embeddings using time-series embeddings and image embeddings. In each of these examples, the pre-trained LLM 140 is configured to receive concatenated data as input, wherein the concatenated data is a concatenation of various embeddings (e.g., time-series embeddings, image embeddings, text embeddings, etc.) received from the various coding branches.

[0040] As previously described, a pre-trained LLM 140 generates semantic embeddings. Semantic embeddings are high-dimensional vector representations that capture the fundamental meaning and relationships within data. These semantic embeddings are generated by the pre-trained LLM 140 from different encoding branches, enabling task modules 150 (e.g., one or more task heads) to encode temporal dependencies and structural patterns in a way that allows them to be processed and interpreted efficiently. These semantic embeddings serve as compact and information-rich representations of the input data (e.g., various modalities of time-series information), making them useful for at least one task (e.g., classification, anomaly detection, clustering, etc.). The MLLM4TS system 100 utilizes the pre-trained LLM 140 to generate semantic embeddings that encode contextual and relational information from various modalities of multivariate time-series data (e.g., multi-channel time-series data), thereby facilitating improved generalization and transfer learning. Semantic embeddings are provided in a structured feature space, where more similar patterns or sequences are mapped to be closer to each other, and less similar patterns are mapped to be farther apart, enabling the pre-trained LLM 140 to identify trends, anomalies, and correlations across different time series datasets.

[0041] Furthermore, given a large number of semantic embeddings (or output lexical units) generated by the pre-trained LLM 140, the MLLM4TS system 100 would require an excessively large prediction head to directly flatten all the semantic embeddings (or output lexical units) of the pre-trained LLM 140 for predictions involving a specific task. Recognizing this technical problem and in order to improve efficiency, the MLLM4TS system 100 applies max or average pooling across the semantic embeddings (or output lexical units) output from the pre-trained LLM 140 to merge these semantic embeddings into a single lexical unit, which is then linearly projected onto each task head of the task module 150 to generate the output data or final prediction for that given task.

[0042] Additionally, such as Figure 1 As shown, the MLLM4TS system 100 includes a task module 150 having at least one output layer (e.g., at least one task head). Each task head may include one or more artificial neural network layers. The task head is configured to perform a specific task in the time series domain using semantic embeddings and to generate output data based on that specific task. The task module 150 is configured to include one or more task heads based on one or more specific tasks for a given application. For example, task module 150 may include a classification head for performing a classification task. Task module 150 may include a prediction head for performing a prediction task. Furthermore, for example, few-shot learning or zero-shot learning may be applied to the task heads (e.g., classification head, prediction head, etc.). Task module 150 may include an anomaly detection head for performing an anomaly detection task. Task module 150 may include an interpolation task head for performing an interpolation task. Task module 150 may include a clustering head for performing a clustering task. In this respect, the MLLM4TS system 100 includes task module 150, which is configured to perform one or more specific tasks for a given application.

[0043] Furthermore, for illustrative and explanatory purposes, Figure 1 The diagram also illustrates a non-limiting data example for the MLLM4TS system 100. The MLLM4TS system 100 processes multivariate time series data. Multivariate time series data includes multichannel time series data. For example, in... Figure 1 In this process, the MLLM4TS system 100 receives multivariate time-series data (e.g., numerical data about time), which includes (i) a first channel containing time-related data on the speed of a first vehicle, (ii) a second channel containing time-related data on the leading distance of the first vehicle relative to a second vehicle, and (iii) a third channel containing time-related data on the speed of the second vehicle. Furthermore, the MLLM4TS system 100 generates a digital image incorporating this multivariate time-series data (e.g., a line plot). For example, as... Figure 1As shown, the MLLM4TS system 100 generates (i) a first line plot of the speed data of a first vehicle over time, (ii) a second line plot of the distance to the leading vehicle over time, and (iii) a third line plot of the speed data of a second vehicle. The first line plot relates to the first channel of the time series data. The second line plot relates to the second channel of the time series data. The third line plot corresponds to the third channel of the time series data. Furthermore, in Figure 1 In the example shown, the digital image displays three line plots in a 2×2 grid format, where (i) the first line plot is drawn in a first color and positioned in the first row and first column of the 2×2 grid, (ii) the second line plot is drawn in a second color and positioned in the first row and second column of the 2×2 grid, and (iii) the third line plot is drawn in a third color and positioned in the second row and first column of the 2×2 grid, and (iv) blank space is positioned in the second row and second column of the 2×2 grid. At this point, the blank space indicates that the multivariate time series data does not include the fourth channel and / or the fourth channel time series data. Figure 1 In the example shown, the MLLM4TS system 100 also receives text data relating to time series data. In this case, the text data includes annotations indicating that the time series data corresponds to “sharp braking points in a roundabout,” thereby providing contextual information relating to a set of graphical representations of multivariate time series data (e.g., numerical data) and digital images.

[0044] In addition, regarding Figure 1 The non-limiting data example shown illustrates that time-series word segmenter 110 generates time-series embeddings based on time-series data. Plotting word segmenter 120 generates image embeddings based on pixels of image patches from digital images. Text word segmenter 130 generates text embeddings based on text data. Concatenated data is generated by concatenating these multimodal embeddings (e.g., time-series embeddings, image embeddings, and text embeddings). A pre-trained LLM 140 receives the concatenated data as input and generates semantic embeddings as output based on the concatenated data. In this particular example, task module 150 includes a classification head. The classification head is configured to generate classification data based on the semantic embeddings as output data for a classification task. In this non-limiting example, the classification head predicts that the semantic embedding belongs to the "Longitudinal Abrupt Fault" class among several classes (e.g., various fault classes). Additionally or alternatively, task module 150 includes an anomaly detection head configured to classify semantic embeddings associated with various modalities of multi-channel time-series information (e.g., time-series data, digital images with graphical representations of time-series data, and textual data with contextual information of time-series data) and generate output data indicating whether the multi-channel time-series information is anomalous or non-anomalous (i.e., normal). For example, in Figure 1In this process, the anomaly detection head determines that the multi-channel time series information is "abnormal" and generates an output indicating that the anomaly was detected. As demonstrated by this non-limiting data example, the advantage of the MLLM4TS system 100 is that it can perform at least one specific task based on multimodal time series information with both local and global perspectives of multi-channel time series data.

[0045] Figure 5 , Figure 6 , Figure 7 and Figure 8 Other non-limiting examples of various digital images are illustrated, which can be generated based on other non-limiting examples of multivariate time series data. The plotting word segmenter 170 is not limited to processing... Figure 1 , Figure 5 , Figure 6 , Figure 7 and Figure 8 The visual representation presented may include a specific version and layout, or it may include other visual representations deemed suitable for multivariate time series data based on several factors (e.g., the type of time series data, the number of channels, a specific application, a specific task, etc.). As previously described, the plotting word segmenter 120 is configured to receive at least one digital image that displays a visual representation (e.g., a graphical representation) of the multivariate time series data. The multivariate time series data may include or correspond to a given number of channels for a given application. For example, in Figure 1 In the example shown, the multivariate time series data includes a first diagnostic data set in a first channel, a second diagnostic data set in a second channel, and a third diagnostic data set in a third channel. As other non-limiting examples, Figure 5 , Figure 6 , Figure 7 and Figure 8 Each of the examples illustrates a digital image generated as input to the visual encoding branch. Each digital image comprises four graphical representations of various time-series data across four channels. In each of these examples, for a given channel, the time-series data is graphically represented as a plot (e.g., a line plot). Each line plot is plotted against the x-axis representing time and the y-axis representing numerical data (e.g., values ​​such as velocity, distance, diagnostic data, sensor data, etc.).

[0046] In each of these examples, four line plots from four channels are displayed together in a single digital image (or common digital image) including pixels. Each example presents a different version (e.g., layout, form, orientation, color, etc.) in which the line plots are displayed in the digital image for processing by the plotting word segmenter 120. For example, Figure 5An example of a digital image 500 is illustrated, in which four line plots are presented in a 2×2 grid format. Specifically, the digital image 500 includes (i) a first line plot of time-series data from a first channel, which is placed in the first row and first column of the grid; (ii) a second line plot of time-series data from a second channel, which is placed in the first row and second column of the grid; (iii) a third line plot of time-series data from a third channel, which is placed in the second row and first column of the grid; and (iv) a fourth line plot of time-series data from a fourth channel, which is placed in the second row and second column of the grid.

[0047] In addition, Figure 5 In the image, digital image 500 displays each line plot, where the x-axis (e.g., time data) extends parallel to the top / bottom edge of digital image 500, and the y-axis (e.g., numerical data) extends parallel to the right / left edge of digital image 500. Additionally, in Figure 5 In the middle, four lines are drawn using the same color. Meanwhile, in... Figure 5 In this context, additional information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed using a different color compared to the four-line plot. For example, in... Figure 5 In this example, each element in the line drawing is displayed in a specific color (e.g., orange or another chosen color), while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another chosen color). However, in other examples, depending on the given application, this other information and the line drawing may be the same color.

[0048] As another example, Figure 6 An example of a digital image 600 is illustrated, where, regarding vertical orientation, four lines are drawn in a vertically stacked manner. (See example image.) Figure 6As shown, the digital image 600 displays a stack, with the top of the stack positioned at the top portion of the digital image 600 and the bottom of the stack positioned at the bottom portion of the digital image 600. In this case, the digital image 600 displays a vertical stack, with the first line plot at the top of the stack and the fourth line plot at the bottom, such that the first line plot of the first channel, the second line plot of the second channel, the third line plot of the third channel, and the fourth line plot of the fourth channel are arranged in this order. Alternatively, the digital image can display a vertical stack, with the fourth line plot at the top of the stack and the first line plot at the bottom, such that the fourth line plot of the fourth channel, the third line plot of the third channel, the second line plot of the second channel, and the first line plot of the first channel are arranged in this order. Furthermore, in each of these cases, the x-axis (e.g., time data) extends parallel to the top / bottom edge of the digital image 600, and the y-axis (e.g., numerical data) extends parallel to the right / left edge of the digital image 600. Additionally, in Figure 6 In the middle, four lines are drawn using the same color. Meanwhile, in... Figure 6 In this context, additional information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed using a different color compared to the four-line plot. For example, in... Figure 6 In this example, each element in the line drawing is displayed in a specific color (e.g., orange or another chosen color), while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another chosen color). However, in other examples, depending on the given application, this other information and the line drawing may be the same color.

[0049] As an example, Figure 7 An example of a digital image 700 is illustrated, in which four line plots are displayed in a horizontally stacked manner with respect to vertical orientation. The digital image 700 includes four line plots arranged in a stacked arrangement, which is horizontally oriented in a vertical view. Specifically, in this horizontally stacked arrangement, the x-axis (e.g., time data) is parallel to the left / right edges of the digital image 700, and the y-axis (e.g., numerical data) is parallel to the top / bottom edges of the digital image 700. Figure 7 In this line drawing, the line plot is oriented such that the x-axis (e.g., time) increases in the direction from the top to the bottom of the digital image 700. Alternatively, the line plot can be illustrated such that the x-axis (e.g., time) increases in the direction from the bottom to the top of the page. Furthermore, in Figure 7In this method, line plots are horizontally stacked with orientation, such that the first line plot of the first channel, the second line plot of the second channel, the third line plot of the third channel, and the fourth line plot of the fourth channel are stacked in the order from left to right of the digital image 700. Alternatively, the line plots can be oriented in the stack such that the fourth line plot of the fourth channel, the third line plot of the third channel, the second line plot of the second channel, and the first line plot of the first channel are stacked in the order from left to right of the digital image. Additionally, in Figure 7 In the middle, four lines are drawn using the same color. Meanwhile, in... Figure 7 In this context, additional information (e.g., y-axis information, x-axis information, plot labels, etc.) is displayed using a different color compared to the four-line plot. For example, in... Figure 7 In this example, each element in the line drawing is displayed in a specific color (e.g., orange or another chosen color), while other information (e.g., y-axis information, x-axis information, and labels) is displayed in a different color (e.g., black or another chosen color). However, in other examples, depending on the given application, this other information and the line drawing may be the same color.

[0050] Figure 8 Another example of a digital image 800 is illustrated, in which four lines are drawn to correspond with... Figure 5 The digital image is displayed in a similar 2×2 grid format to 500. However, Figure 8 Digital Image 800 and Figure 5 The digital image 500 differs from the digital image 800 because the digital image 800 includes color-coded line drawings, such that the first line drawing of the first channel is a first color (e.g., blue or another selected color), the second line drawing of the second channel is a second color (e.g., green or another selected color), the third line drawing of the third channel is a third color (e.g., red or another selected color), and the fourth line drawing of the fourth channel is a fourth color (e.g., purple or another selected color). As demonstrated by this example, each line drawing can be displayed in the selected color via a color-coding scheme. Meanwhile, in Figure 8 In this context, additional information (e.g., y-axis information, x-axis information, labels, etc.) is displayed using colors different from those used in the four-line plot. For example, in... Figure 8In this example, each line plot is displayed using a selected color (e.g., red, blue, green, and purple) according to a color-coding scheme, while other information (e.g., y-axis information, x-axis information, and labels) is displayed using a different color (e.g., black or another selected color). However, in other examples, variance may exist in the color coding of the line plot and other information, depending on the given application. For example, one or more line plots may be color-coded using one or more specific colors, while one or more line plots in other line plots may be color-coded using one or more other specific colors. Furthermore, in Figure 8 In the image, digital image 800 displays each line plot, where the x-axis (e.g., time data) extends parallel to the top / bottom edge of digital image 800, and the y-axis (e.g., numerical data) extends parallel to the right / left edge of digital image 800.

[0051] Figure 9 This is a block diagram of an example of a system 900 including an MLLM4TS system 100. System 900 includes at least a processing system 902. Processing system 902 includes at least one processing device. For example, processing system 902 may include an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any processing technology, or any number and combination of the above. Processing system 902 is operable to provide the functionality described herein.

[0052] System 900 includes at least one sensor system 904. Sensor system 904 includes one or more sensors (e.g., speed sensors, radio radar, infrared sensors, image sensors, GPS sensors, audio sensors, motion sensors, etc.). In particular, sensor system 904 includes one or more sensors that provide multivariate time-series data. Sensor system 904 is operable to communicate with one or more other components of system 900 (e.g., processing system 902 and memory system 910). For example, sensor system 904 can provide sensor data (e.g., time-series data), which is then processed by processing system 902. Relative to one or more components of system 900, sensor system 904 is local, remote, or a combination of the above (e.g., partially local and partially remote). Upon receiving sensor data (e.g., multi-channel time-series data), processing system 902 is configured to process the sensor data (in combination with application 912, MLLM4TS system 100, machine learning (ML) data 914, other relevant data 916, or any number and combination of the above).

[0053] System 900 includes a memory system 910 operatively linked to a processing system 902. In this respect, the processing system 902 communicates data with the memory system 910. The memory system 910 includes at least one non-transitory computer-readable storage medium configured to store various types of data and provide access to such data, at least enabling the processing system 902 to perform the operations and functions disclosed herein. The memory system 910 includes a single memory device or multiple memory devices. The memory system 910 may include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable storage technology. For example, the memory system 910 may include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any number and combination of the foregoing. The memory system 910 includes computer-readable data that, when executed by the processing system 902, is configured to perform at least the functions disclosed herein. Computer-readable data may include instructions, code, routines, various related data, software technology, or any number and combination of the above.

[0054] The memory system 910 includes computer-readable data of an application program 912. The application program 912 is configured to perform the functions discussed in this disclosure with respect to the MLLM4TS system 100. For example, the application program 912 may relate to any number and combination of processes associated with the MLLM4TS system 100, including training, tuning, testing, verification, deployment, adoption, or the above. The application program 912 may also be configured to apply output data from the MLLM4TS system 100 (e.g., class data such as “longitudinal spurious failure class” or “anomaly” labels) to a given application and / or at least one downstream technology system.

[0055] Furthermore, the memory system 910 includes computer-readable data of the MLLM4TS system 100, which is configured to perform, for example, the functions described in this disclosure. Figure 1 , Figure 2 , Figure 3 and Figure 4The operations and functions discussed herein. Memory system 910 includes ML data 914, which includes various data (e.g., parameters, sensor data, time series data, digital images, text data, loss data, training data, channel data, etc.) relating to various aspects (e.g., training, tuning, fine-tuning, testing, verification, deployment, adoption, etc.) of one or more parts of MLLM4TS system 100, as described herein. Memory system 910 includes computer-readable data for other related data 916. Other related data 916 provides various data (e.g., operating system, etc.) that enables system 900 and / or processing system 902 to perform the functions discussed herein. Additionally, system 900 may include one or more I / O devices 906 (e.g., display devices, microphones, speakers, etc.) to receive input and / or provide output to one or more components of system 900.

[0056] Additionally, system 900 includes other functional modules 908, such as any suitable hardware, software, or combinations thereof, that assist or contribute to the functionality of system 900 and MLLM4TS system 100. For example, other functional modules 908 include communication technologies (e.g., wired communication technologies, wireless communication technologies, or combinations thereof) that enable components of system 900 to communicate with each other and / or with one or more other computing devices via at least one communication network. One or more other computing devices (not shown) may include mobile communication devices, smartphones, laptops, tablets, servers, cloud computing systems, etc.

[0057] Figure 10 A schematic diagram depicting the interaction between a computer-controlled machine 1000 and a control system 1002 according to another example embodiment is shown. The computer-controlled machine 1000 includes an actuator 1004 and a sensor 1006. The actuator 1004 may include one or more actuators, and the sensor 1006 may include one or more sensors. The sensor 1006 is configured to sense conditions of the computer-controlled machine 1000. The sensor 1006 may be configured to encode the sensed conditions into a sensor signal 1008 and transmit the sensor signal 1008 to the control system 1002. Non-limiting examples of the sensor 1006 include video, radio radar, LiDAR, ultrasonic sensors, image sensors, audio sensors, motion sensors, etc.

[0058] The control system 1002 is configured to receive sensor signals 1008 from the computer-controlled machine 1000. As described below, the control system 1002 may also be configured to calculate actuator control commands 1010 based on the sensor signals and transmit the actuator control commands 1010 to the actuators 1004 of the computer-controlled machine 1000.

[0059] like Figure 10 As shown, the control system 1002 includes a receiving unit 1012. The receiving unit 1012 can be configured to receive sensor signals 1008 from sensor 1006 and transform the sensor signals 1008 into input signals x. In an alternative embodiment, the sensor signals 1008 are received directly as input signals x, without a receiving unit 1012. Each input signal x may be a portion of each sensor signal 1008. The receiving unit 1012 can be configured to process each sensor signal 1008 to generate each input signal x. The input signals x may include data corresponding to a digital image recorded by sensor 1006.

[0060] The control system 1002 includes a classifier 1014. In this example, the classifier 1014 includes an MLLM4TS system 100 with a task module 150, the task module 150 including at least one classification head as a task head. The classifier 1014 can be configured to classify an input signal x into one or more labels using an ML algorithm via the MLLM4TS system 100. The classifier 1014 is configured to be parameterized by parameters θ. The parameters θ can be stored in and provided by a non-volatile storage device 1016. The classifier 1014 is configured to determine an output signal y based on the input signal x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 1014 can transmit the output signal y to a conversion unit 1018. The conversion unit 1018 is configured to convert the output signal y into an actuator control command 1010. The control system 1002 is configured to transmit actuator control command 1010 to actuator 1004, which is configured to actuate computer-controlled machine 1000 in response to actuator control command 1010. In some embodiments, actuator 1004 is configured to actuate computer-controlled machine 1000 directly based on output signal y.

[0061] Upon receiving an actuator control command 1010, actuator 1004 is configured to perform an action corresponding to the associated actuator control command 1010. Actuator 1004 may include control logic configured to transform the actuator control command 1010 into a second actuator control command for controlling actuator 1004. In one or more embodiments, instead of or attached to the actuator, actuator control command 1010 may be used to control a display.

[0062] In some embodiments, the control system 1002 includes a sensor 1006, replacing or attached to the computer-controlled machine 1000 including the sensor 1006. The control system 1002 may also include an actuator 1004, replacing or attached to the computer-controlled machine 1000 including the actuator 1004. Figure 10 As shown, the control system 1002 also includes a processor 1020 and a memory 1022. The processor 1020 may include one or more processors. The memory 1022 may include one or more non-transitory memory devices. The classifier 1014 of one or more embodiments may be implemented by the control system 1002, which includes a non-volatile storage device 1016, a processor 1020, and a memory 1022.

[0063] The non-volatile storage device 1016 may include one or more persistent data storage devices, such as hard disks, optical drives, tape drives, non-volatile solid-state devices, cloud storage devices, or any other device capable of permanently storing information. The processor 1020 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, graphics processing units, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 1022. Memory 1022 may include a single memory device or multiple memory devices, including but not limited to RAM, ROM, volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0064] Processor 1020 is configured to read into memory 1022 and execute computer-executable instructions residing in non-volatile storage device 1016 and embodying one or more ML algorithms and / or methods of one or more embodiments. Non-volatile storage device 1016 may include one or more operating systems and applications. Non-volatile storage device 1016 may store programs compiled and / or interpreted from computer programs created using various programming languages ​​and / or techniques, including but not limited to (and used individually or in combination) Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL.

[0065] When executed by processor 1020, computer-executable instructions of non-volatile storage device 1016 may cause control system 1002 to implement one or more ML algorithms and / or methods to employ classifier 1014 as disclosed herein. Non-volatile storage device 1016 may also include ML data (including model parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0066] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in various different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, for causing a processor to execute aspects of one or more embodiments. The inherently non-transitory computer-readable storage medium can include tangible media that are volatile and non-volatile, removable and non-removable, implemented in any method or technology, for storing information such as computer-readable instructions, data structures, program modules, or other data. The computer-readable storage medium can also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technologies, portable optical disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. The computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded via a network to an external computer or external storage device.

[0067] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device that functions in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture comprising instructions that implement the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, consistent with one or more embodiments, the functions, actions, and / or operations specified in a flowchart or diagram can be reordered, processed sequentially, and / or processed concurrently. Furthermore, any flowchart or diagram may include more or fewer nodes or blocks than illustrated consistent with one or more embodiments. Additionally, the process, method, or algorithm may be embodied, in whole or in part, using suitable hardware components such as ASICs, FPGAs, state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0068] Figure 11A schematic diagram of a control system 1002 is depicted, configured to control a vehicle 1100, which may be at least partially autonomous or partially autonomous. The vehicle 1100 includes actuators 1004 and sensors 1006. Sensors 1006 may include one or more video sensors, cameras, radio radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., Global Positioning System). One or more of these specific sensors may be integrated into the vehicle 1100. Alternatively or additionally to the one or more specific sensors identified above, sensor 1006 may include a software module configured to determine the state of actuator 1004 upon execution. A non-limiting example of the software module includes a weather information software module configured to determine the current or future weather conditions near the vehicle 1100 or another location.

[0069] The classifier 1014 of the control system 1002 of vehicle 1100 can be configured to classify objects near vehicle 1100 based on an input signal x. In such an embodiment, the output signal y may include information on the classification or characterization of objects near vehicle 1100. An actuator control command 1010 can be determined based on this information. Based on the classification provided by the classifier 1014, the actuator control command 1010 can be used to navigate vehicle 1100 and avoid collisions.

[0070] In some embodiments, vehicle 1100 is at least partially autonomous or fully autonomous. Actuator 1004 may be embodied in brakes, propulsion system, engine, drivetrain, steering of vehicle 1100, etc. Actuator control command 1010 may be determined to control actuator 1004 so that vehicle 1100 avoids collisions with detected objects. Detected objects, such as pedestrians, trees, any suitable labels, etc., may also be identified and classified based on what classifier 1014 deems most likely to be detected. Actuator control command 1010 may be determined based on the classification of objects from digital images generated via sensor 1006.

[0071] In some embodiments, vehicle 1100 is at least a partially autonomous robot. Vehicle 1100 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, walking, or other movement. The mobile robot may be at least a partially autonomous lawnmower or at least a partially autonomous cleaning robot. In such embodiments, actuator control commands 1010 may be determined so that the propulsion unit, steering unit, and / or braking unit of the mobile robot can be controlled, allowing the mobile robot to navigate and / or avoid collisions with objects based on classifications provided by classifier 1014.

[0072] In some embodiments, vehicle 1100 is at least partially autonomous in the form of a gardening robot. In such embodiments, vehicle 1100 may use optical sensors as sensor 1006 to determine the state of plants in the environment adjacent to vehicle 1100. Actuator 1004 may be a nozzle configured to spray chemicals. Based on the identified species and / or identified state of the plants via classifier 1014, actuator control command 1010 may be determined to cause actuator 1004 to spray the plants with an appropriate amount of appropriate chemicals.

[0073] Figure 12 A schematic diagram of a control system 1002 is depicted, which is configured to control a control system 1200 (e.g., a manufacturing machine). The system 1200 may include a stamping cutter, cutting machine, gun drill, manufacturing tool, or the like for manufacturing the system 1202 (such as part of a production line). The control system 1002 may be configured to control an actuator 1004, which is configured to control the control system 1200 (e.g., the manufacturing machine).

[0074] The sensor 1006 of system 1200 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more objects associated with the manufactured product 1204. A classifier 1014 may be configured to make determinations based on one or more of the captured attributes. An actuator 1004 may be configured to control system 1200 (e.g., the manufacturing machine) regarding the subsequent state of the product 1204 based on the determined manufacturing state of the product 1204. The actuator 1004 may also be configured to control system 1200 (e.g., the manufacturing machine) regarding the subsequent state of the product 1206 of system 1200 (e.g., the manufacturing machine) based on the determined state of the product 1204.

[0075] Figure 13 This is a diagram of a control system 1002 configured to control a monitoring system 1300 (e.g., a security system). The monitoring system 1300 can be configured to physically control access through a door 1302. A sensor 1006 can be configured to detect scenes related to determining whether to grant access. The sensor 1006 can be an optical sensor configured to generate and transmit image and / or video data. Such image and / or video data can be used by the control system 1002 to detect and classify objects (e.g., humans, animals, bicycles, weapons, trash cans, etc.) that are within the sensing area of ​​the sensor 1006 near the door 1302.

[0076] Additionally, the control system 1002 can be configured to generate an actuator control command 1010 in response to the classification of one or more objects via image and / or video data from the classifier 1014. The control system 1002 is configured to transmit the actuator control command 1010 to the actuator 1004. In this embodiment, the actuator 1004 is configured to lock or unlock the door 1302 in response to the actuator control command 1010. In some embodiments, non-physical logical access control is also possible.

[0077] The monitoring system 1300 can also be a surveillance system. In such an embodiment, the sensor 1006 includes at least one image sensor or camera configured to detect the monitored scene, and the control system 1002 is configured to control the display 1304. The classifier 1014 is configured to determine the classification of the scene, for example, whether the scene detected by the sensor 1006 is suspicious or not suspicious. The control system 1002 is configured to transmit an actuator control command 1010 to the display 1304 in response to the classification. The display 1304 can be configured to adjust the displayed content in response to the actuator control command 1010. For example, the display 1304 can highlight objects that the classifier 1014 considers suspicious.

[0078] Figure 14 A schematic diagram of a control system 1002 is depicted, configured to control an imaging system 1400, such as a magnetic resonance imaging (MRI) device, an X-ray imaging device, or an ultrasound device. A sensor 1006 may be, for example, an imaging sensor. A classifier 1014 may be configured to determine the classification of all sensed images or portions of sensed images. An actuator control command 1010 is selected based on the classification obtained from the classifier 1014. For example, the classifier 1014 may interpret regions of a digital image as anomalous or non-anomalous. In this case, the actuator control command 1010 can be selected to cause the display 1402 to display the digital image and highlight potentially anomalous regions via the classification provided by the classifier 1014.

[0079] As described in this disclosure, embodiments include several advantageous features and benefits. For example, each embodiment includes an MLLM4TS system 100 configured to perform specific tasks involving system forensics via a machine learning framework that utilizes various modalities of time-series information. Furthermore, the MLLM4TS system 100 is configured to handle multi-channel time-series data while performing at least one specific task. Additionally, the MLLM4TS system 100 provides a unified framework that leverages a multimodal underlying model for time-series analysis. In example embodiments, the MLLM4TS system 100 includes three encoding branches, such as a time-series encoding branch, a visual encoding branch, and a language encoding branch. Specifically, the MLLM4TS system 100 is configured to segment numerical time-series data into patches and generate time-series embeddings in a channel-independent manner to preserve local semantic information. Furthermore, the MLLM4TS system 100 transforms time-series data into graphical representations (e.g., line plots) to capture global context and cross-channel dependencies. The MLLM4TS system 100 is configured to incorporate a graphical representation of multi-channel time-series data into a single digital image. The digital image can be displayed using a color-coding scheme to provide additional visual information leading to improved performance for a given task. The MLLM4TS system 100 includes at least a visual foundation model (e.g., an image encoder 124) to generate image embeddings based on the pixels of the single digital image. Furthermore, the MLLM4TS system 100 is configured to enrich the time-series data with relevant descriptions, task instructions, key statistics in the form of verbal prompts, any relevant textual data, or any amount or combination of the above. Additionally, the MLLM4TS system 100 employs a pre-trained LLM 140 as a central pivot to process the concatenated data of the resulting embeddings from these various modalities for a specific task. The advantage of the MLLM4TS system 100 lies in its unified framework, configurable to handle multi-channel multimodal time-series information and perform one or more different tasks in the time-series domain.

[0080] Furthermore, the above description is intended to be illustrative and not restrictive, and is provided in the context of a particular application and its requirements. Those skilled in the art will appreciate from the foregoing description that the invention can be implemented in various forms and that various embodiments can be implemented individually or in combination. Therefore, although embodiments of the invention have been described in conjunction with specific examples of the invention, the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments, and the true scope of the embodiments and / or methods of the invention is not limited to the embodiments shown and described, as various modifications will become apparent to those skilled in the art upon studying the drawings, specification, and the following claims. Additionally or alternatively, components and functions may be separated or combined in ways different from the various described embodiments, and may be described using different terms. These and other variations, modifications, additions, and improvements may fall within the scope of this disclosure as defined in the following claims.

Claims

1. A computer-implemented method for performing a task in a time-series domain, the computer-implemented method comprising: Receive time-series data including numerical data; Generate a digital image that displays a set of graphical representations of the time series data; A time series embedding is generated using the time series data via a word segmenter; An image embedding is generated using the pixels of the digital image via an image encoder; Semantic embeddings are generated, at least based on the time-series embeddings and the image embeddings, via a pre-trained Large Language Model (LLM); and The task is executed via at least one task header, and output data is generated for the task using the semantic embedding.

2. The computer-implemented method according to claim 1 further includes: Receive text data that provides context for the time series data; Text embeddings are generated using the text data via a pre-trained text encoder; as well as By concatenating the time series embedding, the image embedding, and the text embedding, concatenated data is generated. The pre-trained LLM uses the concatenated data to generate the semantic embedding.

3. The computer-implemented method according to claim 1, wherein: The time-series data includes a set of channels with diagnostic data; and The graphical representation set includes at least one line plot for each channel of the diagnostic data.

4. The computer-implemented method according to claim 1 further includes: Image patches for generating the digital image, The image encoder receives the image patch as input and uses the image patch to generate the image embedding.

5. The computer-implemented method according to claim 1, further comprising: Generate data patches for the time series data. The word segmenter uses the data patch to generate the time series embedding.

6. The computer-implemented method according to claim 5, further comprising: By linking the multivariate time series data associated with each data patch, a univariate time series is generated; as well as The time series embedding is generated using the univariate time series via a one-dimensional convolutional layer.

7. The computer-implemented method according to claim 1, wherein: The set of graphic representations includes at least a first-line drawing and a second-line drawing; The set of graphic representations is color-coded such that the first line drawing is displayed in a different color compared to the second line drawing; and The image embedding is generated based on the colors of the set of graphic representations.

8. The computer-implemented method according to claim 1, wherein: The set of graphical representations includes at least a first-line drawing and a second-line drawing; and The digital image displays the first line drawing and the second line in the same row or column.

9. The computer-implemented method of claim 1, wherein the task includes classification, anomaly detection, prediction, imputation, or clustering.

10. The computer-implemented method according to claim 1, further comprising: The actuator is controlled based on the output data.

11. A system comprising: One or more processors; One or more computer memories that communicate with the one or more processors, the one or more computer memories having computer-readable data stored thereon, the computer-readable data including instructions that, when executed by the one or more processors, cause the one or more processors to perform a method for performing a task in a time-series domain, the method comprising: Receive time-series data including numerical data; Generate a digital image that displays a set of graphical representations of the time series data; A time series embedding is generated using the time series data via a word segmenter; An image embedding is generated using the pixels of the digital image via an image encoder; Semantic embeddings are generated, at least based on the time-series embeddings and the image embeddings, via a pre-trained Large Language Model (LLM); and The task is executed via at least one task header, and output data is generated for the task using the semantic embedding.

12. The system of claim 11, wherein the method further comprises: Receive text data that provides context for the time series data; Text embeddings are generated using the text data via a pre-trained text encoder; as well as By concatenating the time series embedding, the image embedding, and the text embedding, concatenated data is generated. The pre-trained LLM uses the concatenated data to generate the semantic embedding.

13. The system according to claim 11, wherein: The time-series data includes a set of channels with diagnostic data; and The graphical representation set includes at least one line plot for each channel of the diagnostic data.

14. The system of claim 11, wherein the method further comprises: Image patches for generating the digital image, The image encoder receives the image patch as input and uses the image patch to generate the image embedding.

15. The system of claim 11, wherein the method further comprises: Generate data patches for the time series data. The word segmenter uses the data patch to generate the time series embedding.

16. The system of claim 15, wherein the method further comprises: By linking the multivariate time series data associated with each data patch, a univariate time series is generated; as well as The time series embedding is generated using the univariate time series via a one-dimensional convolutional layer.

17. The system according to claim 11, wherein: The set of graphic representations includes at least a first-line drawing and a second-line drawing; The set of graphic representations is color-coded such that the first line drawing is displayed in a different color compared to the second line drawing; and The image embedding is generated based on the colors of the set of graphic representations.

18. The system according to claim 11, wherein: The set of graphical representations includes at least a first-line drawing and a second-line drawing; and The digital image displays the first line drawing and the second line in the same row or column.

19. The system of claim 11, wherein the task includes classification, anomaly detection, prediction, imputation, or clustering.

20. The system of claim 11, further comprising: Actuator The actuator is controlled based at least on the output data.

Images