Time-series data forecasting using multimodal augmentation and fusion

Abstract

Systems and methods for time series forecasting using multimodal augmentation and fusion. Time series data and modality data can be decomposed into seasonal and trend representations using trend-season decomposition (110). Using an encoder-transformer model, time series data embeddings and modality data embeddings can be concatenated from the seasonal and trend representations (120) to obtain crossed representations. Using the encoder-transformer model, the modality data embeddings and the time series data embeddings can be processed separately (130) to obtain singular representations. The crossed representations and the singular representations can be augmented by joint trend-season decomposition (140) to obtain augmented seasonal data and augmented trend data.Using a decoder, augmented seasonal data and augmented trend data can be fused (150) to obtain fused augmented data. A corrective action can be performed (160) to correct predicted future events using a system with a prediction model trained on the fused augmented data.

Description

INFORMATION ABOUT RELATED REGISTRATIONS

[0001] This application claims priority over U.S. Preliminary Application No. 63 / 533,391, filed on August 18, 2023, and U.S. Preliminary Application No. 63 / 539,909, filed on September 22, 2023, and U.S. Patent Application No. 18 / 806,025, filed on August 15, 2024, which are hereby incorporated by reference in their entirety. BACKGROUND Technical area

[0002] The present invention relates to data prediction using artificial intelligence (AI) and in particular to time series data forecasting using multimodal augmentation and fusion. Description of the related technique

[0003] Artificial intelligence has made remarkable progress in predicting future events. For example, the future reliability of a power distribution system for a given location can be predicted using predictor models. However, the accuracy and performance of the predictor model are directly proportional to the amount of reliable training data used. Thus, a predictor model trained with insufficient training data is expected to be inaccurate. SUMMARY

[0004] According to one aspect of the present invention, a computer-implemented method for time series forecasting using multimodal augmentation and fusion is provided, comprising decomposing time series data and modal data into seasonal and trend representations with trend-season decomposition, concatenating, using an encoder-transformer model, time series data embeddings and modal data embeddings from the seasonal representations and the trend representations to obtain crossed representations, processing, using the encoder-transformer model, the modal data embeddings and the time series data embeddings separately to obtain singular representations, augmenting the crossed representations and the singular representations by common trend-season decomposition to obtain augmented seasonal data and augmented trend data, and fusing, using a decoder.from augmented seasonal data and augmented trend data to obtain fused augmented data, and to implement a corrective action to correct predicted future events, using a system with a prediction model trained on the fused augmented data.

[0005] According to another aspect of the present invention, a system for time series forecasting by means of multimodal augmentation and fusion is provided, comprising a storage device and one or more processor devices operationally coupled to the storage device to decompose time series data and modality data into seasonal and trend representations with trend-season decomposition, concatenating time series data embeddings and modality data embeddings from the seasonal representations and the trend representations using an encoder-transformer model to obtain crossed representations, processing the modality data embeddings and the time series data embeddings separately using the encoder-transformer model to obtain singular representations, and augmenting the crossed representations and the singular representations by common trend-season decomposition.to obtain augmented seasonal data and augmented trend data, to fuse augmented seasonal data and augmented trend data using a decoder to obtain fused augmented data, and to perform a corrective action to correct predicted future events using a system with a prediction model trained on the fused augmented data.

[0006] According to another aspect of the present invention, a non-volatile computer program product is provided, comprising a computer-readable storage medium containing program code for time-series forecasting by means of multimodal augmentation and fusion, wherein the program code, when executed on a computer, causes the computer to decompose time-series data and modality data into seasonal and trend representations with trend-season decomposition, concatenate time-series data embeddings and modality data embeddings from the seasonal representations and the trend representations using an encoder-transformer model to obtain crossed representations, and process the modality data embeddings and the time-series data embeddings separately using the encoder-transformer model to obtain singular representations.to augment the crossed representations and the singular representations by joint trend-season decomposition to obtain augmented seasonal data and augmented trend data, to fuse augmented seasonal data and augmented trend data using a decoder to obtain fused augmented data, and to perform a corrective action to correct predicted future events using a system with a prediction model trained on the fused augmented data.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The disclosure will provide details in the following description of preferred embodiments with reference to the following figures, wherein: Fig. 1 is a flowchart illustrating a higher-level overview of a method for time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention; Fig. 2 is a block diagram illustrating a computational system for time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention; Fig. 3 is a block diagram showing a computer software implementation of time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention; Fig. 4 is a block diagram illustrating a practical application of time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention; and Fig. Figure 5 is a block diagram illustrating deep learning neural networks for time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention. DETAILED DESCRIPTION OF PREFERRED EXECUTION FORMS

[0009] According to embodiments of the present invention, systems and methods for time series data forecasting are provided by means of multimodal augmentation and fusion.

[0010] Time-series data forecasting can occur in various complex real-world systems, including weather, healthcare, energy, and more. Accurate time-series data forecasting enables efficient system maintenance, resource allocation, risk mitigation, and improved decision-making. For example, accurately forecasting the future energy consumption of a specific location is crucial for efficiently allocating resources to an energy provider and ensuring a stable energy supply for that location.

[0011] In one embodiment, the present embodiments can enhance artificial intelligence models for time-series forecasting by employing a prediction model trained on fused augmented data. This data can be used to implement corrective actions to mitigate predicted future events, such as system failures, using a system equipped with the trained prediction model. The fused augmented data can be obtained by fusing singular and crossed representations through the integration of shared tokens using a decoder. The fused augmented data exhibits coherent alignment between the time-series and modality data.Singular representations can be obtained by separately processing modality and time-series data using an encoder-transformer model to augment the input dataset. Singular representations can refer to data embeddings that avoid interactions between patches across different modalities. Crossed representations can be obtained by concatenating time-series and modality data using an encoder-transformer model to improve the quality of an input dataset. Crossed representations can refer to data embeddings that exhibit common interactions across patches from different modality data. Time-series and modality data can be decomposed into seasonal and trend data.

[0012] The present embodiments improve the predictive performance of prediction models through multimodal data augmentation. Multimodal data augmentation can include horizontal and vertical augmentations, enriching the training data from different perspectives. In horizontal augmentation, time series and modality data can be concatenated and applied jointly to the encoder. Conversely, vertical augmentation utilizes modality data to increase the number of training samples. Thus, the present embodiments improve predictive performance by augmenting training samples.

[0013] The present embodiments improve the accuracy of prediction models by integrating shared tokens used to fuse multiple modalities. By allowing time-series and modality data to be constructed from a single set of learnable shared tokens, the present embodiments can achieve a coherent alignment of their different granularities, thus improving the interplay between these modalities. Furthermore, the shared tokens enable the present embodiments to interpret the significance of individual components within each modality. Therefore, the present embodiments achieve a significant improvement in prediction accuracy compared to other prediction methods.

[0014] In many scenarios, time series data can be accompanied by modality information. For example, a monitored system can collect both time series and modality data regarding the system's performance. However, methods that use single-modal datasets with data augmentation can become inaccurate if the data is insufficient.

[0015] The present embodiments improve prediction models because they can predict future events in the presence of insufficient time series data (e.g., data scarcity), while maintaining accuracy and performance through multimodal augmentation and fusion, which captures the relevance between multiple modalities.

[0016] With detailed reference to the figures in which identical reference symbols represent the same or similar elements, and initially to Fig. Figure 1 illustrates a general overview of a computer-implemented method for time-series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention. It is noted that the reference numerals for the references in Figure 1 are not provided in Figure 1. Fig. 1 described features in Fig. 3 will be described further.

[0017] In one embodiment, the present embodiments improve the accuracy and performance of prediction models by means of multimodal augmentation and fusion. Time series data 301 and modality data 303 can be collected as inputs. The time series data 301 and modality data 303 can be decomposed into their respective components of the seasonal (e.g., time series-season representation 308, modality-season representation 310) and trend data representations (e.g., time series-trend representation 307, modality-trend representation 309). These decomposed components then undergo multimodal augmentation using a shared encoder 317. During the multimodal augmentation, time series data 301 and modality data 303 can be concatenated to improve the quality of an input dataset and to obtain crossed representations.Additionally, modality data and time series data can be processed separately during multimodal augmentation to augment the input dataset and obtain singular representations. Furthermore, during multimodal augmentation, the crossed representations and the singular representations can be augmented to obtain augmented seasonal data 321 and augmented trend data 331.

[0018] Following multimodal augmentation, the augmented seasonal data and the augmented trend data can be fused by integrating shared tokens to obtain fused augmented data 342 with coherent alignment between the time series data 301 and the modality data 303. The fused augmented data 342 can then be passed individually to a decoder 341, which outputs these combined outputs to the ground-truth time series data 351 (shown in Fig. 3) approximated. A prediction model 353 can be trained with the fused augmented data 342, which can be used to predict future events and to implement a corrective action in accordance with the predicted future events.

[0019] In block 110, collected time series data and modality data can be decomposed into seasonal and trend data.

[0020] In one embodiment, collected time-series data 301 and modality data 303 can be decomposed by time-series decomposition. Time-series data 301 can refer to a set of data collected at regular intervals. For example, in power supply systems, the power output of the power lines can be collected at regular intervals (e.g., seconds, minutes). Modality data 303 can refer to various types of data relevant to the time-series data. Modality data 303 can be text data, audio data, or image data. For example, in the example above, the modality data 303 could be reports, audio recordings, or a graph showing that the performance of a system component of a monitored system has decreased by five percent.

[0021] Trend-seasonal decomposition can refer to the decomposition of time series data 301 (or modality data 303) into two types of representations: trend representations and seasonal representations. The trend representations can refer to smoothed time series data, which can be obtained by taking a moving average of the original data. The seasonal representations can refer to the deviation between the original data and the trend representation.

[0022] To obtain trend representations, the collected time series data can be subjected to patchwise embedding. The time series data 301 can be segmented into several non-overlapping patches for a patch length P, which can be mapped to a d-dimensional latent space using a learnable linear projection. The d-dimensional latent space can be learned by a frozen pre-trained language model. The frozen pre-trained language model can be a transformer model. To obtain seasonal representations, the modality data 303 can be segmented into several patches d, each patch spanning a duration of P time steps. The modality information contained in a patch d can be aggregated into a single embedding vector.Attention-based pooling using the frozen pre-trained language model can be used to account for potential differences in modality significance across different channels.

[0023] The seasonal representations can be mapped into a d-dimensional latent space using a trainable linear projection that can temporarily match the trend representations.

[0024] The seasonal representations and the trend representations can then be subjected to multimodal augmented coding, which may include at least two augmentation processes: vertical augmentation, which is further described in Block 130, and horizontal augmentation, which is further described in Block 120.

[0025] In Block 120, time series data embeddings and modality data embeddings from the seasonal representations and the trend representations can be concatenated to improve the quality of an input dataset and to obtain crossed representations.

[0026] In one embodiment, time-series data embeddings and modality data embeddings can be concatenated to improve the quality of an input dataset (e.g., horizontal augmentation) and to obtain crossed representations (e.g., 325, 329, 333, 339). Crossed representations can refer to data embeddings that exhibit common interactions across patches from different modality data. To obtain crossed representations, the time-series data embeddings (obtained by patch-wise embedding) and the modality data embedding (obtained by patch-wise embedding) can be split based on modality, concatenated, and fed into an encoder model to allow common interactions across patches from different modality data. The encoder model 317 can be a transformer model.

[0027] In Block 130, the modality data embeddings and the time series data embeddings can be processed separately to obtain singular representations.

[0028] In one embodiment, the modality data embeddings and the time-series data embeddings can be processed separately (e.g., vertical augmentation) to obtain singular representations (e.g., 323, 327, 335, 337). Singular representations can refer to embeddings that avoid interactions between patches across different modalities. To obtain singular representations, the modality data embeddings and the time-series data embeddings obtained from patch-wise embeddings can be fed separately into an encoder model 317. The encoder model 317 can be a transformer model.

[0029] Both vertical and horizontal augmentation improve the generalization of the encoder model and thus improve the accuracy and performance of a prediction model. 353 The crossed and singular representations can be further augmented by refining them through joint trend-season decomposition.

[0030] In block 140, the crossed representations and the singular representations can be augmented by joint trend-season decomposition to obtain augmented seasonal data and augmented trend data.

[0031] In one embodiment, the crossed representations and the singular representations can be augmented by joint trend-season decomposition to obtain augmented seasonal representations 321 and augmented trend representations 331. To obtain augmented seasonal representations 321 and augmented trend representations 331, both crossed representations and singular representations can be subjected to trend-season decomposition by an encoder model as described herein.

[0032] Performing a trend-season decomposition on the crossed representations can produce crossed trend representations and crossed seasonal representations. Performing a trend-season decomposition on the singular representations can produce singular trend representations (e.g., 335, 337) and singular seasonal representations (e.g., 323, 327). Performing a trend-season decomposition on the crossed representations can produce crossed trend representations (e.g., 333, 339) and crossed seasonal representations (e.g., 325, 329). During trend-season decomposition, the crossed representations and the singular representations can then be split into their time series and modality data embeddings, which can produce at least sixteen representations that are the different combinations of the time series, modality, crossed, singular, trend and seasonal representations.

[0033] To perform a trend-seasonal decomposition on the time series data of the crossed and singular representations, the trend and seasonal data can be separated, and patch embeddings can be calculated from the trend and seasonal data. To perform a trend-seasonal decomposition on the modality data of the crossed and singular representations, a separate attention parameter for the trend and seasonal data can be used to calculate the patch embeddings corresponding to the trend and seasonal data. The previously obtained patch embeddings of the time series data can then be used as query vectors in the attention mechanism, which can provide specialized embeddings that capture the dynamics of trend and seasonal patterns within the modality data.The augmented seasonal representations 321 and augmented trend representations 331 can then be fused to train a prediction model.

[0034] In block 150, the augmented seasonal and augmented trend data can be merged using a decoder to obtain merged augmented data.

[0035] In one embodiment, the augmented seasonal representations 321 and the augmented trend representations 331 can be fused using a decoder to obtain fused augmented data. To fuse the augmented seasonal representations 321 and the augmented trend representations 331, the possible combinations of the decomposed representations can be aggregated by single-layer linear decoders for the trend and seasonal representations. The loss for each prediction using its corresponding data (e.g., prediction of a seasonal representation of a single time modality) can be calculated individually, which can cumulatively contribute to a combined loss function. A prediction model 353 can then be trained using the combined loss function by learning the weights associated with each prediction using its corresponding data.The prediction model 353 can be a transformer model or other deep learning neural networks such as convolutional neural networks or recurrent neural networks.

[0036] By fusing the decomposed representations, information from different modalities, offering complementary insights, can be captured, thus improving the accuracy and performance of the prediction model. After training the prediction model with multimodal augmentation and fusion, it can be used to predict future events.

[0037] In block 160, a corrective action can be taken to prevent predicted future events using a system with a prediction model trained on the fused augmented data.

[0038] In one embodiment, a corrective action, such as updating the configuration of a monitored system, can be performed to prevent a predicted future event of a system failure.

[0039] In another embodiment, in a healthcare scenario, a corrective action, such as updating public health safety procedures, can be implemented for a monitored public health system to correct and prevent the spread of an infectious disease within a community. Additionally, a corrective action, such as updating a patient's medical treatment, can be implemented to prevent the progression of a disease.

[0040] The present embodiments improve prediction models because they can predict future events in the presence of insufficient time series data (e.g., data scarcity) while maintaining accuracy and performance through multimodal augmentation and fusion.

[0041] With reference to Fig. Figure 2 is a block diagram showing a computing system for time series data forecasting using multimodal augmentation and fusion 200 according to an embodiment of the present invention.

[0042] The computing device 200 includes, for illustrative purposes, the processor device 294, an input / output (I / O) subsystem 290, a memory 291, a data storage device 292, and a communication subsystem 293, and / or other components and devices commonly found in a server or similar computing device. In other embodiments, the computing device 200 may include other or additional components, such as those commonly found in a server computer (e.g., various input / output devices). Additionally, in some embodiments, one or more of the illustrative components may be integrated into another component or otherwise form part of it. For example, in some embodiments, the memory 291, or parts thereof, may be integrated into the processor device 294.

[0043] The processor device 294 can be configured as any type of processor capable of performing the functions described herein. The processor device 294 can be configured as a single processor, as multiple processors, as central processing unit(s) (CPU(s)), as graphics processing unit(s) (GPU(s)), as single- or multi-core processor(s), as digital signal processor(s), as microcontroller(s), or as another processor(s), or as processing / control circuit(s).

[0044] The memory 291 can be configured as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory 291 can store various data and software used during the operation of the computing device 200, such as operating systems, applications, programs, libraries, and drivers. The memory 291 is communicatively coupled to the processor device 294 via the I / O subsystem 290, which can be configured as circuitry and / or components to facilitate input / output operations with the processor device 294, the memory 291, and other components of the computing device 200. For example, the I / O subsystem 290 can be configured as memory controller hubs, input / output control hubs, platform control hubs, integrated control circuits, firmware devices, communication links (e.g.,Point-to-point connections, bus connections, wires, cables, optical fibers, conductor tracks, etc.) and / or other components and subsystems may be configured or otherwise include these to facilitate input / output operations. In some embodiments, the I / O subsystem 290 may form part of a system-on-a-chip (SoC) and be integrated together with the processor device 294, the memory 291, and other components of the computing device 200 on a single integrated circuit chip.

[0045] The data storage device 292 can be configured as any type of device or devices designed for short-term or long-term data storage, such as storage devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device 292 can store program code for time-series forecasting using multimodal augmentation and Fusion 100. Any or all of these program code blocks can be enclosed in a given computing system.

[0046] The communication subsystem 293 of the computing device 200 can be configured as any network interface controller or as any other communication circuit, device, or collection thereof capable of enabling communication between the computing device 200 and other remote devices over a network. The communication subsystem 293 can be configured to use one or more communication technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

[0047] As shown, the computing device 200 can also include one or more peripheral devices 295. The peripheral devices 295 can include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, the peripheral devices 295 can include a display, a touchscreen, a graphics circuit, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, GPS, a camera, and / or other peripheral devices.

[0048] Of course, the computing device 200 can also include other elements (not shown), as a person skilled in the art can readily imagine, as well as omit certain elements. For example, various other sensors, input devices, and / or output devices can be included in the computing device 200, depending on their respective implementation, as a person skilled in the art can readily understand. For example, different types of wireless and / or wired input and / or output devices can be used. In addition, extra processors, controllers, memory, and the like can also be used in various configurations. These and other variations of the computing system 200 are readily conceivable by a person skilled in the art in light of the teachings of the present invention provided herein.

[0049] It is understood that, although this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to a cloud computing environment. Rather, embodiments of the present invention are capable of being implemented in conjunction with any other type of computing environment now known or subsequently developed.

[0050] With reference to Fig. Figure 3 is a block diagram showing a computer software implementation of time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention.

[0051] In one embodiment, a software implementation 300 can include a trained prediction model 353, which can be trained by a training module 350 using fused augmented data 342 and ground truth 351. The fused augmented data 342 can be obtained by a fusion module 340, which can employ a decoder model 341 to fuse augmented seasonal representations 321 and augmented trend representations 331.

[0052] The augmented seasonal representations 321 can include seasonal singular time series data 323, seasonal crossed modality data 325, seasonal singular modality data 327, and seasonal crossed time series data 329. The augmented trend representations 331 can include singular trend time series data 335, crossed trend modality data 333, singular trend modality data 337, and crossed trend time series data 339. The singular data representations, which include seasonal singular time series data 323, singular trend modality data 337, singular trend time series data 335 and seasonal singular modality data 327, can be obtained by vertical augmentation 313 through a module 311 for multimodal augmentation that can employ the encoder model 317.The crossed data representations, which include seasonal crossed time series data 329, crossed trend modality data 333, crossed trend time series data 339 and seasonal crossed modality data 325, can be obtained by horizontal augmentation 315 through a module 311 for multimodal augmentation that can employ the encoder model 317.

[0053] The augmented seasonal representations 321 and augmented trend representations 331 are obtained by the multimodal augmentation module 311 by processing the time-series trend representation 307, the time-series seasonal representation 308, the modality trend representation 309, and the modality-season representation 310. The time-series trend representation 307, the time-series seasonal representation 308, the modality trend representation 309, and the modality-season representation 310 can be decomposed from collected time-series data 301 and modality data 303 using a trend-season decomposition module 305, which can employ a trained language model 306.

[0054] In one embodiment, the trained prediction model 353 can be used by a maintenance system for a monitored system. In another embodiment, the trained prediction model 353 can be used by an artificial intelligence (AI) assistant to support the decision-making process of a decision-making entity.

[0055] With reference to Fig. Figure 4 is a block diagram illustrating a practical application of time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention.

[0056] In one embodiment, a system 400 is shown that includes a maintenance system 406 in the context of a monitored system 402. The monitored system 402 can be any suitable system, including physical systems such as production lines and physical plant operations, electronic systems such as computers or other computerized devices, software systems such as operating systems and applications, and cyber-physical systems that combine physical systems with electronic systems and / or software systems. Exemplary systems can include a wide range of different types, including railway systems, power plants, vehicle sensors, data centers, satellites, and transportation systems. Another type of cyber-physical system can be a network of Internet of Things (IoT) devices, which can include a variety of different types of devices with different respective functions and sensor types.The monitored system 402 can employ a maintenance system 406 which includes a trained prediction model 353 with time series forecasting using multimodal augmentation and fusion 100 and sensors that collect time series data and modality data from the monitored system.

[0057] One or more sensors 404 record information about the state of the monitored system 402. The sensors 404 can be any suitable type of sensor, including, for example, physical sensors such as temperature, humidity, vibration, pressure, voltage, current, magnetic field, electric field, and light sensors, and software sensors such as logging programs installed on a computer system to record information about the state and behavior of the operating system and the applications running on the computer system. The sensor data can include, for example, numerical data and categorical or binary data. The information generated by the sensors 404 can be in any suitable format and can include sensor log information generated using heterogeneous formats.

[0058] The sensors 404 can transmit the logged sensor information to a maintenance system 406 via any suitable communication medium and protocol, including wireless and wired communications. The maintenance system 406 can, for example, predict abnormal or anomalous behavior by monitoring the multivariate time series generated by the sensors 404. Once anomalous behavior has been predicted, the maintenance system 406 communicates with a system control unit to modify one or more parameters of the monitored system 402 in order to prevent and correct the anomalous behavior.

[0059] Examples of corrective actions include changing a security setting for an application or hardware component, changing an operating parameter of an application or hardware component (for example, an operating speed), stopping and / or restarting an application, stopping and / or restarting a hardware component, changing an environmental condition, changing the status or settings of a network interface, etc. The 406 maintenance system automatically corrects or mitigates the anomalous behavior by doing so. By identifying the specific 404 sensors associated with the anomalous classification, the time required to isolate a problem can be reduced.

[0060] Each of the 404 sensors outputs a time series that encodes measurements taken by the sensor over time. For example, the time series can include pairs of information, with each pair containing a measurement and a timestamp representing the time the measurement was taken. Each time series can be divided into segments representing measurements taken by the sensor over a specific period. Time series segments can represent any suitable interval, such as one second, one minute, one hour, or one day. Time series segments can represent a predetermined number of acquisition times instead of a fixed period, covering, for example, one hundred measurements. The 404 sensors can also output modality data collected from the monitored system 402, such as audio recordings, text, or images of system components.

[0061] Maintenance system 406 therefore includes a model trained to handle numerical and categorical data. The model can be based on patch embeddings, where a time series can be partitioned into multiple sets of non-overlapping patch embeddings, one for each categorical value. The distribution of patch embeddings can then be used to determine a normal range and threshold for these values ​​to aid in predicting and identifying anomalies.

[0062] For a complex System 406, the number of sensors can be very large, with each sensor reporting independent streams of time-series data. Predicting the cause of an anomaly in such a system can be challenging. A model can be trained to predict and detect anomalies in an explainable way, reporting not only the time period, type, and value details of an anomaly, but also providing explanations as to why the anomaly is abnormal and how normal data would compare. To explain the results of anomaly prediction, anomaly profiles can be stored to predict and identify the cause of the anomaly. Expected values ​​can be provided as a normal baseline for comparison.

[0063] In another embodiment, a community's energy consumption can be forecast using a trained prediction model. This model is trained using time-series data of household energy consumption and modality data relevant to the community's energy consumption, such as newspaper articles, audio recordings of utility workers, and images. The energy consumption forecast can then be used by an energy department entity to create an energy consumption plan and increase energy delivery to the community to address energy deficits.

[0064] In another embodiment, in a healthcare scenario, the severity of a person's illness can be predicted using time-series health data from the patient, such as vital signs. Modal health data, such as health summary data, audio recordings from medical professionals, or X-rays, can be sent to a network and used to train a prediction model with time-series data using multimodal augmentation and Fusion 100 to obtain a trained prediction model. This trained prediction model can then be used by a health prognosticator, an artificial intelligence assistant, to generate a predicted severity of the patient's illness.The predicted severity of the patient's illness can then be used by medical professionals to support the decision-making process for establishing a medical diagnosis. Predicting the severity of the patient's illness allows for a more timely medical diagnosis and updated medical treatment, potentially leading to a better patient outcome.

[0065] In another embodiment, the spread of an infectious disease within a community can be predicted using time-series health data on confirmed cases, deaths, and recoveries collected at various locations. In addition to collected time-series health data, modality data relating to the infectious disease, such as news articles, records, or images, can be used to train a prediction model with multimodal augmentation and fusion. By predicting the spread of the infectious disease, a public health entity can determine its severity. This severity can then dictate the corrective action the public health entity can take to prevent an increase in the severity of the infectious disease.

[0066] Other practical applications are being considered.

[0067] The present embodiments can employ a predictor model to forecast future events using time series and modality data via multimodal augmentation and fusion. The predictor model can be a deep learning neural network.

[0068] With reference to Fig. Figure 5 is a block diagram illustrating deep learning neural networks for time series data forecasting using multimodal augmentation and fusion according to an embodiment of the present invention.

[0069] A neural network is a generalized system that improves its function and accuracy through exposure to additional empirical data. The neural network is trained by exposure to this empirical data. During training, the neural network stores and adjusts a variety of weights that are applied to the incoming empirical data. By applying these adjusted weights to the data, the network can identify the data as belonging to a specific predefined class from a set of classes, or it can output a probability that the input data belongs to each of the classes.

[0070] The empirical data, also known as training data, from a set of examples can be formatted as a chain of values ​​and fed into the input of the neural network. Each example can be associated with a known outcome or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data can include a variety of different data types and can contain multiple distinct values. The network can have an input neuron for each value that constitutes the example's input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, an array, or a chain, depending on the architecture of the neural network being constructed and trained.

[0071] The neural network "learns" by comparing the output it generates from the input data with known values ​​from the examples and adjusting the stored weights to minimize the differences between the output values ​​and the known values. These adjustments to the stored weights can be made through backpropagation, where the effect of the weights on the output values ​​can be determined by calculating the mathematical gradient and adjusting the weights in a way that shifts the output toward a minimal difference. This optimization, known as gradient descent, is a non-restrictive example of how training can be performed. A subset of examples with known values, not used for training, can be used to test and validate the accuracy of the neural network.

[0072] In operation, the trained neural network can be used with new data that was not previously used in training or validation through generalization. The adjusted weights of the neural network can be applied to the new data, with the weights estimating a function developed from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0073] The deep neural network 500, like a multilayer perceptron, can have an input layer 511 consisting of source neurons 512, one or more computational layer(s) 526 with one or more computational neurons 532, and an output layer 540, with a single output neuron 542 for each possible category into which the input sample could be classified. An input layer 511 can have a number of source neurons 512 equal to the number of data values ​​512 in the input data 511. The computational neurons 532 in the computational layer(s) 526 can also be called hidden layers, since they lie between the source neurons 512 and the output neuron(s) 542 and are not directly observed.Each neuron 532, 542 in a computation layer generates a linear combination of weighted values ​​from the values ​​output by the neurons in a previous layer and applies a non-linear activation function that is differentiable over the range of the linear combination. The weights applied to the value from each previous neuron can be denoted, for example, by w1, w2, ... w. n–1 , w n The output layer provides the network's overall response to the input data. A deep neural network can be fully connected, with each neuron in a computation layer connected to all other neurons in the previous layer, or it can have other configurations of connections between layers. If connections between neurons are missing, the network is said to be partially connected.

[0074] In one embodiment, the computational layers 526 of the trained prediction model 353 can learn relationships between the fused augmented data 342 and the ground truth 351. The output layer 540 of the trained prediction model 353 can then provide the overall response of the network as a probability value for predicting a future event based on the fused augmented data 342 and the ground truth 351. In another embodiment, the encoder model 317 can learn linear projections to map patch embeddings into a d-dimensional latent space to perform a trend-season decomposition. In yet another embodiment, the decoder model 341 can learn the shared tokens between the augmented seasonal representations 321 and the augmented trend representations 331 to fuse both representations and obtain fused augmented data 342.

[0075] Training a deep neural network can involve two phases: a forward phase in which the weights of each neuron are fixed and the input propagates through the network, and a backward phase in which an error value is propagated backward through the network and weight values ​​are updated.

[0076] The computational neurons 532 in the one or more (hidden) computational layer(s) 526 perform a non-linear transformation on the input data 512, which generates a feature space. The classes or categories may be easier to separate in the feature space than in the original data space.

[0077] The embodiments described herein may consist entirely of hardware, entirely of software, or include both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which may include, but is not limited to, firmware, resident software, microcode, etc.

[0078] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium, providing program code for use by or in conjunction with a computer or any command-execution system. A computer-usable or computer-readable medium may include any device that stores, communicates, propagates, or transports the program for use by or in conjunction with the command-execution system, device, or apparatus. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or device or apparatus), or a propagation medium.The medium can include a computer-readable storage medium such as semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid magnetic hard disk, optical disk, etc.

[0079] Any computer program can be tangibly stored in a machine-readable storage medium or device (e.g., program memory or magnetic hard disk) that is readable by a programmable general-purpose or specialized computer to configure and control the operation of a computer when the storage medium or device is read by the computer to perform the procedures described herein. The inventive system can also be considered to be embodied in a computer-readable storage medium configured with a computer program, wherein the storage medium, as configured, causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0080] A data processing system capable of storing and / or executing program code may include at least one processor, which is directly or indirectly coupled to memory elements via a system bus. The memory elements may include local memory, used during the actual execution of the program code, mass storage, and cache memory, which provides temporary storage of at least some program code to reduce the number of times code is retrieved from mass storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intermediary I / O controllers.

[0081] Network adapters can also be connected to the system to allow the data processing system to connect to other data processing systems or remote printers or storage devices via intermediary private or public networks. Modems, cable modems, and Ethernet cards are just some of the types of network adapters currently available.

[0082] As used herein, the term “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combinations thereof, cooperating to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be contained within a central processing unit, a graphics processing unit, and / or a separate processor- or computationally-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memory, etc.).In some embodiments, the hardware processor subsystem may include one or more memories, which may be on-board or off-board, or which may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.).

[0083] In some embodiments, the hardware processor subsystem can include and execute one or more software elements. These software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0084] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0085] These and other variations of a hardware processor subsystem are also considered in accordance with embodiments of the present invention.

[0086] References in the description to "an embodiment" or "an embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, property, and so forth, described in connection with the embodiment, is included in at least one embodiment of the present invention. Thus, the appearances of the expression "in an embodiment," as well as all other variations occurring at various points in the description, do not necessarily all refer to the same embodiment. It should be understood, however, that features of one or more embodiments may be combined in light of the teachings of the present invention provided herein.

[0087] It is understood that the use of any of the following, “ / ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B”, and “at least one of A and B”, is intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of “A, B and / or C” and “at least one of A, B and C”, such wording is intended to include the selection of only the first listed option (A), the selection of only the second listed option (B), the selection of only the third listed option (C), the selection of only the first and second listed options (A and B), the selection of only the first and third listed options (A and C), the selection of only the second and third listed options (B and C), or the selection of all three options (A, B, and C).This can be extended to any number of listed items.

[0088] The foregoing is to be understood in every respect as illustrative and exemplary, but not as limiting, and the scope of protection of the invention disclosed herein is to be determined not from the detailed description, but rather from the claims as interpreted in accordance with the full breadth permitted by patent law. It is understood that the embodiments shown and described herein are merely illustrative of the present invention and that those skilled in the field may implement various modifications without departing from the scope of protection and spirit of the invention. Those skilled in the field could implement various other combinations of features without departing from the scope of protection and spirit of the invention.Since aspects of the invention have thus been described with the details and special features required by patent law, what is claimed and is to be protected by the patent certificate is set out in the attached claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 533,391

[0001] US 63 / 539,909

[0001] US 18 / 806,025

[0001]

Claims

[1] A computer-implemented method for time series forecasting using multimodal augmentation and fusion, comprising: Decomposition (110) of time series data and modality data into seasonal representations and trend representations using trend-season decomposition; Concatenating (120), using an encoder-transformer model, of time series data embeddings and modality data embeddings from the seasonal representations and the trend representations to obtain crossed representations; Process (130) using the encoder-transformer model, the modality data embeddings and the time series data embeddings separately to obtain singular representations; Augmenting (140) the crossed representations and the singular representations by joint trend-season decomposition to obtain augmented seasonal data and augmented trend data; Merge (150), using a decoder, augmented seasonal data and augmented trend data to obtain merged augmented data; and Implementing (160) a corrective action to correct predicted future events, using a system with a prediction model trained on the fused augmented data. [2] The computer-implemented method according to claim 1, wherein performing a corrective action further comprises updating an operating parameter of a hardware component of a monitored system in order to prevent a predicted system failure. [3] The computer-implemented method according to claim 1, wherein carrying out a corrective action further comprises updating a medical diagnosis for a monitored patient in order to correct and prevent the progression of a disease. [4] The computer-implemented method according to claim 1, wherein the decomposition of time series data and modality data further comprises learning patch embeddings using a language model to map the patch embeddings into a d-dimensional latent space. [5] The computer-implemented method according to claim 1, wherein concatenating time-series data and text data using an encoder-transformer model further comprises splitting concatenated embeddings of time-series data and text data to allow crossed interactions. [6] The computer-implemented method according to claim 1, wherein the augmentation of the crossed representations and the singular representations further comprises the computation of attention parameters for the modality data of the crossed representations and the singular representations using patch embeddings of the time series data as query vectors in an attention mechanism. [7] The computer-implemented method according to claim 1, wherein the fusion of the singular representations and the crossed representations using a decoder further comprises aggregating the augmented seasonal data and the augmented trend data to contribute to a loss function for training a prediction model. [8] A system for time series forecasting using multimodal augmentation and fusion, comprising: a storage device (292); and one or more processor devices (294) that are operationally coupled to the storage device to: To decompose time series data and modality data into seasonal and trend representations using trend-season decomposition (110); using an encoder-transformer model to concatenate time series data embeddings and modality data embeddings from the seasonal representations and the trend representations (120) to obtain crossed representations; using the encoder-transformer model to process the modality data embeddings and the time series data embeddings separately (130) to obtain singular representations; to augment the crossed representations and the singular representations by joint trend-season decomposition (140) to obtain augmented seasonal data and augmented trend data; using a decoder to fuse augmented seasonal data and augmented trend data (150) to obtain fused augmented data; and to implement a corrective action (160) to correct predicted future events, using a system with a prediction model trained on the fused augmented data. [9] The system according to claim 8, wherein the one or more processor devices which are operationally coupled with the storage device in order to carry out a corrective action further comprise updating an operating parameter of a hardware component of a monitored system in order to prevent a predicted system failure. [10] The system according to claim 8, wherein the one or more processor devices which are operationally coupled with the storage device to perform a corrective action further comprise performing a corrective action which further comprises updating a medical diagnosis for a monitored patient in order to correct and prevent the progression of a disease. [11] The system according to claim 8, wherein the one or more processor devices which are operationally coupled to the storage device to decompose time series data and modality data further comprise learning patch embeddings using a language model to map the patch embeddings into a d-dimensional latent space. [12] The system according to claim 8, wherein the one or more processor devices which are operationally coupled to the storage device to concatenate time series data and text data using an encoder-transformer model, further comprise splitting concatenate embeddings of time series data and text data to allow crossed interactions. [13] The system according to claim 8, wherein the one or more processor devices which are operationally coupled to the storage device to augment the crossed representations and the singular representations further comprise the computation of attention parameters for the modality data of the crossed representations and the singular representations using patch embeddings of the time series data as query vectors in an attention mechanism. [14] The system according to claim 8, wherein the one or more processor devices which are operationally coupled to the storage device to fuse the singular representations and the crossed representations using a decoder, further comprise aggregating the augmented seasonal data and the augmented trend data to contribute to a loss function for training a prediction model. [15] A non-volatile computer program product comprising a computer-readable storage medium containing program code for time series forecasting using multimodal augmentation and fusion, wherein the program code, when executed on a computer, causes the computer to: Decomposition (110) of time series data and modality data into seasonal and trend representations using trend-season decomposition; Concatenating (120), using an encoder-transformer model, of time series data embeddings and modality data embeddings from the seasonal representations and the trend representations to obtain crossed representations; Process (130) using the encoder-transformer model, the modality data embeddings and the time series data embeddings separately to obtain singular representations; Augmenting (140) the crossed representations and the singular representations by joint trend-season decomposition to obtain augmented seasonal data and augmented trend data; Merge (150), using a decoder, augmented seasonal data and augmented trend data to obtain merged augmented data; and Implementing (160) a corrective action to correct predicted future events, using a system with a prediction model trained on the fused augmented data. [16] The non-volatile computer program product according to claim 15, wherein performing a corrective action further comprises updating an operating parameter of a hardware component of a monitored system in order to prevent a predicted system failure. [17] The non-volatile computer program product according to claim 15, wherein the decomposition of time series data and modality data further comprises learning patch embeddings using a language model to map the patch embeddings into a d-dimensional latent space. [18] The non-volatile computer program product according to claim 15, wherein concatenating time series data and text data using an encoder-transformer model further comprises splitting concatenated embeddings of time series data and text data to allow crossed interactions. [19] The non-volatile computer program product according to claim 15, wherein the augmentation of the crossed representations and the singular representations further comprises the computation of attention parameters for the modality data of the crossed representations and the singular representations using patch embeddings of the time series data as query vectors in an attention mechanism. [20] The non-volatile computer program product according to claim 15, wherein the fusion of the singular representations and the crossed representations using a decoder further comprises aggregating the augmented seasonal data and the augmented trend data to contribute to a loss function for training a prediction model.

Images