Systems and techniques for event prediction from observed sequence of events

Abstract

The present disclosure relates to systems and techniques for event prediction from an observed sequence of events. An event analysis and prediction system is disclosed that is configured to analyze at least one event data stream from a monitored system for the purpose of predicting future events in the system, such as for predictive maintenance of the system. The event analysis and prediction system advantageously predicts, in real time, such faults in the monitored system that would otherwise lead to interruptions. The event analysis and prediction system advantageously utilizes a novel data generation process that essentially translates the problem of fault prediction into a classification problem that is solved using a machine learning algorithm. The event analysis and prediction system is designed to work well even when label data available for model training is limited.

Description

Technical Field

[0001] The apparatus and methods disclosed in this document relate to system event analysis, and more particularly to predicting events based on observed event sequences. Background Art

[0002] Unless otherwise indicated herein, the materials described in this section are not admitted to be prior art for inclusion in this section.

[0003] Many real-world systems emit sequences of observed events, where each event captures a specific behavior of the system (e.g., anomalous / abnormal behavior). For example, modern manufacturing plants often generate many different types of data as streams, such as event logs from machines in the manufacturing plant. The time-stamped sequence of observed events can be collected from a variety of sources, such as software logs, output of anomaly detection engines, equipment diagnostic logs, and more.

[0004] Occasionally, these events may lead to another critical event in the system (i.e., system failure, etc.), which can result in downtime and potential loss of revenue. For example, in a typical manufacturing plant, machines may be placed at specific locations and may form a "line" of machines. In each line of machines, the output of the machine is automatically fed as the input of the next machine in the line, so that the manufacturing process can be automated. However, this does not mean that the operation of the machine will not have any errors or stops at all. In many cases, machines begin to run irregularly or do not continue to function as expected from their operation over time. In such cases, the machine requires a stop command so that a human operator or expert can solve the potential problem, or the machine may stop unpredictably on its own. Regardless of whether this stop is planned or unplanned, every second that the machine is not running is a direct loss of productivity, and therefore a loss that can be quantified in dollars.

[0005] Predictive maintenance (PdM) is a promising approach to predict and prevent such disruptions in manufacturing plants and other real-world systems. Predictive maintenance uses data collected from a system (e.g., from a machine in a manufacturing plant) to make predictions about future failures in the system. This helps detect and mitigate system failures, thus increasing system uptime and reducing system maintenance costs. What is needed is a method for predictive maintenance that better utilizes historical data from a system to accurately and reliably predict future system failures. Summary of the invention

[0006] A method for training a machine learning model to predict future events in a system is disclosed. The method includes receiving, with a processor, event data from the system, the event data indicating events that occurred in the system and times when the events occurred. The method further includes determining, with the processor, a plurality of positive training samples based on the event data. The method further includes determining, with the processor, a plurality of negative training samples based on the event data. The method further includes training, with the processor, a machine learning model using the plurality of positive training samples and the plurality of negative training samples to predict, given a sequence of events that occurred at a particular location during a time window, whether an event of a particular type will occur at a particular location within a first threshold amount of time after the time window.

[0007] A method for predicting possible future events in a system is disclosed. The method includes receiving, using a processor, event data from the system, the event data indicating events that occurred in the system and times when the events occurred. The method further includes determining, using the processor, a sequence of events, the sequence of events including events that occurred at a corresponding location within the system and during a time window before a current time. The method further includes using, using the processor, a machine learning model to predict, based on the sequence of events, whether a particular type of event will occur at the corresponding location within a first threshold amount of time after the time window. The method further includes perceptibly outputting, using a user interface, an alert in response to predicting that the particular type of event will occur at the corresponding location within a first threshold amount of time after the time window. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The foregoing aspects and other features of the system and method are explained in the following description in conjunction with the accompanying drawings.

[0009] Figure 1 An event analysis and prediction system according to the present disclosure is shown.

[0010] Figure 2 An exemplary embodiment of a computing device for analyzing event data streams and predicting future events is shown.

[0011] Figure 3 An example visualization showing an event-location timeline.

[0012] Figure 4 A flow chart showing a method for training a machine learning model to predict future events in a system.

[0013] Figure 5 A flow chart illustrating a method for predicting possible future events in a system.

[0014] Figure 6A step-by-step scan of the event-location timeline is shown.

[0015] Figure 7 An exemplary feature extraction for positive training samples is shown.

[0016] Figure 8 Showing a customized cross-validation split that avoids feature leakage.

[0017] Fig. 9 Shows the conditions for a successful prediction or a false positive.

[0018] Fig.10 A scenario is shown where two outages occur in the same prediction interval.

[0019] Fig.11 An exemplary explanation showing which events contribute to a predicted outage.

[0020] Fig.12 A graph showing the resulting precision-recall curve is shown.

[0021] Fig.13 Shows visualization of data generated across different locations using t-SNE plots. DETAILED DESCRIPTION

[0022] In order to promote an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the accompanying drawings and described in the following written specification. It should be understood that no limitation of the scope of the present disclosure is intended thereby. It should also be understood that the present disclosure includes any changes and modifications to the illustrated embodiments, and includes further applications of the principles of the present disclosure that would normally occur to a person skilled in the art to which the present disclosure belongs.

[0023] Overview

[0024] Figure 1 An event analysis and prediction system 10 in accordance with the present disclosure is shown. The event analysis and prediction system 10 is configured to analyze at least one event data stream 106 from a monitored system 104 for the purpose of predicting future events in the system 104, such as for predictive maintenance of the system 104. As will be discussed in more detail below, the event analysis and prediction system 10 advantageously predicts in real time such failures in the monitored system 104 that would otherwise result in an outage. The event analysis and prediction system 10 advantageously utilizes a novel data generation process that essentially transforms the problem of failure prediction into a classification problem that is solved using a machine learning algorithm. The event analysis and prediction system 10 is designed to work well even when limited labeled data is available for model training.

[0025] Throughout the disclosure, the monitored system 104 will be described primarily using an illustrative example of a manufacturing plant with multiple machines, various events about which may occur and which are monitored to generate event data. However, it should be appreciated that the event analysis and prediction system 10 may be applied to any system that generates event data, either by self-monitoring or by some other monitoring mechanism.

[0026] In the illustrative example, the monitored system 104 is a manufacturing plant including a plurality of machines 40. The machines of the manufacturing plant may include, for example, welding stations, storage tanks, mixers, compressors, centrifuges, etc. Each machine of the manufacturing plant is installed at a specific location in the manufacturing plant and has a specific configuration. The machines may be interconnected to form a line of machines. In each line of machines, the output of the machine may be automatically fed as the input of the next machine in the line, so that the manufacturing process can be fully automated. When the machine is running, the machine is continuously monitored by sensors. These sensors measure various parameters of the machine, such as the position of different components, the angles between components, the forces applied, the pressures, and more. Therefore, each location continuously emits different types of data during its operation, which may be related to events such as anomalies, interruptions, and other types of event data over time.

[0027] As used in this article, the term "location" of a machine is not limited to its narrow literal meaning, but refers to a "machine" (or station or equipment) that operates or runs at a specific physical location using a specific configuration (e.g., functional unit, work location, and tool location). Therefore, each location can be identified using a unique identifier (location_id) so that it can be identified and selected in a database or other software system. For each location, a number of parameters are specified by operators or domain experts in the factory. Therefore, these parameters are associated with each location, and the values ​​of these parameters are continuously monitored using sensors and then subsequently stored in a database. Parameters can be any value from a machine, such as the physical coordinates of a component at that location, a measurement of the force, pressure, or speed applied, etc.

[0028] Continue to refer Figure 1, the event data stream 106 includes at least data indicating an event that occurred in the monitored system 104 and a timestamp indicating the time when the event occurred. In the illustrated example, the anomaly detection engine (ADE) 20 monitors data (e.g., sensor data) received from the machine 40 in the monitored system 104 in real time. The ADE 20 is configured in advance by the user to detect various types of abnormal events, such as outliers, sudden changes in the mean or variance, gradual mean shifts, increases in zero values, etc. Whenever such an anomaly occurs, the details of the anomaly are recorded in real time in the anomaly database 22. Some of these anomalies are not critical and may appear purely because of the rules that mark these anomalies. On the other hand, some of these anomalies may indicate potential faults in the machine 40 that need to be corrected. Occasionally, these faults can lead to the paralysis of the machine 40, which causes an interruption of the factory line. Whenever such an interruption occurs, a separate interruption processing system 30 records the details of the interruption in a separate interruption database 32. Therefore, the event analysis and prediction system 10 and / or the monitored system 104 combine two systems for recording events. The event analysis and prediction system 10 combines two sources of event data in the event data stream 106 into a single database 102 .

[0029] As used herein, "event" means any single discrete moment that occurs with respect to a system at a specific time. Events are characterized by timestamps and locations. Depending on the nature of the system (i.e., the monitored system 104), events can be classified into any number of different event types. In the illustrative example of a manufacturing plant, an event refers to any single discrete moment that occurs at a specific location, machine, and / or component at a manufacturing plant at a specific time. For example, it is assumed that the manufacturing plant includes a welding machine w located at a manufacturing line m. Assume that at a specific time, such as 2018-06-12 09:55:22, an event e1 occurs with respect to welding machine w. For example, event e1 may be that a sensor of welding machine w indicates that some measured parameters have abnormal values. Subsequently, at a specific time, such as 2018-06-12 10:51:02, another event e2 occurs with respect to welding machine w. For example, event e2 may be that the operation of the machine w is interrupted, i.e., its operation stops suddenly. According to this example, event e1 and event e2 can be logically classified into two event types: "abnormal" (ie, event e1) and "interruption" (ie, event e2).

[0030] As used herein, an "anomaly" is an event that represents a moment when any measurable or observable parameter from any location or any component of a system behaves abnormally, for example, has an irregular value outside a predetermined or expected range, or tends to become a value outside a predetermined or expected range. The predetermined or expected range can be fixed or based on an observed trend for the parameter. In the manufacturing plant example, the temperature of some chambers of some machines is outside a predetermined allowable range, or the speed of a rotating shaft in some other machine is lower than its average speed, which may be reported as an anomaly in the event data stream 106. Like other events, anomalies are characterized by a timestamp and a location. In addition, anomalies are further characterized by the parameters being measured, the technology used to determine the anomaly, and / or the level of the anomaly.

[0031] As used herein, an "interruption" is an event representing a moment when the process of a system is stopped, for example, due to a failure of a component of the system or due to intervention by a human operator or an automated intervention system. In the manufacturing plant example, an interruption may cause delays in the manufacturing line / process. In addition to requiring the intervention of an expert (e.g., a line operator) to resolve what caused the interruption, these delays may result in the stoppage of the manufacturing component and have a quantifiable monetary impact. Like other events, an interruption is characterized by a timestamp and a location. In addition, an interruption is further characterized by its identifier code or description, the duration of the interruption, the type of interruption, and / or manually entered text describing the interruption.

[0032] It should be appreciated that exception and interruption are merely exemplary event categories, and although the classification scheme is used herein to describe the events of event data stream 106, it should be appreciated that events can be classified into any number of different event categories. In a broad sense, the combination of other event categories can also be used similarly with other scenes, settings, configurations or neighborhoods. For example, any other event category (such as warning or precursor events) can be used as an alternative, rather than using the exception from the anomaly detection engine as an alternative. Such event categories can also be component-change events (that is, over time, some components in the machine often wear out during its operation, and it needs to be replaced) or any other alarms from any other monitoring system installed in a location or factory. Similarly, other key event categories can be used here, such as waste-detection events (that is, components or products built by the machine are not properly manufactured, and they need to be discarded) or any other failures of sites that need to be prevented in advance, rather than using interruptions.

[0033] It should be appreciated that the event classes do not have to be those from the manufacturing domain, but any other events from any other domain may be similarly used, for example, 1) delivery delays may be used to predict disruptions or any other critical events in the transportation domain, and 2) anomalies from patient monitoring systems may be used to predict critical events such as heart attacks in the healthcare domain to prevent any sudden illness of the patient, etc.

[0034] In at least some embodiments, in addition to data indicating an event that occurred in the monitored system 104 and a timestamp indicating the time when the event occurred, the event data stream 106 may further include certain metadata, such as where the event occurred, what caused the event, what sensor detected the event, a human-entered text description of the event, an indication of whether the event was a planned or unplanned outage, the duration of the event, and other parameters or contextual information, etc. In any case, the event analysis and prediction system 10 is configured to analyze the data of the event data stream 106 and store it in the database 102.

[0035] Reference again Figure 1 , the task of the event analysis and prediction system 10 is to predict future outages given a data set of historical events stored in a database 102. To this end, the event analysis and prediction system 10 combines event prediction software with a machine learning model 124 that is trained to perform event prediction tasks, such as outage prediction. In the machine learning training pipeline 50, the machine learning model 124 is periodically trained and updated offline using the anomaly, outage, and other event data collected in the database 102. In addition, in the real-time machine learning inference pipeline 60, the latest version of the machine learning model 124 is deployed and used for real-time prediction of outages. When an outage is predicted, the event analysis and prediction system 10 provides an alert 70 to the operator of the monitored system 104.

[0036] The event analysis and prediction system 10 described herein has various advantages. In particular, as discussed in more detail below, the problem of interruption prediction is advantageously viewed as a binary classification, in which the output of the system 10 represents the probability that an interruption will occur soon. In this way, the output of the system 10 is easily understood and interpreted by the operator. In addition, the system 10 is advantageously able to predict the probability of an interruption with high accuracy. As a result, the production line is less likely to stop for inspection due to a false positive. In addition, the interruption prediction is easily interpretable. This will help the operator understand the reasons behind the prediction and will also help them perform the inspection. Explainable predictions are also important for early detection of faults in the machine learning model 124. Finally, the system 10 is advantageously able to predict the probability of an interruption in real time. This is important because operators need to know as soon as possible whether an interruption is likely to occur so that they can take the necessary measures to prevent it.

[0037] Exemplary Hardware Embodiment

[0038] refer to Figure 2 , describes an exemplary embodiment of a computing device 100 (e.g., a server) that can be used to analyze event data streams 106 from monitored systems 104 and predict future events. The computing device 100 includes a processor 110, a memory 120, a display screen 130, a user interface 140, and at least one network communication module 150. It will be appreciated that the illustrated embodiment of the computing device 100 is only one exemplary embodiment and is merely representative of any of the various ways or configurations of a server, desktop computer, laptop computer, mobile phone, tablet computer, or any other computing device that can operate in the manner set forth herein. The computing device 100 communicates with a database 102, which can be hosted by another device or stored in the computing device 100's own memory 120.

[0039] The processor 110 is configured to execute instructions to operate the computing device 100, thereby implementing the features, functions, characteristics, etc. described herein. To this end, the processor 110 is operably connected to the memory 120, the display screen 130, and the network communication module 150. The processor 110 generally includes one or more processors, which can run in parallel or otherwise cooperate with each other. Those of ordinary skill in the art will recognize that a "processor" includes any hardware system, hardware mechanism, or hardware component that processes data, signals, or other information. Therefore, the processor 110 may include a system having a central processing unit, a graphics processing unit, a plurality of processing units, a dedicated circuit for implementing a function, a programmable logic, or other processing system.

[0040] The memory 120 is configured to store data and program instructions that, when executed by the processor 110, enable the computing device 100 to perform the various operations described herein. As will be appreciated by one of ordinary skill in the art, the memory 120 may be any type of device capable of storing information accessible to the processor 110, such as a memory card, ROM, RAM, hard drive, disk, flash memory, or any of a variety of other computer-readable media used as data storage devices.

[0041] Display screen 130 may include any of various known types of displays configured to display a graphical user interface, such as an LCD or OLED screen. User interface 140 may include various interfaces for operating computing device 100, such as buttons, switches, a keyboard or other keypad, a speaker, and a microphone. Alternatively or additionally, display screen 130 may include a touch screen configured to receive touch input from a user.

[0042] The network communication module 150 may include one or more transceivers, modems, processors, memories, oscillators, antennas, or other hardware conventionally included in communication modules to enable communication with various other devices. In particular, the network communication module 150 typically includes: an Ethernet adapter or module, which is configured to implement communication with a wired or wireless network; and / or a router (not shown), which is configured to implement communication with various other devices. In addition, the network communication module 150 may include module (not shown) and one or more cellular modems configured to communicate with a wireless telephone network.

[0043] In at least some embodiments, the memory 120 stores program instructions of an event analysis program 122, which can be used to analyze the event data stream 106 from the monitored system 104 and predict future events, and which includes at least a machine learning model 124 configured for real-time prediction of future events (e.g., outages) in the monitored system 104. The database 102 stores at least a plurality of event data 160, which includes raw data indicating events occurring in the monitored system 104, timestamps indicating the time when the event occurred, and various metadata discussed above. These event data 160 are used to train the machine learning model 124 in an offline setting to perform the task of outage prediction.

[0044] Reformulating event prediction as a classification problem

[0045] Since events can occur at different locations and at different times, the following definitions are utilized to distinguish events occurring in the monitored system 104. First, the set of events is defined (Definition 2.1) as a way of referring to all anomalies and outages within the monitored system 104. In particular, let E be the set of all events, defined as E:=A∪I. In this and other definitions, A is the set of all anomalies and I is the set of all outages. Thus, E represents the set of all events present in the database 102.

[0046] Next, we define a location function (Definition 2.2) that relates a set of events E to a set of locations L. In particular, let loc:E→L be a location function whose input is a particular event e and whose output is a particular location l where the event e occurs. In this and other definitions, e is any particular event such that e∈E, L is the set of all locations, and l is any particular location such that l∈L.

[0047] Similarly, we define a timestamp function (Definition 2.3) that associates a set of events E with the times at which they occur. In particular, let ts:E→t be a timestamp function whose input is a particular event e and whose output is the timestamp t when event e occurs. In this and other definitions, t is such that Any timestamp.

[0048] The specific time t and the specific location l at which any specific event e occurs can be uniquely identified together with both the location function loc and the timestamp function ts.

[0049] Using these functions, we define an event-location set consisting of all events that have occurred at a given location (Definition 2.4). As used in this paper, an “event-location set” is a sequence of timestamps of events that have occurred at a particular location. In particular, let H l is the set of all events that have occurred at a specific location l, and it is defined as:

[0050] H l :={e∈E|loc(e)=l}.

[0051] However, the event-location set contains all events that have ever occurred at that location. In order to discretize this set into a timeline, a time interval is defined (Definition 2.5). In particular, let the time interval [t1, t2] be the set of all times or timestamps between and including the start time t1 and the end time t2, which is defined as:

[0052]

[0053] Similarly, the half-open time interval and the open time interval are defined as:

[0054]

[0055] Next, an event-location timeline is defined (Definition 2.6) as referring to all events that have occurred at a given location before a fixed time. In particular, let the timeline is the set of all events that have occurred at position l in the time interval (-∞, t2] (i.e., at time t2 or before time t2). Event-position timeline is defined as:

[0056]

[0057] It should be appreciated that this event-location timeline is specific to a single location and provides a way to represent the past of a location from a given point in time. Figure 3An example visualization 200 of an event-location timeline including event data between exemplary dates January 21, 2022 and January 31, 2022 is shown. In the example timeline 200, each box represents an abnormal event (indicated by "ANO") or an outage event (indicated by "INT"). The timeline 200 captures the temporal relationships between events from various data sets in a single view, such as certain anomaly groups typically appear before outages occur.

[0058] However, given a sequence of events, the function that predicts whether a disruption will occur is not unique to a single location. Instead, the function will predict disruptions across multiple locations. Therefore, the set of all event-location timelines is defined (Definition 2.7) to account for all timelines at all locations. In particular, let is the set of all timelines at all positions l, and it is defined as:

[0059]

[0060] in, is the event-location timeline at each specific location l at time t.

[0061] Next, define a predictor function (Definition 2.8) for predicting whether a disruption will occur soon (i.e., within a first threshold amount of time A) at a particular location. In particular, let predict:H→[0,1] be a predictor function whose input is the event-location timeline H at a particular location l at a particular time t, and whose output is the probability that a disruption will occur soon at that location. As defined, the predictor function makes the implicit assumption that the probability of a disruption occurring soon at location l at time t depends only on the event-location timeline seen at that location l at that time t.

[0062] The goal of the training process of the machine learning model 124 is to learn a predictor function. The output of the prediction is constrained between 1 (it is very likely that an interruption will occur) and 0 (it is very likely that an interruption will not occur). One of the reasons why learning predictions is challenging is because it currently looks at the entire timeline extending from time t back to when data collection first began. However, based on input from domain experts, a simplifying assumption can be made that only recent events predict the probability of an interruption occurring soon. Therefore, when making a prediction of whether an interruption will occur soon, all events that occurred before the monitored window duration M can be ignored. Therefore, an event window between time t (the time of interest) and tM can be defined to simplify the task of interruption prediction.

[0063] The event window (Definition 2.9) is defined to constrain the event-position set to a specific finite time interval. In particular, let is a window of events occurring at position l and in time interval (t1, t2], which is defined as:

[0064]

[0065] Using this definition of the event window, and using a specific monitoring window duration M (e.g., 7 days), the event monitoring window is defined for the task of interruption prediction: Finally, a simplified predictor function is defined (Definition 2.10) that exploits this event monitoring window Rather than the entire event location timeline In particular, is a simplified predictor function that takes an event window with duration M at some location l and time t and returns the probability that an interruption will occur soon at that location l. is the set of all possible windows of events of duration M and is defined as:

[0066]

[0067] Given an event monitoring window The output of the simp_predict function can be interpreted as a conditional probability that an outage will occur. As discussed in more detail below, the machine learning model 124 is trained to approximate the simplified predictor function simp_predict using positive and negative training samples generated from the event data 160 collected from the monitored system 104. The machine learning model 124 can take a variety of forms suitable for binary classification problems, such as logistic regression, SVM, LDA, random forest, and XGBoost.

[0068] Methods for training and using models to predict events

[0069] Various operations and processes for operating the computing device 100 to train a machine learning model using historical data from the event data stream 106 of the monitored system 104 and to utilize the trained machine learning model 124 to predict the probability of future events that may occur in the monitored system 104 based on real-time data of the event data stream 106 are described below. In these descriptions, statements that a method, processor, and / or system is performing a task or function refer to a controller or processor (e.g., the processor 110 of the computing device 100) executing programmed instructions stored in a non-transitory computer-readable storage medium (e.g., the memory 120 of the computing device 100) operably connected to the controller or processor to manipulate data or operate one or more components in the computing device 100 or database 102 to perform the task or function. In addition, the steps of the method may be performed in any feasible chronological order, regardless of the order shown in the figures or the order in which the steps are described.

[0070] To better understand the problems that may be solved using event prediction, these methods are described with reference to an illustrative example in which the monitored system 104 is a manufacturing plant with multiple machines, various events may occur with respect to these machines, and these machines are monitored to generate event data. In the illustrative example, the method is used to predict interruptions in the manufacturing plant based on anomalies that have already occurred in the manufacturing plant. However, it should be recognized that the systems and methods described herein may be applied to any field, and references to the manufacturing field and its terminology herein should be understood to be exemplary only.

[0071] Figure 4 A flow chart of a method 300 for training a machine learning model to predict future events in a system is shown. The method 300 advantageously converts the event data stream received from the monitored system 104 into a number of positive training samples and negative training samples for a binary classification problem. This helps to effectively transform the event sequence prediction task into a simpler classification problem, which is relatively easier to solve using a machine learning algorithm. In addition, the process of generating positive and negative training is a procedural process that does not require manual human-annotation of the training data. Therefore, the machine learning model 124 can be trained on real-world data from the monitored system 104 without the need to manually label historical data from the system 104.

[0072] The method 300 begins by receiving a plurality of event data from a system (block 310). In particular, the processor 110 receives (one or more) event data streams 106 from the monitored system 104. As discussed above, the (one or more) event data streams 106 include at least data indicating events that occurred in the monitored system 104 and a timestamp indicating the time when the event occurred. In addition, as discussed above, the (one or more) event data streams 106 may further include certain metadata. In at least some embodiments, the processor 110 stores the events from the (one or more) event data streams 106 in the database 102 (i.e., event data 160).

[0073] The method 300 continues by determining a plurality of event sequences (block 320). In particular, the processor 110 determines a plurality of event sequences or more particularly a plurality of event windows at all locations L based on the event data 160. As will be discussed in more detail below, multiple event windows Each event window in is a candidate for generating a positive training sample or a negative training sample. However, it should be recognized that in some embodiments, multiple event windows at all positions L are determined. The steps of FIG. 1 and FIG. 2 are performed simultaneously or in combination with the process for forming positive training samples and negative training samples described below, and the description should not be understood as requiring a specific order of operations or excluding feasible alternative order of operations.

[0074] Events in multiple event windows Each window includes events that occur at the same corresponding location l within the system 104 and during a corresponding time window / time interval [t1, t2]. In at least some embodiments, each corresponding time window / time interval (t1, t2] has the same length equal to the monitoring window duration M.

[0075] To determine multiple event windows The processor 110 first determines a plurality of event-location sets or more specifically multiple event-location timelines Each event-location timeline The event data 160 includes all events that occurred at the same corresponding location l in the system 104 until the current time t. Next, the processor 110 generates a timeline for each event-location Determine the same corresponding position l but in different corresponding time windows / intervals (t a ,t b The processor 110 forms a plurality of event windows at all positions L according to the subset of the event sequence occurring at each position l.

[0076] In some embodiments, for each event-location timeline The processor 110 determines a subset of the event window occurring at the corresponding location l using a sliding window or step-wise scanning method. Figure 6 A step-by-step scan of the event-location set 500 is shown. The processor 110 starts from the minimum timestamp To the maximum timestamp Scan event-location collection 500 (H l ). Used in H l The minimum and maximum timestamps for generating event windows in are and In the illustration, anomalies are shown as white circles on the event-location set 500, and interruptions are shown as black circles on the event-location set 500. The processor 110 generates an event-location set 500 by defining a time interval spanning the event-location set 500. At step 2, the processor 110 defines a second event window by sliding the time interval to the right by a duration that is a fraction α times the duration of the monitoring window M. Thus, the next time window / time interval becomes And the second event window is defined across the time window / time interval. In one embodiment, the score α is between 0 and 1 and is selected by cross validation. The processor 110 repeats the process for steps 3, 4, 5, 6, etc. until The processor 110 repeats this process for each location l in the system 104 to define multiple event windows.

[0077] Each training sample (positive or negative) will include (1) a corresponding location l defined within the system 104 and a corresponding time window / time interval (t a ,t b ] The corresponding event window that occurred during data, and (2) indicates whether a particular type of event (e.g., an interrupt) occurs in the corresponding time window (t a ,t b ] within a first threshold amount of time A (e.g., 7 days) after ]. However, in order to apply machine learning, in at least some embodiments, the processor 110 converts multiple event windows into is converted into feature data that can be processed by the machine learning model 124. In particular, for multiple event windows For each corresponding event window in the training data (or at least those used as training samples), the processor 110 extracts feature data from the corresponding event window and forms a feature vector based on the feature data. The feature vector is data that defines or characterizes the corresponding event window and is provided as input to the machine learning model 124 during training.

[0078] From the Event Window And / or the feature data extracted from each event therein may include any of various features that may be beneficial to the classification task (e.g., n-grams, pattern-based features, timestamps, etc.). In one embodiment, the processor 110 uses a unigram feature extraction (bag of unigrams) method to generate features. In particular, for each corresponding event type in the multiple event types of the event data 160, the processor 110 determines the corresponding count of how many events of the corresponding event type occurred during the corresponding event sequence. These counts of each event type in the corresponding event window are combined to form a feature vector that defines or characterizes the corresponding event window. In this method, a feature vocabulary is defined according to a set of all unique values ​​of the event type field (e.g., each type of anomaly) in the event data 160. In some embodiments, since the abnormal event has no type field, a parameter field is used instead to represent the type. The parameter field can be used to represent the type of anomaly because it represents which sensor measurements at this location have seen anomalies. It is assumed that the anomaly of certain parameters will help detect interruptions. Therefore, each unique anomaly type (i.e., a unique value in the parameter field) is represented as a feature, and its value is the count of the anomaly type in the event window.

[0079] Figure 7 An exemplary feature extraction for a positive training sample is shown. In particular, the event-location timeline 600 includes anomalies of type 1, 2, 3, and 4. For the exemplary event window, there is one anomaly of type 1, zero anomalies of type 2, three anomalies of type 3, and zero anomalies of type 4. Therefore, the processor 110 forms a feature vector 602 of length 4 and having values ​​[1, 0, 3, 0]. However, it should be appreciated that the feature vector 602 can incorporate any number of additional features extracted from the event window.

[0080] return Figure 4 , the method 300 continues to generate a plurality of positive training samples (block 330). In particular, in order to generate positive training samples (ie, feature label pairs), the processor 110 determines a plurality of event windows Which event windows in may be marked with a classification label c=1, which indicates that a particular type of event (e.g., an outage) occurs within a first threshold amount of time A (e.g., 7 days) after the event window. In at least some embodiments, processor 110 generates positive training samples first because outages are rare. Figure 7 In the example of , the interruption occurs after the event window, and therefore the processor 110 determines that the classification label 604 should be equal to 1, and forms a positive training sample from the feature vector and the classification label.

[0081] More generally, the processor 110 is a plurality of event windows. Each corresponding event window in Determine whether an event indicating a specific type (e.g., an interruption) occurs within a corresponding time window / time interval (t a ,t b ] after the first threshold amount of time A. For multiple event windows Each corresponding event window in In response to indicating that an event of a particular type is within a corresponding time window / time interval (t a ,t b ], that is, whenever c=1, the processor 110 selects the corresponding classification label c from the corresponding event window and the corresponding classification label c form positive training samples.

[0082] Reference again Figure 6 , the processor 101 first generates positive samples by looking at the event windows that precede each interruption. In particular, assuming an interruption occurs at a certain location l at time t (black circle in the illustration), the processor 110 considers event windows in a subset of event windows at the location that precedes the interruption within a first threshold amount of time A. Such event windows are labeled as positive training samples 502, and the corresponding feature vectors are paired with the classification label c=1 to form positive training samples. It should be appreciated that although only one is shown for each interruption, multiple different previous event windows can be used to form multiple different positive training samples based on one interruption.

[0083] In some embodiments, processor 110 takes into account event windows Where W is a second threshold amount of time before the interruption. If the candidate event window precedes the interruption by less than the threshold amount of time W, the processor 110 ignores the event window in generating positive training samples. The second threshold amount of time W is a non-zero value (e.g., 30 minutes) selected so that the machine learning model 124 does not learn features that are very close to the interruption. In other words, in some embodiments, for a given event window, the processor 110 forms a positive training sample from the event window only when a particular type of event (e.g., an interruption) occurs at a time t that is at least a second threshold amount of time W after the corresponding time window / time interval but also within a first threshold amount of time A after the corresponding time window / time interval. The reason for ignoring the second threshold amount of time W is so that the machine learning model 124 will not learn patterns that only become apparent in the case of a very close interruption. While these may help improve the accuracy of the predictions, the predictions made will be of little use because the operator has little time available to react to the predictions.

[0084] For each positive training sample, the processor 110 also records the timestamp t when the associated subsequent interruption occurs. As will be discussed below, this timestamp t will be used to divide the training sample into a training set and a validation set. For each position l, the processor 110 also records all windows for which the positive training sample was created. As will be discussed below, this is used to avoid overlap between the positive training sample and the negative training sample at each position l.

[0085] The method 300 continues by generating a plurality of negative training samples (block 340). In particular, to generate negative training samples (ie, feature-label pairs), the processor 110 determines the event window Which event windows in should be marked with a classification label c=0, which indicates that a particular type of event (e.g., an outage) has not occurred within a first threshold amount of time (e.g., 7 days) after the event window. In some embodiments, for multiple event windows Each corresponding event window in In response to indicating that an event of a particular type has not occurred within a corresponding time window / time interval (t a ,t b ], that is, whenever c=0, the processor 110 selects the corresponding classification label c from the corresponding event window and the corresponding classification label c form negative training samples.

[0086] In some embodiments, when the classification label c=0 for a particular event window, the processor 110 only forms negative training samples from the event window if the event window does not overlap with any event window at the same position l as any positive training sample. In other words, for multiple event windows Each corresponding event window in In response to (1) indicating that an event of a particular type has not occurred within a corresponding time window / time interval (t a ,t b ] and (2) the corresponding time window / time interval (t a ,t b ] does not overlap with the corresponding time window of any one of the multiple positive training samples at the same position l, the processor 110 selects the corresponding event window and the corresponding classification label c form negative training samples. For this reason, in at least some embodiments, the processor 110 generates negative training samples only after generating positive training samples.

[0087] Reference again Figure 6 After generating the positive training samples, the processor 101 generates negative training samples by looking at event windows that do not precede the interruption by a first threshold amount of time A. If such event windows do not overlap with any positive training samples 502 in the same event-location timeline 500, they are marked as negative training samples 504, and the corresponding feature vectors are paired with a classification label c=0 to form negative training samples. However, if the event window does not precede the interruption by a first threshold amount of time A, but does overlap with one of the positive training samples 502 in the same event-location timeline 500, such event windows are marked as rejected training samples 506, or are otherwise simply ignored.

[0088] In some embodiments, the processor 110 and the processor 110 are connected according to each event-location set H. l In particular, in some embodiments, the processor 110 iteratively defines a time window, determines a classification label, generates a positive training sample or a negative training sample from the time window or ignores the time window, and then slides the window to define the next time window, repeating the process until reaching In such an embodiment, some efficiencies may be found. For example, in some embodiments, during a scan of each event-location timeline, if the current time window overlaps with a positive training sample, the processor 110 slides the time window to the end of the overlapping positive training sample. Figure 6 , it can be seen that between steps 3 and 4, the window is slid to the end of the leftmost positive training sample 502, thereby effectively skipping the middle window that would also overlap with the leftmost positive training sample 502.

[0089] The method 300 continues to train the machine learning model using the plurality of positive training samples and the plurality of negative training samples (block 350). In particular, once the plurality of positive training samples and the plurality of negative training samples have been generated, the processor 110 trains the machine learning model 124 to learn a simplified predictor function simp_predict based on the plurality of positive training samples and the plurality of negative training samples. More particularly, given a monitored event window at a particular location l Processor 110 trains machine learning model 124 to determine a predicted value for a classification label c. The output of machine learning model 124 is a value between 0 and 1 indicating the probability that a particular type of event (e.g., an outage) will occur within a first threshold amount of time A (e.g., 7 days) after a monitored time window (tM, t].

[0090] In at least one embodiment, the processor 110 trains the machine learning model 124 in a supervised manner by feeding the input feature vector of each training sample into the machine learning model 124 and determining the training loss based on the comparison with the associated classification label c. Based on each training sample, the parameters of the machine learning model 124 (e.g., kernel weights, model coefficients, etc.) are updated and improved until the machine learning model 124 achieves satisfactory performance. In one embodiment, the processor 110 uses a hyperopt library to optimize the model parameters.

[0091] Figure 5 A flow chart of a method 400 for predicting possible future events in a system is shown. The method 400 advantageously utilizes a machine learning model 124 trained using the methods discussed above to enable accurate and real-time prediction of future events of a particular type (e.g., an outage) in a system. In this way, operators of such a system can be warned before such an event occurs and have the opportunity to resolve problems in the system that otherwise might result in undesirable outcomes, such as an outage.

[0092] The method 400 begins by receiving a plurality of event data from a system (block 410). In particular, the processor 110 receives (one or more) event data streams 106 from the monitored system 104. As discussed above, the (one or more) event data streams 106 include at least data indicating events that occurred in the monitored system 104 and a timestamp indicating the time when the event occurred. In addition, the (one or more) event data streams 106 may further include certain metadata, as discussed above. In at least some embodiments, the processor 110 stores the events from the (one or more) event data streams 106 in the database 102 (i.e., event data 160).

[0093] Method 400 continues by determining, for each location in the system, a corresponding sequence of events that have occurred at the corresponding location during the monitored time window (block 420). In particular, for some or all of the corresponding locations l∈L in system 104, processor 110 determines a sequence of events, or more particularly a corresponding monitoring window of events, using, for example, Definition 2.9. It includes all events that occur at the corresponding location l during the monitored time window (e.g., (tM, t]). Each monitored event window will be used as a basis for predicting whether a particular type of event (eg, an interruption) will occur at the corresponding location l within a first threshold amount of time A after the monitored time window (eg, (tM, t]).

[0094] Method 400 continues by using a machine learning model to predict whether a future outage will occur at a particular location based on the features of the formed event sequence (block 430). The processor 110 uses the trained machine learning model 124 to predict whether a particular type of event (eg, an interruption) will occur at the corresponding location l within a first threshold amount of time A after a monitored time window (eg, (tM, t]).

[0095] More specifically, in at least some embodiments, for each monitored event window The processor 110 extracts features therefrom and forms a feature vector using the same process as described above for generating positive training samples and negative training samples. The processor 110 feeds the corresponding feature vector into the machine learning model 124 as input and determines the output of the machine learning model 124 based on the feature vector.

[0096] In at least some embodiments, the output of the machine learning model 124 is a probability between 0 and 1 that a particular type of event (e.g., an interruption) will occur at the corresponding location l within a first threshold amount of time A after the monitored time window (e.g., (tM, t]). To convert the probability into a prediction, the processor 110 compares the output probability with a detection threshold T. If the probability exceeds the detection threshold T, the processor 110 predicts that the particular type of event (e.g., an interruption) will occur at the corresponding location l within a first threshold amount of time A after the monitored time window (e.g., (tM, t]). In other words, the prediction process (Definition 2.11) can be defined as follows. For any given location l and corresponding event window whenever Processor 110 then outputs a prediction that an interrupt will occur. The value of T can be adjusted to achieve a desired tradeoff between precision and recall.

[0097] In at least some embodiments, once the system outputs a prediction, the prediction remains active for a certain period of time. The time period P is an adjustable parameter, and in some embodiments, the time period P is set to be equal to the first threshold time amount A used in the training process discussed above. The reason for having an effective prediction period is duality. First, since the machine learning model 124 is trained to predict binary classification problems, the system does not know exactly when the interruption will occur. Therefore, the duration when the prediction remains valid provides a time interval for the time when the expected interruption occurs. Secondly, if a valid prediction has been made for the location, the system can avoid creating repeated predictions at the same location. Finally, the system advantageously provides sufficient warnings to the operator, which gives the operator enough time to respond to the prediction before the failure occurs. As discussed above, this minimum warning duration (e.g., 30 minutes) is represented as W.

[0098] The method 400 continues by outputting an alarm in response to predicting that an outage will occur in the future at one of the locations (block 440). In particular, in response to predicting that a particular type of event (e.g., an outage) will occur at a corresponding location l within a first threshold amount of time A after a monitored time window (e.g., (tM, t]), the processor 110 perceptibly outputs an alarm via some user interface. In some embodiments, the alarm is provided to an operator of the monitored system 104 and indicates at least the location l where the event is predicted to occur. In some embodiments, the processor 110 operates a display device or otherwise causes a display device to be operated to display a graphical alarm on a display screen. In some embodiments, the processor 110 operates a speaker or otherwise causes a speaker to be operated to display a graphical alarm on a display screen. In some embodiments, the processor 110 operates the network communication module 150 to transmit a message indicating the prediction to some other device, which causes the other device to perceptibly output an alarm.

[0099] In at least some embodiments, in response to a previous alert having been output within a predetermined amount of time (e.g., P) regarding a previous prediction that a particular type of event (e.g., an interruption) will occur at a corresponding location, the processor 110 suppresses the output of an alert. In other words, if there is already a valid prediction that an event will occur at a particular location l, the processor 110 does not send another alert predicting that the same event will occur at the same particular location l.

[0100] Evaluation Metrics and Cross-Validation

[0101] In addition to methods for training and using machine learning models 124, a simulator-based method for cross-validation of data for time non-independent and identically distributed (non-iid) data containing time-lagged features is disclosed. The method mimics real-world deployment scenarios and provides an estimate of the expected performance of the machine learning model 124 in production. This addresses a significant challenge faced by machine learning practitioners in which the performance of a trained model drops significantly between training and deployment.

[0102] Two methods for evaluation and cross-validation are discussed in this paper. In particular, a preliminary approach is first presented, its shortcomings are then highlighted, and then a second simulator-based approach is presented.

[0103] In the preliminary method, the training data is divided by time. In particular, due to the presence of overlapping windows during the data generation process, the independent and identically distributed assumption of the data samples required for randomized cross-validation does not hold. In addition, due to the time order in which the data is generated, randomized cross-validation will leak future information. Therefore, cross-validation techniques that are more suitable for time series data are more suitable for the present disclosure. In this method, the machine learning model 124 is trained and validated on different folds of the data. However, unlike randomized cross-validation, the selected folds are continuous time blocks.

[0104] Figure 8 shows a custom cross-validation split to avoid feature leakage. In the first fold, for the TRAIN ] to train the machine learning model 124, and about the data generated by the timestamp from (t TRAIN +M,t TRAIN +M+R] timestamp to verify the machine learning model 124, where t TRAIN is the end of the training duration of the first fold, M is the monitoring window duration and R is the model retraining duration. In the second fold, for the models with TRAIN +R] to train the machine learning model 124, and about the data generated by the timestamp from (t TRAIN +M+R,t TRAIN +M+2R] timestamps to validate the machine learning model 124. Similarly, in the kth fold, the generated data with the time interval (-∞, t TRAIN +(k-1)R] to train the machine learning model 124, and TRAIN +M+(k-1)R,t TRAIN +M+kR] timestamp generated data to verify the machine learning model 124.

[0105] Not right TRAIN The reason for validating immediately M days later is that, since the machine learning model 124 will be implemented in production, labels cannot be assigned to data within the past M days, because it is necessary to wait for future data to know whether the data corresponds to an outage. In addition, the machine learning model 124 cannot be validated with respect to M days after the last day in the training set, because the features created for those timestamps will overlap with the features in the training set due to the use of features lagged M days later, resulting in feature leakage. The evaluation metrics used for this preliminary approach are precision, recall, and f-score.

[0106]

[0107] One of the main challenges in deploying machine learning systems in real-world industrial settings is that the performance of the model during experiments does not reflect the actual performance measured in production. The evaluation metrics defined previously suffer from the same problem. This is because in production, the machine learning model 124 is expected to be continuously evaluated as new anomalies and interruption events flow in. However, since our cross-validation samples come from generated data, the positive samples include all the data just before each interruption, while the negative samples include windows far away from the interruption. In real-world testing, the observed data will be somewhere in between the spectrum of positive and negative samples. Therefore, a second simulator-based approach is defined that mimics real-world deployments.

[0108] In the second approach, a simulator-based approach is utilized that mimics a real-world deployment. In this simulator-based approach, the training set segmentation is created in exactly the same manner as discussed with respect to the preliminary approach. However, the manner of verification is different. As mentioned, the problem with verification on generated data is that they are more likely to be correct because they only include windows just before or further away from the interruption. Therefore, to simulate a real-world scenario, whenever a new anomaly occurs at any location, an inference is performed and the processor 110 forms an event window at that location and time. And a prediction is made using the simplified predictor function simp_predict (Definition 2.10) and the prediction threshold T (Definition 2.11). Once such a prediction is made, it can be either a true positive (i.e., a successful prediction) or a false positive. A prediction made for location l at time t is a successful prediction if a disruption occurs at that location in the time interval [t+W, t+P], where W is the minimum warning duration and P is the validity duration of the prediction.

[0109] Fig. 9The conditions under which a prediction is a success or a false positive are shown. Reasoning is performed similarly for all anomalies that occur at all locations within the cross-validation duration. Once the validation step for the current fold is completed, the processor 110 moves to the next fold, retrains the machine learning model 124 on the corresponding training split, and performs the validation step as previously described. The processor 110 repeats this process until the last fold is reached. In the preliminary method described above, each sample is evaluated, while in the simulator method, the processor 110 evaluates the predictions rather than the samples. Therefore, the evaluation metric is modified as follows:

[0110]

[0111] Therefore, the performance of the machine learning model 124 is measured based on whether the machine learning model 24 successfully predicts whether a particular type of event (e.g., an outage) occurs within a first threshold amount of time A after the time window, rather than based on successfully predicting the classification label c of each specific positive and negative validation sample. In the experimental data set, it was found that in some cases, multiple outages will occur very close in time. Using the preliminary method, when multiple true positive samples are correctly predicted, the validation process will see the multiple true positive samples, however, in the simulator method, these outages will only account for one successful prediction because multiple outages will be covered by the same prediction.

[0112] Fig.10 shows a case where two breaks occur in the same prediction interval. This results in a single true positive prediction using the simulator method, rather than two true positive examples using the preliminary method. In the preliminary method, this would result in two true positives, while in the simulator method, this results in only a single true positive. Therefore, the evaluation metric in the preliminary method artificially inflates the expected performance, while the simulator method gives a more reliable estimate of what can be expected in production.

[0113] Interpretability and Explanability

[0114] Explanations of the predictions of the machine learning model 124 using an existing interpretability framework are provided herein. These explanations are useful in helping system operators take corrective actions. In particular, in addition to model accuracy, it is also expected that the predictions are interpretable. This means that the machine operator using the system can understand the factors that drive the predictions of the machine learning model 124. In general, as statistical learning techniques become more complex, their interpretability decreases. However, recent advances in explainable AI have achieved interpretability without sacrificing the accuracy or generality of predictions. One such technique utilized herein is the Shapley additive explanation, which has been applied to various settings. The Shapley additive explanation works well with tree-based models (such as random forests and XGBoost) by returning a Shapley value for each feature that represents the contribution of the feature to a particular prediction. Therefore, the predictions of the machine learning model 124 can be explained by plotting the individual Shapley values ​​for each feature.

[0115] Fig.11 An exemplary explanation showing which events contribute to a predicted disruption. In some embodiments, processor 110 operates a display device or otherwise causes a display device to be operated to display a graphical explanation (e.g., a graph, etc.) that explains which events contribute to a predicted disruption. Fig.11 In , a waterfall plot is shown, where the horizontal axis represents the logarithm of the probability of an interruption occurring. Therefore, a value greater than 0 on the horizontal axis means that the probability of an interruption is greater than 0.5, and vice versa. The vertical axis shows the 14 features that contribute the most to the prediction. Each horizontal bar represents the contribution of that particular feature. The white bars indicate the corresponding features that contribute to the increase in the log odds (and therefore increase the probability of an interruption), while the black bars indicate the corresponding features that contribute to the decrease in the log odds. The longer the length of the bar, the greater the contribution of the feature. E[f(X)] represents the base rate at which an interruption occurs, and f(X) represents the log odds of an interruption occurring when a prediction is made for feature X. The graph shown was generated using the Shapley library. This is an additional step in our existing workflow and does not affect the accuracy of the model. The performance overhead is also minimal.

[0116] Experimental Results

[0117] The system and method disclosed herein are evaluated on a real data set of abnormal-interruption events originating from an actual production line. The results show that the system and method predict faults with moderate accuracy and reduce the downtime of the production line. The machine learning model 124 is trained on data generated from 19 unique locations. The generated data has 368 positive samples and 957 negative samples. Training, validation, and testing are performed on data collected from three different time periods. The machine learning model 124 achieves an F-score of 0.70 on a cross-validation set. Threshold selection is performed by plotting precision-recall curves. Fig.12 A graph showing the precision-recall curve obtained by the results is shown. Since precision is more favored than recall in the evaluation, a threshold T of 0.9 is selected based on the curve. On the cross-validation set, this threshold T gives a precision of 0.71 and a recall of 0.36. On the test set, a precision of 0.76 and a recall of 0.62 are achieved.

[0118] The high variance in performance in the cross validation and test sets is due to the small sample sizes of the cross validation and test sets. Error analysis was also performed to identify where the machine learning model 124 performed poorly. It was found that the performance of the machine learning model 124 degraded at two specific locations. Upon further investigation, the outages at these two locations did not appear to conform to a potential pattern of predictable anomalies or outages. Therefore, the machine learning model 124 did not accurately predict outages at these locations. However, if these two locations were excluded, the F-score on the cross validation set increased to 0.74.

[0119] The distribution of the generated data across different locations is also examined. Fig.13 A visualization of the data generated across different locations using a t-SNE plot is shown. It is found that the data generated at certain locations form distinct clusters. The reason for this is that these locations have similar workstation and tool positioning, but belong to different lines within the factory. The two locations that our model did not accurately predict also clustered into two clusters numbered 4 and 14 in the t-SNE plot. These insights allow the machine learning model 124 to be curated to specific types of locations, as the underlying distribution can be very different for different locations. At the same time, a large number of locations were not separated into clearly defined clusters, yet the machine learning model 124 was able to accurately predict outages at these locations.

[0120] Embodiments within the scope of the present disclosure may also include non-transitory computer-readable storage media or machine-readable media for carrying or having computer-executable instructions (also referred to as program instructions) or data structures stored thereon. Such non-transitory computer-readable storage media or machine-readable media may be any available medium that can be accessed by a general or special-purpose computer. As an example and without limitation, such non-transitory computer-readable storage media or machine-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to carry or store the required program code device in the form of computer-executable instructions or data structures. The combination of the above should also be included in the scope of non-transitory computer-readable storage media or machine-readable media.

[0121] Computer executable instructions include, for example, instructions and data that cause a general purpose computer, a special purpose computer, or a special purpose processing device to perform a certain function or group of functions. Computer executable instructions also include program modules that are executed by computers in a stand-alone or network environment. Typically, program modules include routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. Computer executable instructions, associated data structures, and program modules represent examples of program code means for performing the steps of the methods disclosed herein. A specific sequence of such executable instructions or associated data structures represents an example of corresponding actions for implementing the functions described in such steps.

[0122] Although the present disclosure has been illustrated and described in the drawings and the foregoing description, they should be considered illustrative rather than restrictive in nature. It should be understood that only preferred embodiments have been presented, and all changes, modifications, and further applications within the spirit of the present disclosure are desired to be protected.

Claims

1. A method for training a machine learning model to predict future events in a system, the method comprising: receiving, with a processor, event data from the system, the event data indicating an event occurring in the system and a time when the event occurred; determining, with the processor, a plurality of positive training samples based on the event data; determining, with the processor, a plurality of negative training samples based on the event data; Utilizing the processor, the plurality of positive training samples and the plurality of negative training samples are used to train a machine learning model to predict whether a particular type of event will occur at a particular location within a first threshold amount of time after a time window, given a sequence of events that occurred at the particular location during the time window.

2. The method according to claim 1, wherein: Each of the plurality of positive training samples and the plurality of negative training samples comprises: data defining respective sequences of events occurring at respective locations within the system and during respective time windows; and A classification label indicating whether the particular type of event occurred at the corresponding location within a first threshold amount of time after the corresponding time window.

3. The method according to claim 2, further comprising, for each corresponding training sample in the plurality of positive training samples and the plurality of negative training samples: Extracting feature data from the corresponding event sequence; and forming a corresponding feature vector according to the feature data extracted from the corresponding event sequence, in, The data defining the respective event sequence of the respective training sample comprises the respective feature vector.

4. The method according to claim 3, wherein: The event indicated in the event data is each of a plurality of event types, and extracting feature data from the corresponding event sequence further comprises: for each respective event type of the plurality of event types, determining a respective count of how many events of the respective event type occurred at the respective location during the respective event sequence, The corresponding feature vector includes a corresponding count for each of the multiple event types.

5. The method according to claim 1, further comprising: A plurality of event sequences are determined, each of the plurality of event sequences comprising events occurring at a same respective location within the system and during a respective time window.

6. The method of claim 5, wherein determining the plurality of event sequences further comprises: Based on the event data, determining a plurality of event-location sets, each event-location set comprising all events in the event data that occur at a same corresponding location within the system; as well as For each event-location set in the plurality of event-location sets, determining a corresponding subset of event sequences, each event sequence in the corresponding subset of event sequences comprising events occurring at the same corresponding location within different corresponding time windows, The plurality of event sequences include a corresponding subset of the event sequences for each event-location set of the plurality of event-location sets.

7. The method of claim 6, wherein determining a corresponding subset of the sequence of events further comprises: By sliding the time window by a predetermined time interval, a different corresponding time window is determined for each event sequence in the corresponding subset of the event sequences.

8. The method according to claim 5, further comprising: A respective classification label is determined for each respective event sequence of the plurality of event sequences, the respective classification label indicating whether the particular type of event occurs at the respective location within a first threshold amount of time after the respective time window.

9. The method according to claim 8, wherein determining the plurality of positive training samples further comprises, for each corresponding event sequence in the plurality of event sequences: In response to the respective classification label indicating that the particular type of event occurred at the respective location within a first threshold amount of time after the respective time window, forming a positive training sample of the plurality of positive training samples based on the respective event sequence and the respective classification label.

10. The method according to claim 9, wherein the determining the plurality of positive training samples further comprises, for each corresponding event sequence in the plurality of event sequences: In response to (i) the corresponding classification label indicating that the specific type of event occurs at the corresponding location within a first threshold amount of time after the corresponding time window, and (ii) the specific type of event occurs at the corresponding location after a second threshold amount of time after the corresponding time window, a positive training sample of the multiple positive training samples is formed according to the corresponding event sequence and the corresponding classification label.

11. The method according to claim 8, wherein determining the plurality of negative training samples further comprises, for each corresponding event sequence in the plurality of event sequences: In response to the corresponding classification label indicating that the particular type of event did not occur at the corresponding location within a first threshold amount of time after the corresponding time window, a negative training sample of the plurality of negative training samples is formed based on the corresponding event sequence and the corresponding classification label.

12. The method according to claim 11, wherein determining the plurality of negative training samples further comprises, for each corresponding event sequence in the plurality of event sequences: The negative training sample is formed in response to (i) the corresponding classification label indicating that the specific type of event did not occur at the corresponding location within a first threshold amount of time after the corresponding time window, and (ii) the corresponding time window does not overlap with the corresponding time window of any one of the multiple positive training samples at the same corresponding location.

13. The method according to claim 1, further comprising: The machine learning model is validated using a simulator-based approach, in which performance is measured based on whether the machine learning model successfully predicts whether the particular type of event occurs within a first threshold amount of time after the time window.

14. A method for predicting possible future events in a system, the method comprising: receiving, with a processor, event data from the system, the event data indicating an event occurring in the system and a time when the event occurred; determining, with the processor, a sequence of events, the sequence of events comprising events occurring at corresponding locations within the system and during a time window prior to a current time; using, with the processor, a machine learning model to predict, based on the sequence of events, whether a particular type of event will occur at the corresponding location within a first threshold amount of time after the time window; and In response to predicting that the particular type of event will occur at the respective location within a first threshold amount of time after the time window, an alert is perceptibly outputted using a user interface.

15. The method according to claim 14, wherein the prediction further comprises: extracting feature data from the corresponding event sequence; forming a feature vector according to the feature data; as well as The feature vector is fed into the machine learning model.

16. The method according to claim 14, wherein: The event indicated in the event data is each of a plurality of event types, and extracting feature data from the corresponding event sequence further comprises: for each respective event type of the plurality of event types, determining a respective count of how many events of the respective event type occurred at the respective location during the respective event sequence, The feature vector includes a corresponding count for each of the plurality of event types.

17. The method according to claim 14, wherein the prediction further comprises: determining, using the machine learning model, a probability that the particular type of event will occur at the corresponding location within a first threshold amount of time after the time window; as well as The probability is compared to a predetermined detection threshold.

18. The method of claim 17, wherein the perceptibly outputting further comprises: In response to the probability exceeding the predetermined detection threshold, the alarm is perceptibly output.

19. The method according to claim 14, wherein: The determining the sequence of events further comprises: determining, for each respective location of a plurality of different locations within the system, a respective sequence of events, the respective sequence of events comprising events occurring at the respective location within the system and during the time window; and The predicting further includes determining, for each respective location of a plurality of different locations within the system, whether the particular type of event will occur at the respective location within a first threshold amount of time after the time window.

20. The method of claim 14, wherein the outputting further comprises: In response to a previous alert having been output within a predetermined amount of time regarding a previous prediction that the particular type of event will occur at the respective location, output of the alert is suppressed.

Images