Passive Inference of Signal Tracking in Multivariate Anomaly Detection

Abstract

Systems, methods, and other embodiments related to passive inference of signal following in multivariate anomaly detection are described. In one embodiment, a method for inferring signal following in a machine learning (ML) model includes calculating a mean standard deviation of measurements of time series signals in a set of time series signals, training an ML model to predict values ​​of the signals, predicting values ​​of each of the signals with the trained ML model, generating a time series set of residuals between the predicted values ​​and the measurements, calculating a mean standard deviation of the set of residuals, determining that signal following exists in the trained ML model if a ratio of the mean standard deviation of the measurements to the mean standard deviation of the set of residuals exceeds a threshold, and presenting an alert indicating that signal following exists in the trained ML model.

Description

Background Art

[0001] Background For monitoring the operation of a variety of assets such as data centers, airliners, and oil refineries, a number of sensors can be used. The time-series sensor data or signals from the sensors can be used for machine learning (ML) time-series predictive monitoring to detect an incipient failure of the asset under monitoring before a failure occurs. ML time-series prediction typically involves training an ML model to learn the correlation between time-series signals of the system under monitoring, using the trained ML model to predict the "expected," "normal," or "correct" values of the time-series signals, and issuing an alarm for the deviation between the observed signal behavior and the predicted signal behavior. This enables corrective measures to be taken in response to the alarm before the cause of the abnormal signal behavior leads to a failure of the asset under monitoring.

[0002] ML time-series prediction is subject to an undesirable and potentially dangerous phenomenon called "Following" or "Signal Following." Following occurs when an anomaly appears in the signal under monitoring, but instead of predicting a value that reflects the expected behavior of the signal, the trained ML model predicts a value that matches the observed behavior of the signal. Thus, the ML algorithm generates a predicted value that "follows" the anomaly that accidentally matches the abnormal behavior of the monitoring signal. When following occurs, the difference between the ML prediction and the actual measurement signal is small, and the anomaly does not generate an alarm. Following is costly and dangerous because it results in alarms being missed in the use case of prediction. Missed alarms can lead to catastrophic consequences such as a sudden uncontrollable failure of the asset under monitoring without a prior warning from the ML predictive monitoring system. The following phenomenon can occur using any type of ML algorithm under certain conditions described herein.

Summary of the Invention

[0003] Summary In one embodiment, a method executed by a computer for inferring signal following within a machine learning model, the method comprising calculating an average standard deviation of measured values of time series signals within a set of multiple time series signals, training a machine learning model to predict values of the signals, predicting values of each of the signals with the trained machine learning model, generating a time series set of residuals between the predicted values and the measured values for each of the signals, calculating an average standard deviation of the set of residuals, determining that signal following exists within the trained machine learning model when a ratio of the average standard deviation of the measured values to the average standard deviation of the set of residuals exceeds a threshold, and presenting an alert indicating that signal following exists within the trained machine learning model.

[0004] In one embodiment, the method further comprises calculating the ratio by dividing the average standard deviation of the measured values by the average standard deviation of the set of residuals, and the threshold that the ratio exceeds is between 1 and 1.5.

[0005] In one embodiment, the method further comprises retrieving a quantitative following degree of the machine learning model based on the value of the ratio.

[0006] In one embodiment, the training of the machine learning model and the prediction of the values are performed only once when inferring signal following within the machine learning model.

[0007] In one embodiment, the machine learning model is a non-linear non-parametric regression model.

[0008] In one embodiment, a non-transitory computer-readable medium includes computer-executable instructions stored on the non-transitory computer-readable medium for inferring signal tracking within a machine learning model. When the computer-executable instructions are executed by at least a processor of the computer, the computer is caused to calculate an average standard deviation of measured values of time-series signals within a set of multiple time-series signals, train a machine learning model to predict values of the signals, predict values of each of the signals with the trained machine learning model, generate a time-series set of residuals between the predicted values and the measured values for each of the signals, calculate an average standard deviation of the set of residuals, determine that signal tracking exists within the trained machine learning model when a ratio of the average standard deviation of the measured values to the average standard deviation of the set of residuals exceeds a threshold, and present an alert indicating that signal tracking exists within the trained machine learning model.

[0009] In one embodiment, the instructions further cause the computer to calculate the ratio by dividing the average standard deviation of the measured values by the average standard deviation of the set of residuals, and the threshold that the ratio exceeds is between 1 and 1.5.

[0010] In one embodiment, the instructions further cause the computer to search for a quantitative tracking degree of the machine learning model based on the value of the ratio.

[0011] In one embodiment, the training of the machine learning model and the prediction of values are performed only once when inferring signal tracking within the machine learning model.

[0012] In one embodiment, the machine learning model is a multivariate state estimation technology model.

[0013] In one embodiment, a computing system includes a processor, a memory operably connected to the processor, and a non-transitory computer-readable medium. The non-transitory computer-readable medium is operably connected to the processor and the memory and stores computer-executable instructions for inferring signal tracking within a machine learning model. When the computer-executable instructions are executed at least by the computer's processor, the computing system is caused to calculate an average standard deviation of measured values of time-series signals within a set of multiple time-series signals, train a machine learning model to predict values of the signals, predict values of each of the signals with the trained machine learning model, generate a time-series set of residuals between the predicted values and the measured values for each of the signals, calculate an average standard deviation of the set of residuals, determine that signal tracking exists within the trained machine learning model when a ratio of the average standard deviation of the measured values to the average standard deviation of the set of residuals exceeds a threshold, and present an alert indicating that signal tracking exists within the trained machine learning model.

[0014] In one embodiment, the instructions further cause the computing system to calculate a ratio by dividing the average standard deviation of the measured values by the average standard deviation of the set of residuals, and the threshold that the ratio exceeds is between 1 and 1.5.

[0015] In one embodiment, the instructions further cause the computing system to search for a quantitative tracking degree of the machine learning model based on the value of the ratio.

[0016] In one embodiment, the training of the machine learning model and the prediction of values are performed only once when inferring signal tracking within the machine learning model.

[0017] In one embodiment, the alert includes a ratio, increasing the number of training vectors in the ML model, filtering the signal input to the ML model to reduce noise, and increasing the number of signals, and recommendations for techniques to reduce signal tracking in the machine learning model. A computing system.

[0018] The accompanying drawings are incorporated herein and constitute a part of this specification, and illustrate various systems, methods, and other embodiments of the present disclosure. It will be understood that the exemplary element boundaries in the figures (e.g., boxes, groups of boxes, or other shapes) represent one embodiment of the boundary. In some embodiments, one element may be implemented as multiple elements, or multiple elements may be implemented as one element. In some embodiments, an element shown as an internal component of another element may be implemented as an external component, and vice versa. Further, the elements may not be drawn to scale.

Brief Description of the Drawings

[0019]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0020] Detailed Description Systems and methods for passive inference of signal tracking in multivariate anomaly detection are described herein.

[0021] An undesirable phenomenon called "tracking" or "signal tracking" can negatively affect the ability of any type of non-linear non-parametric (NLNP) regression algorithm (collectively, the "ML anomaly detection" algorithm) used for multivariate anomaly detection, including neural networks (NNs), support vector machines (SVMs), auto-associative kernel regression (AAKR), and similarity-based modeling (SBM) such as multivariate state estimation technology (MSET) (including MSET2 owned by Oracle). Tracking is a problem for any time series anomaly detection ML. This has been verified for all of the major regression-based time series anomaly detection algorithms listed above. The term "tracking" stems from the observation that, particularly in small ML models with a poor signal-to-noise ratio, due to the complex interactions of signal dependencies, the ML estimation "tracks" the signal degradation and incorrectly predicts values that partially or fully mimic the degradation. Anomaly in time series prediction applications is usually found when the actual signal (i.e., the observed signal) deviates from the model estimation. When the model estimation tracks the measured signal due to the tracking phenomenon, the degradation is not detected, which is dangerous in safety-critical industries and can be costly in industries where undetected anomalies can lead to catastrophic failures.

[0022] Generally, the tendency to follow is the result of one or more of: (i) too few signals in the ML model, (ii) a measurement noise component that is too large superimposed on the deterministic structure of the signal, or (iii) too few correlations between the monitoring signals. This has been empirically verified. Thus, when the presence of the following phenomenon is detected in the model by implementing the systems and methods herein, the systems and methods further assist the user in taking measures to modify the model by: (i) adding additional signals into the model, (ii) reducing the noise level (e.g., by implementing a smoothing filter), and / or (iii) adding additional correlated signals into the model, checking the modified model to determine whether the changes to the model reduce the following, and re-checking the model for following in the final model verification to show that the following has been removed.

[0023] Original following metric as a benchmark for inferential following metrics The original technique of trace characterization for analyzing an ML anomaly detection algorithm to determine the susceptibility of an ML model to its influence was developed as an initial solution to the problem of trace. This original trace characterization technique is described in more detail in U.S. Patent Application No. 17 / 086,855, filed on November 2, 2020, by inventors Zexi CHEN, Kenny GROSS, Ashin GEORGE, and Guang WANG, entitled "Characterizing Susceptibility of a Machine-Learning Model to Follow Signal Degradation and Evaluating Possible Mitigation Strategies," the entire disclosure of which is hereby incorporated by reference in its entirety. The original trace characterization technique is a useful tool for validating a new ML anomaly detection model to ensure the absence of trace. The original trace characterization technique works extremely well for ML anomaly detection models for assets with hundreds of sensors and sampling intervals of seconds to minutes. The original trace characterization technique evaluates the model for the presence of trace and quantitatively characterizes the degree of trace in the model with a trace metric (FM). When it is discovered that the ML model has some degree of trace, the original trace characterization technique can further be useful in guiding mitigation steps to remove the trace in the model before the model is used for predictive maintenance on physical assets.

[0024] In the context of the systems and methods for passive inference of signal trace in multivariate anomaly detection described herein, the trace metric (FM) created by the original trace characterization technique can be referred to and used as a benchmark to illustrate the robustness and improvements achieved by the inferential trace metric (iFM) shown and described herein. Accordingly, this section presents an introduction to identifying and quantitatively characterizing the trace phenomenon using the trace metric.

[0025] At a high level, the original tracking characterization technique involves training an ML model on a dataset of time series signals without anomalies, then recursively running the ML model by inserting a simple ramp-like fault signature into each signal one at a time, and finding the tracking metric for the ML model for that signal. The tracking metric (FM) that quantitatively characterizes the degree of tracking (between 0 and 1) within the ML model for a given time series signal in the original tracking characterization technique is given by

[0026] [Number]

[0027] as given below. In the above equation, m residuals is the slope of the residual between the signal predicted by the ML model and the monitoring signal with the inserted fault signature, and m ramp is the slope of the inserted ramp-like fault signature. The residual is the point-by-point difference between the monitoring signal data at a given point in time (or observation) and the ML model estimate at that point in time. A low ratio of the residual slope to the ramp slope indicates that the ML model is tracking the inserted signal degradation, and correspondingly a high (close to 1) tracking metric is determined for the performance of the ML model with respect to that signal. This process performs one full execution of the ML algorithm for each signal in the dataset (both via the training stage and the monitoring stage).

[0028] Note that other more complex degradation modes can be inserted into the monitoring signal, but there is no loss of generality when using the simple ramp degradation mode. An ML model that tends to track degradation will do so for any injected fault signature. However, the ramp degradation mode is conveniently linear and is thus easily used to generate a simple FM according to the above equation.

[0029] Figure 1 shows a plot 100 of a monitoring (or measurement) signal (Signal Example #5) compared to an ML model estimate (or prediction) that shows little following. Plot 100 includes two subplots, namely, (i) a signal amplitude subplot 105 at the top showing the amplitude values of the monitoring signal 110 and the estimated signal 115 plotted over time with respect to an amplitude axis 120 and an observation axis 125, and (ii) a residual amplitude subplot 155 at the bottom showing the residual 160 between the monitoring signal 110 and the ML model estimate 115 plotted over time with respect to an amplitude axis 165 and an observation axis 170, with the slope 175 of the lamp degradation mode inserted into the monitoring signal 110 superimposed on the residual 160. The lamp degradation mode is inserted into the monitoring signal 110 between observations 3750 and 5000. Plot 100 depicts an ML model estimate that does not contain following. It is clear that there is degradation in the measurement signal 110 as shown in the signal amplitude subplot 105. As seen in the residual amplitude subplot 155, the residual 160 coincides with the slope of the degradation 175. Since the slope of the residual 160 is tilted towards (or coincides with) the slope of the degradation 175, the following metric approaches zero, indicating an ML model estimate that does not show following.

[0030] Figure 2 shows a plot 200 of a monitoring (or measurement) signal (Signal Example #5) compared to an ML model estimate (or prediction), showing a small amount of tracking. Plot 200 includes two sub - plots, namely, (i) a signal amplitude sub - plot 205 at the top showing the amplitude values of the monitoring signal 210 and the estimated signal 215 plotted over time with respect to the amplitude axis 220 and the observation axis 225, and (ii) a residual amplitude sub - plot 255 at the bottom showing the residual 260 between the monitoring signal 210 and the ML model estimate 215 plotted over time with respect to the amplitude axis 265 and the observation axis 270. The slope 275 of the lamp degradation mode inserted into the monitoring signal 110 is superimposed on the residual 160. The lamp degradation mode is inserted into the monitoring signal 210 between observation 3750 and observation 5000. Plot 200 depicts an ML model estimate that contains some tracking. The degradation in the measurement signal 210 is evident in the signal amplitude sub - plot 205, but the tracking may not be immediately apparent in the estimated signal 215. However, as seen in the residual amplitude sub - plot 255, the slope of the residual 260 does not match the slope 275 of the inserted degradation mode, indicating an ML model estimate showing tracking.

[0031] Figure 3 shows a comparison plot 300 of two sets of residuals, namely, the residual 160 of the monitoring signal 110 in Figure 1 and the residual 260 of the monitoring signal 210 in Figure 2. Both the residual 160 and the residual 260 are plotted with respect to the amplitude axis 305 and the observation axis 310, and the slope 315 of the lamp degradation mode inserted into both signals 110 and 210 is superimposed on the residuals 160 and 260. The residual 110 (shown as a simple line) shows little tracking, while the residual 210 (shown as a dotted line) shows some tracking as seen by the deviation of the residual from the line of slope 315. Plot 300 shows two sets of residuals 160 and 260 on the same plot for comparison, further showing the deviation that the ML estimate has received since the tracking phenomenon has been present within the ML model.

[0032] Problems Solved by the New Inferential Tracking Metric Currently, the Internet of Things (IoT) digital transformation initiatives are driving the adoption of higher density monitoring across industries, and for this reason, the original solutions may require prohibitively large processing and / or memory requirements to perform the follow-up detection. It is not uncommon for an enterprise to have a historian signal database that archives telemetry from IoT sensors for over a decade (especially in the utility and oil and gas sectors). Additionally, entities across all imaginable sectors are upgrading their sensors and data collection systems for their entities and achieving much higher sampling rates than in previous years. Together, these phenomena lead to very long sensor streams containing hundreds of thousands, millions, and even billions of observations. For example, the latest airplanes now have 75,000 sensors, and the latest oil refineries and medium-sized data centers can each have one million sensors. At the same time, data acquisition (DAQ) units (devices that convert the measurement results of physical phenomena into digital values) typically have the ability to deliver at a sampling rate of dozens of milliseconds down to a sampling interval of a single millisecond (i.e., a sampling rate at the kHz level).

[0033] The original tracking characterization technique is subject to several scalability constraints that limit the use of this technique. The original tracking characterization (or tracking metric) solution is excellent for monitoring systems with sparser monitoring that features tens to hundreds of sensors sampled at intervals of seconds to minutes, but the recursive execution of N sensors cannot be easily extended to higher density monitoring characterized by very large values of N and / or a large number of observations per sensor due to orders of magnitude larger processing and memory requirements. This presents a computationally difficult problem to solve for growing enterprise groups, whose data historians (software programs that record data from ongoing processes or store it in archives) have collection signal data from very high sampling rate sensors and / or signal data archived over many years. This data can only be used to train new ML models if the computationally difficult problems inherent in data volume are solved. The availability of state-of-the-art high performance computing (HPC) and cloud computing "chokes" (i.e., cannot complete training) on the training of ML models on signal archives with hundreds of millions to hundreds of billions of observations. This is because (i) the memory footprint required to train the ML model increases as the square of the number of sensors under monitoring, and (ii) the computational cost increases as the cube of the number of observations in the historical dataset used to train the machine learning model to maintain a uniformly dense training vector. In the original tracking characterization process, such training of the ML model needs to be repeated for each signal in the signal database. Thus, the original tracking characterization tools are limited by the memory footprint and total computational cost to confirm the absence of tracking in the model and cannot be used for IoT customers with hundreds of millions to hundreds of billions of observations in their data historian signal archives.

[0034] There are also several limitations to the original tracking characterization technique and other metrics outputs of this technique. First, artificially inserting degradation on a signal essentially changes the signal trend that lifts the FM value if degradation already exists within the signal. Second, the original tracking metric does not help in differentiating the severity of tracking when high levels of tracking occur. Third, the original tracking characterization technique evaluates how the ML model makes estimations when there are abnormal trends, but does not address noise, which is a major factor in tracking. When an ML algorithm is developed, one important goal / feature is to learn the underlying behavior of the system under monitoring. Time series prediction and anomaly detection MLs in particular start to learn the critical modes of the system, but as additional measurement noise is introduced, the mode dominance becomes even less clear, and the algorithm starts to learn or follow the noise mode in addition to the nominal behavior. Further, in some applications of anomaly detection, many monitoring signals may be "range-bound", where range-bound means that in the case of some mechanical systems, electromechanical systems, and even hydraulic flow systems, the control variable duty cycle can only fall between zero (idle) and maximum (100%). In the case of this type of range-bound variable, it is physically meaningless to superimpose a ramp beyond the 100% duty cycle maximum setting.

[0035] The systems and methods described herein address the issues of sensor quantity and sampling rate by performing a technique (passive inference of signal tracking in multivariate anomaly detection) that does not require many training / execution iterations to evaluate tracking and quantitatively characterize the degree of tracking. Instead, the new passive inference of the tracking technique only runs the ML algorithm once. By running the ML algorithm only once, in a state without anomalies within the customer dataset, the degree of tracking of the resulting ML model can be inferred from the calculated standard deviation of the residuals (the residuals are the pairwise differences between the measured signal and the ML model estimates of this signal) generated from a single run of the ML. This standard deviation of the residuals between the measured signal and the ML model estimates functions as the inferential tracking metric (iFM) as described herein.

[0036] Advantageously, the systems and methods for passive inference of follow - up in multivariate anomaly detection described herein can easily scale to current and future high - density monitoring use cases that require hundreds of millions to hundreds of billions of observations without the computational cost that the original solution cannot support. Another advantage is that the implementation of the systems and methods for passive inference of follow - up in multivariate anomaly detection for existing ML monitoring systems makes the existing ML monitoring systems more accurate and reduces the false alarm rate for IoT end - customer ML predictive anomaly discovery.

[0037] Also, the systems and methods for passive inference of follow - up in multivariate anomaly detection described herein not only solve the problem of FM inflation due to the presence of undetected follow - up in the signal, but also have the ability to quantify follow - up from both degradation and noise, leading to more robust and general metrics.

[0038] Furthermore, the systems and methods for passive inference of follow - up in multivariate anomaly detection described herein eliminate the need to insert a ramp or other synthetic degradation signature into the signal, thereby avoiding the concern that the inserted ramp exceeds a significant value on the range - bound signal.

[0039] Examples of the environment FIG. 4 shows one embodiment of a system 400 related to passive inference of signal follow - up in multivariate anomaly detection.

[0040] In one embodiment, the system 400 includes a time-series data service 405 and an enterprise network 410 connected by a network 415 such as the Internet or a private network connected to the Internet. The time-series data service 405 is directly connected to a data acquisition unit (DAQ) such as a sensor (such as sensor 420) or a remote terminal unit (RTU) via a network 425, or is indirectly connected to a sensor (such as sensor 430) or DAQ via one or one or (one or one or) a plurality of upstream devices 435. In one embodiment, the networks 415 and 425 are the same network, and in another embodiment, the networks 415 and 425 are separate networks.

[0041] In one embodiment, the time series data service 405 includes various systems, such as a passive signal following inference component 440, a sensor interface server 450, a web interface server 455, and a data store 460. Each of these systems 440, 450, 455, 460 is configured to use logic, for example, in various software modules, to perform the functions described as being performed by these systems. In one embodiment, the components of the time series data service 405 are implemented on one or more hardware computing devices or hosts interconnected by a data network or a cloud network (such as the server-side network 465). For example, the components of the time series data service 405 can be executed by one or more computing hardware forms, such as a standard (or general-purpose) central processing unit (CPU) form, a high-density input / output (I / O) form, a graphics processing unit (GPU) form, an HPC form, etc., of network-connected computing devices. In one embodiment, the components of the time series data service 405 are each implemented by a dedicated computing device. In one embodiment, although the multiple or all components of the time series data service 405 are represented as separate units in FIG. 4, they are implemented by a common (or shared) computing device. In one embodiment, the components of the time series data service 405 can be implemented across a number of computing devices.

[0042] In one embodiment, the passive signal following inference component 440 includes an ML model training component 442, an iFM generation component 444, and a following alert component 446. In one embodiment, these components are each implemented as software modules. In one embodiment, the ML model training component 442 is configured to train and generate a machine learning model to predict or estimate correct signal values or expected signal values from training data representing the normal correct operation of the system, as described in more detail herein, for example. In one embodiment, the iFM generation component 444 is configured to infer the degree of following within the ML model (by generating an iFM) based on, for example, the standard deviation of the set of actual values within the time series signal and the standard deviation of the set of residuals between the estimated and actual values of the time series signal, as shown and described in more detail herein. In one embodiment, the following alert component 446 is configured to issue various forms of alerts when an unacceptably high level of following is indicated by the iFM, as described in more detail herein, for example.

[0043] In one embodiment, the components of the time series data service 405 communicate with each other via electronic messages or electronic signals. These electronic messages and electronic signals can be configured as calls to functions or procedures that access the characteristics or data of the components, such as application programming interface (API) calls. In one embodiment, these electronic messages or electronic signals are sent between hosts in a format compatible with the Transmission Control Protocol / Internet Protocol (TCP / IP) or other computer networking protocols. Each component of the time series data service 405 can (i) generate or construct an electronic message or electronic signal to issue a command or request to another component, (ii) use the infrastructure of the scheduling, dispatching, and routing system 405 to send that message or signal to another component, and (iii) parse the content of the received electronic message or electronic signal to identify the commands or requests that the component can execute. In response to identifying the command, the component will automatically execute the command or request.

[0044] In one embodiment, the time series data service 405 can be implemented as a service in a cloud infrastructure. In one embodiment, the time series data service 405 can be hosted by a dedicated third party, for example, in an Infrastructure as a Service (IaaS), Platform as a Service (PaaS), or Software as a Service (SaaS) architecture. In one embodiment, the time series data service 405 can be implemented in an on-premises infrastructure, such as a set of one or more dedicated servers.

[0045] In one embodiment, the time series data service 405 may be hosted by a third party and / or operated by a third party for the benefit of a number of account owners / tenants, each of whom conducts a business and has an associated enterprise network 410. In one embodiment, both the time series service 405 and the enterprise network 410 may be associated with an entity operating in any of a variety of high density sensor (i.e., using a large number of sensors) IoT industries (such as oil and gas production, utilities, aviation, consumer electronics, data center IT, etc.). In one embodiment, the time series data service 405 is configured to predictively detect or discover abnormal operations in assets having random downtime by operating the time series data service 405 according to the systems and methods described herein using logic such as software modules.

[0046] In one embodiment, the sensors or DAQs 420, 430 are configured to monitor physical phenomena occurring within or around an asset (such as a machine, device, system, facility, etc.). In one embodiment, the sensors or DAQs 420, 430 may be operably connected to the asset or, alternatively, may be configured to detect and monitor physical phenomena occurring within or around the asset. The asset generally includes any type of machine or facility having components that perform measurable activities. The sensors or DAQs 420, 430 can be network-connected sensors for monitoring any type of physical phenomenon. The network connection of the sensors or DAQs 420, 430 can be wired or wireless. The sensors 420, 430 can include, but are not limited to, voltage sensors, current sensors, temperature sensors, pressure sensors, rotational speed sensors, thermometers, flow meter sensors, vibration sensors, microphones, optical sensors, electromagnetic radiation sensors, proximity sensors, occupancy sensors, motion sensors, gyroscopes, inclinometers, accelerometers, global positioning system (GPS) sensors, torque sensors, flex sensors, moisture monitors, liquid level sensors, electronic noses, nuclear radiation detectors, or any of a variety of other sensors or transducers for generating electrical signals that describe the detected or sensed physical behavior.

[0047] In one embodiment, sensors 420, 430 are connected to a sensor interface server 450 via a network 425. In one embodiment, the sensor interface server 450 is configured to collect readings from sensors 420, 430 using logic such as a software module and store these readings as observations within a time series data structure, e.g., within a data store 460. In one embodiment, the sensor interface server 450 generates a time series data structure (such as a time series data archive) within the data store 460 and includes a data historian module for interacting with this structure. The sensor interface server 450 is configured to interact with the sensors by exposing one or more application programming interfaces (APIs) configured to receive readings from the sensors using, for example, sensor data formats and communication protocols applicable to the various sensors 420, 430. The sensor data format is generally defined by the sensor device. The communication protocol can be a custom protocol (such as a legacy protocol preceding IoT implementations), or various IoT or machine-to-machine (M2M) protocols, e.g., Constrained Application Protocol (CoAP), Data Distribution Service (DDS), Device Profile for Web Services (DPWS), Hypertext Transfer (Transport) Protocol / Representational State Transfer (HTTP / REST), Message Queuing Telemetry Transport (MQTT), Universal Plug and Play (UPnP), Extensible Messaging and Presence Protocol (XMPP), ZeroMQ, etc., and any other communication protocol that can be carried over the Transmission Control Protocol (Internet Protocol or User Datagram Protocol (TCP / IP or UDP) transport protocol). SCADA protocols, e.g., OLE for Process Control Unified Architecture (OPC UA), Modbus RTU, RP-570, Profibus, Conitel, IEC60870-5-101 or 104, IEC61850, and DNP3, etc., can also be adopted when extended to operate over TCP / IP or UDP.In one embodiment, the sensor interface server 450 polls the sensors 420, 430 to retrieve sensor readings. In one embodiment, the sensor interface server passively receives sensor readings actively transmitted by the sensors 420, 430.

[0048] For simplicity and clarity of explanation, the enterprise network 410 is presented by a local area network 470 at the site, which is operably connected with one or more personal computers 475, or servers 480, and one or more remote user computers 485 connected to the enterprise network 410 via the network 415 or other suitable communication networks or combinations of networks. The personal computer 475 and the remote user computer 485 can be, for example, desktop computers, laptop computers, tablet computers, smartphones, or other devices having the ability to connect to the local area network 470 or the network 415 or other synchronization capabilities. The computers of the enterprise network 410 interface with the time series data service 405 via the network 415 or another suitable communication network or combination of networks.

[0049] In one embodiment, a remote computing system (such as the computing system of enterprise network 410) can access information or applications provided by time series data service 405 via web interface server 455. For example, computers 475, 480, 485 of enterprise network 410 can access passive signal following inference component 440 from time series data series data service 405. In one embodiment, the remote computing system can send a request to web interface server 455 and receive a response from web interface server 455. In one example, access to information or applications can be performed by using a web browser on personal computer 475 or remote user computer 485. For example, these computing devices 475, 480, 485 of enterprise network 410 can request and receive a web page-based graphical user interface (GUI) for accessing monitoring information and alarm information provided by passive signal following inference component 440. In one example, these communications may be exchanged between web interface server 455 and server 480 and take the form of a remote representational state transfer (REST) request using JavaScript Object Notation (JSON) as, for example, a data exchange format, or a Simple Object Access Protocol (SOAP) request going to and coming from an XML server.

[0050] In one embodiment, the data store 460 includes one or more databases (such as a data historian archive, a time series database, a signal database, etc.), or other data structures configured to store and supply time series data received from sensors 420, 430 by the sensor interface server 450. The time series data consists of values sensed at regular or irregular intervals by one or more sensors, and can be stored in relation to both the sensors that sensed the values and the time. In one embodiment, the time series database is an Oracle® database configured to store and supply time series data. In some configuration examples, the data store 460 can be implemented using one or more Oracle® Exadata computing shapes, network-connected storage (NAS) devices, and / or other dedicated server devices. The API call can include a query to the database. The query can be composed in SQL and executed in an SQL runtime, or other appropriate query languages and environments can be used.

[0051] In one embodiment, the upstream device 435 can be a third-party service for managing IoT-connected devices. Alternatively, in one embodiment, the upstream device 435 can also be a gateway device configured to enable the sensor 430 to communicate with the sensor interface server 450 (for example, the sensor 430 is not IoT-compatible and thus cannot communicate directly with the sensor interface server 450).

[0052] Characterize tracking quantitatively with inferential tracking metrics The systems and methods for passive inference of tracking in multivariate anomaly detection described herein present a fundamentally different approach to the characterization of tracking in any type of ML anomaly detection algorithm called inferential tracking metrics (iFM).

[0053] The process for characterizing drift using iFM does not require introducing any degradation signature into any of the monitoring signals to induce and then characterize drift, as in the original drift characterization techniques. Instead, the new iFM is the implementation of synthetic pattern recognition that infers the presence of drift in the ML model by analyzing the measurement noise already present on the monitoring signals. For all measured IoT signals resulting from physical transducers, there is not necessarily any measurement noise. Such signals are generally represented as quantitative variables. (Note that for categorical variables, i.e., state variables such as on / off, open / closed, engaged / disengaged, or other variables that can take one of a possible number of discrete states of a constant, there is no noise and there is also no possibility of drift existing within the signal of that variable.) Whether the noise level is small for high-precision signals monitoring a process with a very stable noise level or large due to either poor sensor accuracy (high uncertainty) and / or monitoring a noisy physical process, the new iFM technology produces a highly accurate inference of the drift metric, as shown herein.

[0054] The new iFM is a computational technique for identifying how much drift is occurring within the ML model without continuously inducing degradation in each signal as in the original drift characterization techniques. The iFM is predicated on the idea that if the ML model is operating correctly, the nominal behavior is learned during ML training and then, during monitoring, an abnormal degradation signature is detected.

[0055] Therefore, for an ML model that does not exhibit drift, the standard deviation of the residuals (STD or i) resulting from monitoring the signal is assumed to correlate with the standard deviation of the signal itself.

[0056] All measurement signals have a deterministic component with some superimposed random measurement noise. ML algorithms cannot predict randomness, including random measurement noise. The ML algorithm prediction of random measurement noise is an indication of following by the ML algorithm. Thus, as described above, when the ML model behaves very well and without following, the noise on the residuals is assumed to correlate with the noise on the incoming raw measurement signal. However, if it is found that the standard deviation of the residuals is small, decreasing, or "shrinking" compared to the noise on the original measurement signal, this is an indication of following. Such shrinking residuals mean that the ML model is following the randomness. The presence of shrinking residuals can be shown when the standard deviation of the residuals is smaller than the standard deviation of the measurement signal being analyzed. Furthermore, the greater the degree of following by the ML model, the greater the reduction in random noise within the residuals. The fact that this characteristic in the behavior of the signal is transformed by non-linear non-parametric (NLNP) regression can be utilized for a robust and high-fidelity characterization of following for any NLNP regression-based anomaly detection service. This is not to directly calculate following by inducing anomalies for each signal under monitoring and recursively evaluating ML N times for N signals as in the original following characterization technique, but rather the degree of following within the ML model can be inferred by running the ML model once on anomaly-free data, calculating the residuals, calculating the standard deviation (STD) of the residuals, compiling a list of the ratios of the corresponding STD of the residuals to the STD of the raw signal, and integrating over a database of signals to obtain a quantitative metric (iFM) as an indication of the tendency of the model to show following. Thus, in one embodiment, the training and value prediction of the ML model are performed only once when inferring signal following within the machine learning model.

[0057] There is a one-to-one mapping between the new iFM metric and the ML model's degree of tracking, but this mapping is only correct within the same database of signals. Note that any database of time series signals will have different signal-to-noise ratios (SNRs), different in-signal correlation patterns, and different sampling rates. Due to the non-linear complexity within the in-signal correlation pattern, it is not possible to mathematically derive any analytical formulas to predict the tracking of any database of time series signals, nor to compile a lookup table approach to tracking based on a single completion of an execution model, calculate residuals, calculate standard deviations, and integrate the steps outlined above.

[0058] The new iFM tracks the ratio of the original raw signal STD to the calculated residual STD calculated for each signal in the database. Advantageously, the ML model only needs to be run once when characterizing the tracking, as opposed to the original tracking characterization technique that used N recursive executions of the ML model for N signals. After calculating a table of the ratios of the calculated residual STD to the raw signal STD for each signal, these values are averaged across the entire database to reflect any outlier behavior and give an overall theoretical representation of the tracking effect.

[0059] Examples of methods for inferential characterization of tracking In one embodiment, each step of the method executed by the computer described herein can be executed by a processor (such as the processor 910 shown and described with reference to FIG. 9) of one or more computing devices configured to (i) access a memory (such as the memory 915 shown and described with reference to FIG. 9 and / or other computing device components), and (ii) cause the system to execute the steps of the method (such as the passive inference of signal tracking within the multivariate anomaly detection logic 930 shown and described with reference to FIG. 9) using logic. For example, the processor accesses and reads from and writes to the memory to execute each step of the method executed by the computer described herein. These steps can include (i) retrieving any necessary information, (ii) calculating, determining, generating, classifying, or otherwise creating any data, and (iii) storing any data calculated, determined, generated, classified, or otherwise created for subsequent use. References to storage or saving refer to storage as a data structure within the memory or storage device / disk of the computing device (such as the memory 915 or storage device / disk 935 of the computing device 905 or a remote computer 965 shown and described with reference to FIG. 9, or within the data store 460 shown and described with reference to FIG. 4).

[0060] In one embodiment, each subsequent step of the method automatically begins in response to parsing a received signal or retrieved saved data indicating that the previous step has been executed to at least the extent necessary for the subsequent step to begin. Generally, the received signal or retrieved saved data indicates the completion of the previous step.

[0061] FIG. 5 shows an embodiment of a method 500 related to passive inference of signal tracking in multivariate anomaly detection. The method 500 shows, in a detailed flowchart, an embodiment of a process for calculating and tracking the iFM of any database of time series signals.

[0062] In one embodiment, the steps of method 500 are performed by a passive signal inference component 440 (shown and described with reference to FIG. 4). In one embodiment, the passive signal inference component 440 is a dedicated computing device (such as computing device 905) configured with passive inference of signal following within the multivariate anomaly detection logic 930. In one embodiment, the passive signal inference component 440 is one or more modules of a dedicated computing device configured with the logic 930.

[0063] Method 500 can be automatically initiated based on various triggers, for example, in response to receiving a signal on the network, or in response to parsing stored data indicating that (i) a user (or administrator) of the time series data service 405 has initiated method 500, (ii) method 500 is scheduled to start at a specified time or time interval, or (iii) a user (or administrator) of the time series data service 405 is executing a command to train an ML model on the signal database. Method 500 parses the received signal or retrieved stored data and, in response to determining that this signal or stored data indicates that method 500 should begin, starts at start block 505. The process proceeds to process block 510.

[0064] In process block 510, the processor initializes the signal database with M time series signals. In one embodiment, the processor receives parameters that describe the signal database, including the number of signals M, the number of observations, or the length O of the signal database, and generates a database of M synthetic signals of length O. For example, the parameters may be input by the user via a graphical user interface (GUI) of a high-fidelity signal synthesizer, and the signal synthesizer generates simulations of signals produced by sensors of the system under monitoring during normal non-faulty operation. The database of synthetic signals may be stored, for example, in data store 460. In one embodiment, the processor accepts a selection of a particular signal database having M signals, stored, for example, in data store 460, retrieves (in whole or in part) the signal database from storage within data store 460, and holds it in memory so that it can be accessed at any time. In one embodiment, the signal database having M signals may be the database of synthetic signals that was generated and stored prior to selection. In one embodiment, the signal database having M signals may be a database of actual signals recorded from sensors such as sensors 420, 430, and stored prior to selection. Thus, when the processor finishes initializing the signal database with M signals, the processing in process block 510 is complete, and the process proceeds to process block 515.

[0065] In process block 515, the processor initializes the signal counter i to a value of 1. The signal counter i is an index that indicates which signal in the database (within the range from signal number 1 to signal number M) is the current signal under consideration in the processing loop described with reference to blocks 520, 525, and 530 below. In one embodiment, the processor declares an integer variable i for the signal counter and executes an assignment operation to give i the integer value 1. When the processor finishes initializing the signal counter i to a value of 1, the processing in process block 515 is complete, and the process proceeds to decision block 520.

[0066] Blocks 520, 525, and 530 form a processing loop that iterates for each of the M signals in the signal database. The processing loop is represented by the preconditions shown in decision block 520 and is shown and described with reference to FIG. 5 as a while loop. Alternatively, the processing loop may be executed as a do-while loop that has postconditions represented according to the loop. In decision block 520, the processor determines whether a signal remains (in block 525) in the signal database for evaluation. In one embodiment, the processor makes this determination by evaluating whether the value of signal counter i is less than or equal to the number M of signals in the database. The processor searches for the value of the number M of signals in the database and the current value of signal counter i. The processor compares the value of M with the value of i. If the value of i is less than or equal to M, the signal remains in the signal database for evaluation. If the value of i is not less than or equal to M (i.e., greater than M), the signal does not remain in the database for evaluation. Thus, if the processor determines that a signal remains in the signal database (i≤M?: YES), the processing in decision block 520 is complete and the processing proceeds to process block 525. If the processor determines that no signal remains in the signal database (i≤M?: NO), the processing in decision block 520 is complete, the loop of blocks 520, 525, and 530 ends, and the processing proceeds to process block 535.

[0067] In process block 525, the processor calculates and stores the standard deviation for signal number i. In one embodiment, the processor (i) extracts the amplitude value of signal number i at each observation between 1 and the total number of observations of signal number i (including both ends) by parsing, for example, signal number i to search for the amplitude value (amplitude obs ) of signal number i at each observation of signal number i from the signal database, and (ii) for example, Equation 2

[0068]

Number

[0069] By solving the equations shown, calculate the average amplitude (signalMean) for signal number i from all the searched amplitudes and the total number of observations (numObs) for signal number i, (iii) for example for example Equation 3

[0070]

Number

[0071] By solving the equations shown, calculate the variable (signalVariance) for signal number i from the average amplitude of signal number i, all the searched amplitudes, and the total number of observations for signal number i, and (iv) for example for example Equation 4

[0072]

Number

[0073] Calculate the standard deviation (signalSTD) for signal number i by solving the equations shown from the variable of signal number i, thereby calculating the standard deviation of signal number i. In each of the above steps, the processor can save the results (amplitude, mean, variable, and standard deviation) for further processing by writing the results as a data structure to memory or a storage device. For example, the mean, variable, and standard deviation of a signal can be saved in a data structure related to signal number i, such as a metadata field of a data structure for the signal in a signal database. Thus, when the processor finishes calculating and saving the standard deviation of signal number i, the processing at process block 525 is complete and the processing proceeds to process block 530.

[0074] In process block 530, the processor increments the value of signal counter i by 1. In one embodiment, the value of i is retrieved, the sum of the value of i and 1 is found, and the value of this sum is saved as the value of i. This advances to the next signal under consideration in the loop. Thus, when the processor finishes incrementing the value of signal counter i by 1, the processing in process block 530 is complete and the process returns to decision block 520.

[0075] Following completion of the loop of blocks 520, 525, and 530, the process advances to process block 535. In process block 535, the processor calculates and stores the average value of the standard deviations of all M signals. In one embodiment, the processor (i) searches for the standard deviation for each of the M signals from the data structures associated with each signal, for example, by parsing the respective data structures of the M signals to place and extract the standard deviation of the signal, and (ii) calculates the average (mean) standard deviation (signalSTDMean) of the M signals from all the extracted standard deviations, for example, by solving the equation shown in Equation 5

[0076]

Number

[0077] to calculate the average value of the standard deviations for all M signals. The processor can save the calculated average of the standard deviations for all M signals for further processing by writing the result to memory or a storage device as a data structure. For example, the average of the standard deviations of all M signals can be saved in a data structure associated with the signal database, such as a metadata field of the signal database. At this point, the processor has calculated one average standard deviation of the measured values of the M time series signals. Thus, when the processor finishes calculating and storing the average value of the standard deviations of all M signals, the processing in process block 535 is complete and the process advances to process block 540.

[0078] In process block 540, the processor trains a machine learning model in the training portion of the signal database and generates a machine learning model.

[0079] In one embodiment, the processor executes a supervised learning process to construct a state estimation model for each signal within a signal set selected from a signal database of M signals. In one embodiment, when M is small, all M signals in the signal database can be selected for inclusion in the state estimation model. In one embodiment, when M is large, sampling of the signals in the signal database can be selected for inclusion in the state estimation model. The processor selects a training data set (signal data over a time / observation interval representing the normal operating state of the system represented by the signals in the signal database). For example, this can be the first or initial portion of the signals in the signal database that have been confirmed to represent normal operation. In one embodiment, the ML state estimation model is an MSET model or an MSET2 model. In one embodiment, the ML state estimation model implements an ML anomaly detection algorithm other than MSET or MSET2. In one embodiment, the ML state estimation model is a non-linear non-parametric regression model. For each signal within the signal set, the state estimation model is trained using the training data to generate an accurate estimate of the signal based on the values and correlations among the other signals within the set. The processor then saves the state estimation model, thereby generating the state estimation model. In this way, the processor is training a machine learning model to predict or estimate the values of time series signals.

[0080] In one embodiment, the processor configures a fault detection model to identify an abnormal deviation between a measurement signal and an estimated value of the signal. In one embodiment, the fault detection model uses a Sequential Probability Ratio Test (SPRT), which calculates the cumulative sum of the log-likelihood ratios of each sequential residual between the measurement signal and the estimated value to detect an abnormal deviation from normal operation (or a fault). The processor selects a threshold for the cumulative sum of the log-likelihood ratios at which the null hypothesis (no fault is detected) should be accepted for the SPRT. The processor also selects a threshold for the cumulative sum of the log-likelihood ratios at which the alternative hypothesis (a fault is detected) should be accepted for the SPRT. The processor then saves the fault detection model configured in relation to the state estimation model, thereby generating a fault detection model.

[0081] Accordingly, when the processor finishes training the machine learning model on the training portion of the signal database and generating the machine learning model, the processing at process block 540 is complete and the processing proceeds to process block 545.

[0082] At process block 545, the processor monitors the signal with the ML model. In one embodiment, the processor selects a set of monitoring data (signal data over a time / observation interval for which the trained ML model will determine whether there is an abnormal deviation in the signal). For example, the monitoring data can be the second or later portion of the signal in the signal database. In one embodiment, the signal database is divided into a training portion that includes all observations starting from the first observation in the signal database up to (but not including) the selected observation (training data), and a monitoring portion that includes all observations starting from the selected observation up to the final observation (or second selected observation) in the signal database (monitoring data). In one embodiment, the monitoring data is the remaining portion of the observations in the signal database other than the training data. The processor searches the memory for the selected observations, sets the selected observations as the starting point of the monitoring, searches for the final observation, and sets the final observation as the ending point of the monitoring.

[0083] For each observed value from the start to the end of the monitoring, the processor executes a trained ML model on the value of the signal at the time of observation to generate a predicted value or an estimated value for each of the signals at the time of observation. The processor stores the estimated value for each of the signals at the time of observation. In one embodiment, next, the processor repeats the prediction and storage for each observation until the monitoring is complete, generating a set of estimated values stored for each signal at each observation. In this way, the processor is predicting or estimating the value of each of the signals with a trained machine learning model.

[0084] Accordingly, when the processor finishes monitoring the signal with the ML model, the processing at process block 545 is complete and the processing proceeds to process block 550.

[0085] At process block 550, the processor stores a set of residuals between the ML model estimated values and the actual values of the signals for each signal in the signal database. In one embodiment, for each signal (m) in the database of M signals with numObs observations of length for each observation (obs) of the signal, the processor searches for the value of the signal estimated by the ML model for that observation (estimate obs ) and the actual measured value of the signal for that observation (measurement obs ), and the processor takes the difference between the ML model prediction and the measurement at the time of observation to obtain the residual value (residual obs ) for that observation, as shown, for example, in Equation 6

[0086]

Number

[0087] ) is calculated. Thus, to generate the residuals for each observation, a differential operation between the actual monitored value and the estimated or predicted value is performed. The processor stores the residual for each observation obs of each signal m. Thus, the set of residuals for signal m is the set of all residuals from obs = 1 to obs = numObs. The processor stores the set of residuals for each signal associated with signal m, for example, by entering the residual value for each observation in a field related to the observation (or other data structure) within the signal data structure. The signal data structure for each signal m can have fields for the measured value, estimated value, and residual value at each observation, arranged in a time series in the order of observation. The signal data structure can be stored in a signal database. In one embodiment, the residuals are calculated as part of the operation of a fault detection model such as the above SPRT model and can be used both for SPRT fault detection and for characterizing the tracking described herein. In this way, the processor generates a time series set of residuals between the predicted value and the measured value for each of the signals.

[0088] Thus, when the processor finishes storing the set of residuals between the ML model estimate and the actual value of the signal for each signal in the signal database, the processing in process block 550 is complete and the process proceeds to process block 555.

[0089] In process block 555, the processor initializes the residual set counter j to a value of 1. As described above, the processor has generated a set of residuals for each signal in the database. The set counter j is an index indicating which set of residuals (within the range from the set of residuals for signal number 1 to the set of residuals for signal number M) is the current set of residuals under consideration in the processing loop shown and described with reference to blocks 560, 565, and 570 below. In one embodiment, the processor declares an integer variable j for the set counter and executes an assignment operation to give j the integer value 1. When the processor finishes initializing the set counter j to a value of 1, the processing in process block 555 is complete and the process proceeds to decision block 560.

[0090] Blocks 560, 565, and 570 form a processing loop that repeats for each of the M sets of residuals. The processing loop is shown and described with reference to FIG. 5 as a while loop with the preconditions represented in decision block 560. Alternatively, the processing loop may be executed as a do~while loop with the postconditions represented according to the loop. In decision block 560, the processor evaluates whether the value of the set counter j is less than or equal to the number M of residuals in the same way as shown and described for the signal counter i with reference to the upper decision block 520 above, in order to determine whether a set of residuals remains (in block 565) for evaluation. If the processor determines that a set of residuals remains (j ≦ M?: YES), the processing in decision block 560 is complete and the process proceeds to process block 565. If the processor determines that no set of residuals remains (j ≦ M?: NO), the processing in decision block 560 is complete, the loop of blocks 560, 565, and 570 ends, and the process proceeds to process block 575.

[0091] In process block 565, the processor calculates and stores the standard deviation for the set of residuals with signal number j (also referred to herein as the set number j of residuals). In one embodiment, the processor (i) extracts the residual values at each observation between 1 and the total number of observations of signal number j (including both ends) by parsing, for example, signal number j to retrieve the residual values of the set number j of residuals at each observation of signal number j from the signal database, (ii) calculates the average residual value (residualMean) of the set number j of residuals from all retrieved residuals and the total number of residuals (numObs) of the set number j of residuals by solving the equation shown, for example, in Equation 7 obs ) by, for example, solving the equation shown in Equation 7

[0092]

Equation

[0093] shown in Equation 7, and (iii) calculates the standard deviation of the set number j of residuals from all retrieved residuals and the average residual value (residualMean) of the set number j of residuals by, for example, solving the equation shown, for example, in Equation 8

[0094]

Number

[0095] By solving the equation shown in, calculate the variable (residualVariance) of residual group number j from the average residual of residual group number j, all searched residuals, and the total number of observations, and (iv) for example, Equation 9

[0096]

Number

[0097] By solving the equation shown in, calculate the standard deviation (residualSTD) of residual group number j from the variable of residual group number j, thereby calculating the standard deviation for residual group number j. In each of the above steps, the processor can save the results (amplitude, mean, variable, and standard deviation) for further processing by writing the results as a data structure to a memory or storage device. For example, the mean, variable, and standard deviation of the residual group can be saved in a data structure related to residual group number j. For example, these values can be saved as metadata of the data structure in the signal database field for signal number j related to the residual group of signal number j. Therefore, when the processor finishes calculating and saving the standard deviation of residual group number j, the processing in process block 565 is completed, and the process proceeds to process block 570.

[0098] In process block 570, the processor increments the value of group counter j by 1 in the same way as shown and described above for signal counter i with reference to process block 530. The processing in process block 570 is completed, and the process returns to decision block 560.

[0099] Following completion of the loops of blocks 560, 565, and 570, the process proceeds to process block 575. At process block 575, the processor calculates the average value of the residual standard deviations (i.e., the average of the standard deviations of the sets of residuals). In one embodiment, the processor retrieves each of the M standard deviations calculated for M sets of residuals for the signal from memory or a storage device. The processor then calculates the sum of each of the standard deviations of the residuals and stores this sum for subsequent processing. The processor then divides the sum by M (the number of signals, the number of sets of residuals for such signals, and the number of standard deviations of the sets of residuals) to find the average value of the standard deviations of the residuals. In one embodiment, the processor determines the average value of the standard deviations of the residuals (residualSTDMean) by solving Equation 10.

[0100] [Number]

[0101] The processor then stores the average for subsequent processing in a data structure associated with, for example, the signal database, e.g., as metadata of the signal database, or in a data structure associated with the ML model. At this point, the processor has calculated the average (in this case the mean) standard deviation of the sets of residuals. Thus, when the processor finishes calculating the average value of the standard deviations of the residuals, the processing at process block 575 is complete and the process proceeds to process block 580.

[0102] At process block 580, the processor calculates an inferential following metric (iFM) from the average value of the standard deviations of the signals (calculated and stored as described with reference to process block 535 above) and the average value of the standard deviations of the residuals. In one embodiment, the iFM is the ratio of the average value of the standard deviations of the signals to the average value of the standard deviations of the residuals, i.e., as represented by Equation 11.

[0103] [Number]

[0104] Next, the processor writes the iFM to memory or a storage device for subsequent reference or processing, for example, in the metadata for the signal database or in the metadata for the ML model or other storage locations related to the ML model. In this way, the processor calculates the iFM ratio by dividing the average standard deviation of the measurements by the average standard deviation of the set of residuals. Thus, when the processor has completed calculating the inference following metric from the average value of the standard deviation of the signal and the average value of the standard deviation of the residuals, the processing at process block 580 is complete and the process proceeds to decision block 585.

[0105] In decision block 585, the processor evaluates whether the iFM meets the alarm threshold. In one embodiment, the alarm threshold indicates that the level of following (represented by iFM) detected or inferred by the ML model is unacceptably high. The threshold can be previously defined by the user and saved for retrieval and use in decision block 585. In one embodiment, a model with a very low degree of following has an iFM of 1 or less, and an iFM value between 1 and 1.5, for example, an iFM of about 1.3, is the tolerance threshold that distinguishes minimal following, which has a minor impact on the model prediction, from unacceptable following that affects the model prediction. The alarm threshold for the iFM value can be adjusted lower, for example, to 1.1 or even lower, when the noise level of the signal is low. The alarm threshold for the iFM value can be adjusted higher, for example, up to 2 or even higher, when the noise level of the signal is high. In one embodiment, the processor evaluates whether the iFM is greater than the alarm threshold. The processor retrieves the value of the iFM and the value of the alarm threshold. The processor compares the value of the iFM with the value of the alarm threshold. If the value of the iFM is greater than the value of the alarm threshold (iFM>Threshold?: YES), the level of following present in the ML model is unacceptably high, the processing in decision block 585 is complete, and the processing proceeds to process block 590. If the value of the iFM is not greater than the value of the alarm threshold (i.e., it is less than or equal to the value of the alarm threshold) (iFM>Threshold?: NO), the level of following present in the ML model is acceptable, the processing in decision block 585 is complete, and the processing proceeds to end block 595, where process 500 ends. Thus, the processor determines whether signal following exists in the trained machine learning model based on whether the ratio of the mean standard deviation of the measured values to the mean standard deviation of the set of residuals exceeds the threshold.

[0106] In process block 590, the processor triggers an alarm regarding the level of tracking occurring within the ML model. In one embodiment, the processor generates an alarm, alert, or warning indicating that the ML model in the evaluation stage exhibits an unacceptably high amount of tracking. In one embodiment, the alarm is a message indicating an unacceptably high level of tracking occurring within the ML model. The message can include an indication of the value of the iFM of the ML model, the value of the threshold, an indication of the degree of tracking (whether the tracking is moderate or high), the signal on which the tracking occurs, and an explanation of the significance of the tracking. Each of these message contents can be retrieved from memory or a storage device and used by the processor to generate the alarm message. The alarm can be text only or can include graphical elements. The alarm message can take the form of processing instructions such as API requests or commands to other parts of the computing system 400. In one embodiment, the processor retrieves the alarm message content from memory or a storage device, generates the alarm message, and sends the message for presentation to the user. In one embodiment, the message is an email or text message and is sent to the inbox associated with the user. In one embodiment, the message is sent to and presented on a display within a text interface or graphical user interface, such as the interface of the time series data service 405. In this way, the processor presents an alarm indicating the presence of signal tracking within the trained machine learning model. Thus, when the processor finishes triggering an alarm regarding the level of tracking occurring within the ML model, the processing in process block 590 is complete, and the process proceeds to end block 595, where process 500 ends.

[0107] In one embodiment, when the warning threshold shown and described with reference to decision block 585 of process 500 is satisfied, in addition to triggering a warning at process block 590, a relaxation analysis is triggered. In response to determining that there is a tracking in the ML model, the system can suggest one or more of the following tracking relaxation techniques to reduce the tracking in the ML model: increasing the number of training vectors used to train the ML model, performing filtering operations on the training and monitoring signals to reduce noise, and changing the number of monitoring signals. In one embodiment, the recommended relaxation technique is included in the generated warning message. In this way, the warning can include the ratio (iFM) and a recommendation of a technique for reducing signal tracking in a machine learning model selected from one or more of increasing the number of training vectors in the ML model, applying a filter to the signal input to the ML model to reduce noise, and increasing the number of signals.

[0108] Effectiveness of iFM To demonstrate the effectiveness of passive inference of signal following using iFM, two typical signal database examples consisting of N = 20 signals each are considered herein. The original following characterization technique declares the first of these databases of signals to demonstrate nearly zero following when analyzed by an ML model. The original following characterization technique indicates the second of these databases of signals to induce following when analyzed using at least the MSET, LSTM neural network, and linear regression ML models. Using these two datasets, two separate ML models are trained. For example, the ML model can be an MSET model. The first half of each database (i.e., the first half of the observations) was used to train the ML model. The second half (i.e., the second half of the observations) is used for monitoring. Thus, the ML model predicts what "should be" based on each signal and the patterns present in all the other N - 1 signals in the signal database with the trained model. The first trained model does not show to a detectable extent of following. The second trained model is the case where following begins to occur at an unacceptable level. The following table tracks the values for the new iFM and the original following characterization technique FM, both as examples of the model.

[0109] Table 1 includes values regarding ML model following of the MSET ML model showing minimum following.

[0110]

Table 1

[0111] Table 2 includes values regarding ML model following of the model showing an unacceptable degree of following.

[0112]

Table 2

[0113] The second column of the table shows the average standard deviation of 20 ML model residuals calculated by subtracting the MSET estimate from the monitoring part (monitoring or second half) of the signal. The fourth column of the table shows the average standard deviation of the original signal. As described above, there is a correlation between the ML model residuals and the original signal, and thus the STD of the residuals is assumed to be similar to the STD of the original signal. Therefore, the ratio between the STD of the original signal and the STD of the residuals increases when tracking occurs within the ML model. This ratio is the iFM and is shown in the fifth column of the table. Tracking is independent of the severity of the measurement noise in the database, and tracking is the driving factor. If the original signal-to-noise ratio is insufficient, tracking increases. To demonstrate this phenomenon, the aforementioned process of ML model training and monitoring on 20 signal databases was repeated as the noise component on the signal increased. The first column of the table shows the standard deviation (STD) coefficient of the Gaussian noise added to the signal. The sixth column of the table displays the original or "typical" following metric (FM) created by the original following characterization technique. Comparing the FM in the sixth column with the iFM STD ratio in the fifth column results in a very similar pattern, thereby verifying the logical basis of the iFM.

[0114] Figure 6 shows a plot 600 depicting the change in the tracking metric values with the increase in the noise ratio for two ML models, which demonstrates the ability to identify noise-driven tracking using passive inference of signal tracking in multivariate anomaly detection. Plot 600 compares the two ML models discussed in Table 1 and Table 2 above. One ML model (indicated by diamonds in Plot 600) shows a low degree of tracking, while the other ML model (indicated by circles in Plot 600) shows a high degree of tracking. To clarify the relationship between noise and tracking, the iFM and FM at different levels of noise for these two ML models are plotted against the axes of the metric (either FM or iFM) value and the level of noise. In Plot 600, the noise added to the original signal increases with a much finer granularity than that recorded in the table above. Plot 600 includes two subplots, namely: (i) an iFM subplot 605 at the top, showing the iFM value 610 (described in Table 1) of the ML model indicating minimum tracking at gradually increasing noise levels and the iFM value 615 (described in Table 2) of the ML model indicating prohibitively high tracking, plotted against the iFM axis 620 and the added noise standard deviation coefficient axis 620; and (ii) an FM subplot 655 at the bottom, showing the FM value 660 (described in Table 1) of the ML model indicating minimum tracking at gradually increasing noise levels and the FM value 665 (described in Table 2) of the ML model indicating prohibitively high tracking, plotted against the FM axis 670 and the added noise standard deviation coefficient axis 675.

[0115] As shown in the table above, it is clear that as the noise on the signal increases, the tracking also increases. This behavior is evident in the iFM subplot 605 and is mirrored in the FM subplot 655. Advantageously, the new technique described herein (iFM technique) for passive inference of signal tracking in multivariate anomaly detection provides a greater separation between the metric values of the ML model showing minimum tracking (values 610 and 660) and the metric values of the ML model showing significant tracking (values 615 and 665) than the FM created by the original tracking characterization technique. This is clearly seen by comparing subplots 605 and 655. The greater separation and discriminative ability are excellent characteristics of the model characterization empirical parameters, as this ability enables clearer decisions to be made based on such parameters.

[0116] Models with very low degrees of tracking have an iFM of nominally 1 or less. Note that for higher dimensional models it can be less than 1. However, if tracking is present, the iFM will be greater than 1. The greater the value of the iFM above 1, the worse the tracking exhibited by the model. Based on extensive parametric testing using many time series datasets with different numbers of signals, different numbers of observations, and different noise ratios, in one embodiment, the alarm condition should be set to iFM > 1.3, indicating that mitigation actions, such as adding additional signals to the model if physically possible for the asset under monitoring, performing moving window smoothing to reduce the noise ratio on the signal, etc., are guaranteed.

[0117] FIG. 7 shows a plot 700 showing the change in the tracking metric value with increasing noise ratio for three ML models, where the size of the curve is monotonically mapped to the degree of tracking for any given ratio of the standard deviation between the observed signal and the residual (or iFM). Plot 700 compares three ML model examples, with the first ML model (shown as a circle in plot 700) showing the minimum degree of tracking, the second ML model (shown as a diamond in plot 700) showing a medium amount of tracking, and the third ML model (shown as an x in plot 700) showing a high amount of tracking. Note that both the second and third models show an unacceptable degree of tracking. Plot 700 consists of two subplots, namely (i) an upper iFM subplot 705 showing the iFM values 710 of the first ML model showing minimum tracking, the iFM values 715 of the second ML model showing increased tracking, and the iFM values 720 of the third ML model showing a high degree of tracking, plotted against the iFM axis 725 and the added noise standard deviation coefficient axis 730 at gradually increasing noise levels respectively, and (ii) a lower FM subplot 755 showing the FM values 760 of the first ML model showing minimum tracking, the FM values 765 of the second ML model showing increased tracking, and the iFM (iFM) values 770 of the third ML model showing a high degree of tracking, plotted against the FM axis 775 and the added noise standard deviation coefficient axis 780 at gradually increasing noise levels respectively.

[0118] As can be seen by comparing the iFM subplot 705 and the FM subplot 755, for the ratio between the standard deviation of any given iFM, or the observed signal, and the standard deviation of the residuals between the ML model predicted signal and the observed signal, the magnitudes of the iFM curves 710, 715, and 720 monotonically map to the degree of tracking indicated by the FM values 760, 765, 770 created by the original tracking characterization technique. The family of iFM curves has different spreads for small and large models. Thus, the systems and methods for passive inference of signal tracking in multivariate anomaly detection described herein, as well as the parametric inference tracking metric characterization and mitigation use cases it requires, constitute a systematic and quantitative framework for evaluating iFM parametric curves. In one embodiment, the family of curves can be generated according to the systems and methods described herein for each possible number of monitoring signals (up to a reasonable maximum, perhaps tens or hundreds of millions of monitoring signals) and the family of curves stored in the library. Then, for any given use case (e.g., for 23 signals, or 8, or 150, or 5000, etc.), the appropriate family of curves can be retrieved from the library to indicate the quantitative degree of tracking (FM) by inputting the iFM value. Thus, the inference metric (iFM) can be used by a processor to examine the quantitative metric (FM). The processor then searches for the quantitative degree of tracking (FM) of the machine learning model based on the value of the ratio (iFM). The FM can then be presented with an alarm regarding unacceptable levels of tracking, along with the iFM that triggered the alarm.

[0119] High-level process for inferring tracking within an ML model FIG. 8 shows one embodiment of a method 800 related to passive inference of signal following in multivariate anomaly detection. Method 800 shows a high-level overview of a process for calculating and tracking the iFM of any database of time series signals. In one embodiment, the steps of method 800 are performed by a passive signal inference component 440 (shown and described with reference to FIG. 4). In one embodiment, method 800 can be automatically initiated based on various triggers as described above with reference to method 500. Method 800 begins at start block 805 and proceeds to process block 810. At process block 810, the processor calculates the mean standard deviation of the measurements of the time series signals within a set of multiple time series signals, as shown and described with reference to blocks 515, 520, 525, and 535 of method 500, for example. Next, at process block 815, the processor trains a machine learning model to predict the values of the time series signals, as shown and described with reference to block 540 of method 500, for example. The process then proceeds to process block 820, where the processor predicts the respective values of the signals with the trained machine learning model, as shown and described with reference to block 545 of method 500, for example. The process proceeds to process block 825, where the processor generates a time series set of the residuals between the predicted and measured values for each of the signals, as shown and described with reference to block 550 of method 500, for example. Next, at process block 830, the processor calculates the mean standard deviation of the set of residuals, as shown and described with reference to blocks 555, 560, 565, 570, and 575 of method 500, for example. The process then proceeds to process block 835, where the processor determines that there is signal following within the trained machine learning model if the ratio of the mean standard deviation of the measurements to the mean standard deviation of the set of residuals exceeds a threshold, as shown and described with reference to blocks 580 and 585 of method 500, for example. The process then proceeds to process block 840, where the processor presents an alert indicating that there is signal following within the trained machine learning model, as shown and described with reference to block 590 of method 500, for example. The process then proceeds to end block 845, where method 800 is complete.

[0120] Selected advantages The systems and methods described herein for passive inference of signal following in multivariate anomaly detection demonstrate several advantages. For example, an ML anomaly detection algorithm enhanced with passive inference of signal following as described herein exhibits a lower Type-II error probability (i.e., false alarm miss probability or "MAP") than conventional exemplars. Oracle's Anomaly Detection Service (ADS), as well as GE PREDIX's Predictive Anomaly Discovery tool, Microsoft Azure, Amazon AWS_Sitewise, SAP / Siemens Intelligent Asset Management, or any IoT cloud operation where a customer analyzes large-scale time series databases for machine learning prediction, can be improved by implementing the systems and methods for passive inference of signal following in multivariate anomaly detection as described herein. Detection and characterization of signal following in predictive ML models for large-scale time series databases are enabled by the systems and methods described herein, and this type of detection and characterization of signal following was previously infeasible with large-scale time series datasets.

[0121] Embodiments of computing devices FIG. 9 shows an example of a dedicated computing device having one or more of the systems and methods described herein and / or a computing device configured and / or programmed as equivalent thereto, computing device example 900. The computing device example can be a computer 905 including a processor 910, a memory 915, and an input / output port 920 operably connected by a bus 925. In one example, the computer 905 can include passive inference of signal tracking in a multivariate anomaly detection logic 930 configured to facilitate passive inference of signal tracking in multivariate anomaly detection similar to the logic, systems, and methods shown and described with reference to FIGS. 4-8. In different examples, the logic 930 can be implemented in hardware, a non-transitory computer-readable medium storing instructions 937, firmware, and / or a combination thereof. The logic 930 is shown as a hardware component mounted on a bus 908, but it should be understood that in other embodiments, the logic 930 may be implemented in the processor 902, stored in the memory 904, or stored on the disk 906.

[0122] In one embodiment, the logic 930 or the computer is a means (e.g., structure, i.e., hardware, non-transitory computer-readable medium, firmware) for performing the described actions. In some embodiments, the computing device can be a server operating within a cloud computing system, a server configured in a software as a service (SaaS) architecture, a smartphone, a laptop, a tablet computing device, etc.

[0123] The means can be implemented, for example, as an ASIC programmed to provide passive inference of signal tracking in multivariate anomaly detection. The means can also be implemented as stored computer-executable instructions presented to the computer 900 as data 916 temporarily stored in the memory 904 and then executed by the processor 902.

[0124] Logic 930 can also provide means (e.g., hardware, non-transitory computer-readable media storing executable instructions, firmware) for performing passive inference of signal following in multivariate anomaly detection.

[0125] Generally describing the configuration example of computer 900, processor 902 can be various processors including dual microprocessors and other multiprocessor architectures. Memory 904 can include volatile memory and / or non-volatile memory. Non-volatile memory can include, for example, ROM, PROM, etc. Volatile memory can include, for example, RAM, SRAM, DRAM, etc.

[0126] Storage disk 906 can be operably connected to computer 900, for example, via input / output (I / O) interface (e.g., card, device) 918 and input / output port 910 controlled by at least input / output (I / O) controller 940. Disk 906 can be, for example, a magnetic disk drive, a solid state disk drive, a floppy disk drive, a tape drive, a Zip drive, a flash memory card, a memory stick, etc. Further, disk 906 can be a CD-ROM drive, a CD-R drive, a CD-RW drive, a DVD ROM, etc. Memory 904 can store, for example, process 914 and / or data 916. Disk 906 and / or memory 904 can store an operating system that controls and allocates the resources of computer 900.

[0127] Computer 900 can interact with, control, and / or be controlled by an input / output (I / O) device via an input / output (I / O) controller 940, an I / O interface 918, and an I / O port 910. The I / O devices can include one or more displays 970, a printer 972 (such as an inkjet printer, a laser printer, a 3D printer, etc.), and an audio output device 974 (such as speakers or headphones), a text input device 980 (such as a keyboard), a pointing or selection device 982 (such as a mouse, a trackball, a touchpad, a touch screen, a joystick, a pointing stick, a stylus mouse, etc.), an audio input device 984 (such as a microphone), a video input device 986 (such as a video camera or a still camera), a video card (not shown), a disk 935, a network device 920, a sensor 990, etc. The I / O port 910 can include, for example, a serial port, a parallel port, and a USB port.

[0128] Computer 900 can operate within a network environment and thus can be connected to a network device 920 via an I / O interface 918 and / or an I / O port 910. Through the network device 920, computer 900 can interact with a network 960. Through the network 960, computer 900 can be logically connected to a remote computer 965 and a sensor 990. Networks with which computer 900 can interact include, but are not limited to, LANs, WANs, and other networks. In one embodiment, computer 900 can be connected to sensor 990 via an I / O port 910 or network 960 to receive sensed information from and / or control sensor 990.

[0129] Embodiments of Software Modules Generally, software instructions are designed to be executed by one or more appropriately programmed processors that access memory. These software instructions can include, for example, computer-executable code and source code that can be compiled into computer-executable code. These software instructions can also include instructions written in an interpreted programming language such as a scripting language.

[0130] In a complex system, such instructions can be arranged within program modules, each having a module that performs a specific task, process, function, or operation. The entire set of modules can be controlled or coordinated by an operating system (OS) or other form of organizational platform in the operation of the modules.

[0131] In one embodiment, one or more of the components described herein are configured as modules stored on a non-transitory computer-readable medium. A module is composed of stored software instructions that, when executed by at least a processor accessing memory or a storage device, cause a computing device to perform the corresponding functions described herein.

[0132] Cloud or enterprise embodiment In one embodiment, the system (time series data service 105) is a computing / data processing system that includes applications for enterprise organizations or an aggregation of distributed applications. The applications and computing system can be configured to operate in or be implemented as a cloud-based network computing system, infrastructure as a service (IAAS), platform as a service (PAAS), or software as a service (SAAS) architecture, or other types of networked computing solutions. In one embodiment, the system provides at least one or more of the functions disclosed herein and a graphical user interface for accessing and operating the functions, and is a centralized server-side application accessed by many users via a computing device / terminal that communicates with this computing system (functioning as a server) through a computer network.

[0133] Definitions and Other Embodiments In another embodiment, the described methods and / or their equivalents can be executed by computer-executable instructions. Thus, in one embodiment, a non-transitory computer-readable / memory medium is composed of stored computer-executable instructions of an algorithm / executable application that, when executed by a machine, causes the machine (and / or related components) to execute the method. Examples of machines include, but are not limited to, processors, computers, servers operating within a cloud computing system, servers configured within a software as a service (SaaS) architecture, smartphones, etc. In one embodiment, the computing device is executed by one or more executable algorithms configured to execute any of the disclosed methods.

[0134] In one or more embodiments, the disclosed method or its equivalent is implemented by either computer hardware configured to execute the method or computer instructions embodied in modules stored on a non-transitory computer-readable medium, which instructions, when executed by at least a processor of a computing device, are configured as an executable algorithm to execute the method.

[0135] For simplicity of explanation, the illustrative methodologies in the figures are shown and described as a series of blocks of an algorithm, but it should be understood that the methodologies are not limited by the order of the blocks. Some blocks may be performed in a different order than shown and described, and / or concurrently with other blocks. Further, fewer than all of the illustrated blocks may be used to execute the example of the methodology. The blocks may be combined or separated into multiple actions / components. Additionally and / or alternatively, other methodologies may use additional actions not shown in the blocks.

[0136] The following includes definitions of selected terms used herein. The definitions include various examples and / or forms of components that are within the scope of the term and can be used in implementations. The examples are not intended to be limiting. Both the singular and plural forms of the terms can be within the scope of the definitions.

[0137] References to "one embodiment", "an embodiment", "one example", "an example", etc. indicate that the embodiment or example so described may include a particular feature, structure, characteristic, property, element, or limitation, but not that all embodiments or examples necessarily include such particular feature, structure, characteristic, property, element, or limitation. Further, repeated use of the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may.

[0138] AAKR: Auto-Associative Kernel Regression API: Application Programming Interface ASIC: Application Specific Integrated Circuit CD: Compact Disc CD-R: Recordable CD CE-RW: Rewritable CD CoAP: Constrained Application Protocol CPU: Central Processing Unit DAQ: Data Acquisition DDS: Data Distribution Service DPWS: Device Profile for Web Services DRAM: Dynamic RAM DVD: Digital Versatile Disc and / or Digital Video Disc EPROM: Erasable PROM EEPROM: Electrically Erasable PROM FM: Follow Metric GPU: Graphics Processing Unit HPC: High Performance Computing HTTP: Hypertext Transfer Protocol I / O: Input / Output IAAS: Infrastructure as a Service iFM: Inferential Follow Metric IoT: Internet of Things JSON: JavaScript Object Notation LAN: Local Area Network M2M: Machine-to-Machine ML: Machine Learning MSET: Multivariate State Estimation Technique MSET2: Multivariate State Estimation Technique owned by Oracle MQTT: Message Queuing Telemetry Transport NLNP: Nonlinear Nonparametric NN: Neural Network PAAS: Platform as a Service PCI: Peripheral Component Interconnect PCIE: PCI Express PROM: Programmable ROM RAM: Random Access Memory REST: Representational State Transfer ROM: Read-Only Memory RTU: Remote Terminal Unit SAAS: Software as a Service SBM: Similarity-Based Modeling SCADA: Supervisory Control and Data Acquisition SNR: Signal-to-Noise Ratio SOAP: Simple Object Access Protocol SPRT: Sequential Probability Ratio Test SRAM: Synchronous RAM SQL: Structured Query Language STD: Standard Deviation SVM: Support Vector Machine TCP / IP: Transmission Control Protocol / Internet Protocol UDP: User Datagram Protocol UPnP: Universal Plug and Play USB: Universal Serial Bus XML: Extensible Markup Language XMPP: Extensible Messaging and Presence Protocol WAN: Wide Area Network As used herein, "data structure" refers to the organization of data within a computing system that is stored in memory, a storage device, or other computerized systems. A data structure can be, for example, any one of a data field, a data file, a data array, a data record, a database, a data table, a graph, a tree, a linked list, etc. A data structure can be formed from many other data structures and can contain this data structure (e.g., a database contains many data records). Other examples of data structures are also possible according to other embodiments.

[0139] As used herein, "computer-readable medium" or "computer storage medium" refers to a non-transitory medium that stores instructions and / or data configured to execute one or more of the disclosed functions when executed. The data can function as instructions in some embodiments. The computer-readable medium can take forms including, but not limited to, non-volatile media and volatile media. Non-volatile media can include, for example, optical disks and magnetic disks. Volatile media can include, for example, semiconductor memory and dynamic memory. Common forms of computer-readable medium include, but are not limited to, floppy disks, flexible disks, hard disks, magnetic tapes, other magnetic media, application specific integrated circuits (ASICs), programmable logic devices, compact discs (CDs), other optical media, random access memory (RAM), read only memory (ROM), memory chips or cards, memory sticks, solid state storage devices (SSDs), flash drives, and other media that a computer, processor, or other electronic device can function with. Depending on the type, when selected for implementation in one embodiment, the medium can include stored instructions of an algorithm configured to execute one or more of the disclosed and / or claimed functions.

[0140] As used herein, "logic" refers to components that are executed by a computer or electrical hardware, a non-transitory medium having stored instructions of an executable application or program module, and / or a combination thereof, for performing any of the functions or actions disclosed herein and / or for causing the performance of functions or actions from another logic, method, and / or system as disclosed herein. Equivalent logics can include firmware, microprocessors programmed with algorithms, individual logics (such as ASICs), at least one circuit, analog circuits, digital circuits, programmed logic devices, memory devices containing instructions of algorithms, etc., all of which can be configured to perform one or more of the disclosed functions. In one embodiment, the logic can include one or more gates, combinations of gates, or other circuit components configured to perform one or more of the disclosed functions. When multiple logics are described, it may be possible to integrate the multiple logics into one logic. Similarly, when a single logic is described, it may be possible to distribute this single logic among multiple logics. In one embodiment, one or more of these logics are corresponding structures related to performing the disclosed and / or claimed functions. The choice of which type of logic to execute can be based on the desired system conditions or specifications. For example, if speed increase is a consideration, hardware will be selected to perform the function. If cost reduction is a consideration, stored instructions / executable applications will be selected to perform the function.

[0141] An "operable connection", i.e., a connection that enables an entity to be "operably connected", is one through which signals, physical communication, and / or logical communication can be transmitted and / or received. An operable connection can include a physical interface, an electrical interface, and / or a data interface. An operable connection can include different combinations of interfaces and / or connections sufficient to enable operable control. For example, two entities can be operably connected to communicate signals directly with each other or through one or more intermediate entities (e.g., a processor, an operating system, logic, a non-transitory computer-readable medium). Logical communication channels and / or physical communication channels can be used to create an operable connection.

[0142] As used herein, "user" includes, but is not limited to, one or more persons, computers or other devices, or combinations thereof.

[0143] Although the disclosed embodiments have been illustrated and described in considerable detail, it is not the intention to limit or in any way limit the scope of the appended claims to such detail. Of course, it is not possible to describe every conceivable combination of components or methodologies for purposes of illustrating various aspects of the subject matter. Accordingly, the disclosure is not limited to the specific details or exemplary examples shown and described. Accordingly, the disclosure is intended to embrace alterations, modifications, and variations that fall within the scope of the appended claims.

[0144] To the extent that the terms "includes" or "including" are used in the detailed description or the claims, these terms are intended to include in a manner similar to the term "comprising" as interpreted when the term is used as a transitional word in a claim.

[0145] To the extent that the term "or" is used in the detailed description or claims (e.g., A or B), this term is intended to mean "A or B or both". When the applicant intends to indicate "only A or B but not both", the phrase "only A or B but not both" is used. Accordingly, the use of the term "or" in this specification is an inclusive use and not an exclusive use.

Claims

1. A method executed by a computer for inferring signal tracking within a machine learning model, comprising: calculating an average standard deviation of measured values of time series signals within a set of multiple time series signals; training the machine learning model to predict values of the signals; predicting, with the trained machine learning model, each value of the signals; generating a time series set of residuals between the predicted values and the measured values for each of the signals; calculating an average standard deviation of the set of residuals; determining that signal tracking exists in the trained machine learning model when a ratio of the average standard deviation of the measured values to the average standard deviation of the set of residuals exceeds a threshold; presenting an alert indicating that the signal tracking exists in the trained machine learning model. A method executed by a computer, comprising the above steps.

2. The method executed by a computer according to claim 1, further comprising calculating the ratio by dividing the average standard deviation of the measured values by the average standard deviation of the set of residuals, wherein the threshold that the ratio exceeds is between 1 and 1.

5.

3. The method executed by a computer according to claim 1 or claim 2, further comprising retrieving a quantitative tracking degree of the machine learning model based on the value of the ratio.

4. The method executed by a computer according to claim 1, wherein the training of the machine learning model and the prediction of the values are performed only once when inferring signal tracking within the machine learning model.

5. The method executed by a computer according to claim 1, claim 2, or claim 4, wherein the machine learning model is a non-linear non-parametric regression model.

6. A program for causing at least a processor of a computer to execute the method according to any one of claims 1 to 5.

7. A computing system, comprising: a processor; a memory operably connected to the processor; a non-transitory computer-readable medium A computing system comprising a processor and a memory, the non-transitory computer-readable medium being operably connected to the processor and the memory and storing computer-executable instructions for inferring signal tracking within a machine learning model, the computer-executable instructions causing the computing system to execute the method according to any one of claims 1 to 5 when executed at least by a processor of the computer.

Images