Monitoring the performance of predictive computer-implemented models

Abstract

According to one aspect, a computer-implemented method is provided for monitoring the performance of a predictive computer-implemented model PCIM for monitoring the state of a first system. The PCIM receives observations as input for a plurality of features relating to the first system, and determines whether to issue a state alert based on the observations. The method includes: obtaining reference information for the PCIM, wherein the reference information for the PCIM includes a first set of values ​​for a plurality of features related to the first system during a first time period; determining a set of reference probability distributions based on the first set of values, the set of reference probability distributions including a corresponding reference probability distribution for each feature determined based on the value of a corresponding feature in the first set of values; obtaining operational information for the PCIM, wherein the operational information for the PCIM includes a second set of values ​​for the plurality of features related to the first system during a second time period after the first time period; determining a set of operational probability distributions based on the second set of values, the set of operational probability distributions including a corresponding operational probability distribution for each feature determined based on the value of a corresponding feature in the second set of values; determining a drift metric for the PCIM, the drift metric representing a measure of performance drift of the PCIM between the first time period and the second time period, wherein the drift metric is based on a comparison of the set of reference probability distributions with the set of operational probability distributions; and outputting the drift metric.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to monitoring of the performance of a predictive computer-implemented model (PCIM) for monitoring system conditions, in particular to a computer-implemented method, apparatus and computer program product for monitoring the performance of a PCIM. BACKGROUND

[0002] Predictive computer-implemented models PCIM (also referred to herein as "predictive models" and "prediction models") are becoming more and more prevalent in many platforms and systems that aim to identify possible disruptions of a service or system in advance and address the issues with minimal disruption to end users of the service or system. In the context of healthcare-based imaging systems (e.g., magnetic resonance imaging (MRI), computed tomography (CT), image-guided therapy (IGT), etc.), building a predictive model and using it to warn relevant parties (e.g., monitoring engineers and service engineers located away from the service or system) of possible system issues and failures enables these system issues and failures to be repaired before they fail, so as to prevent system downtime and / or reduce system downtime. Machine learning and statistics-based models have been developed to enable predictive maintenance of systems.

[0003] Typically, these predictive models are built using historical data and patterns that are extracted from data logs of the system by data scientists together with subject matter experts. Based on new incoming data, these models are able to determine a score for the system on a daily basis and issue a warning for the system when appropriate. However, the system can evolve over time, e.g., there can be changes in the system hardware and / or there can be subtle drifts or changes in usage patterns, software, firmware, etc. This can cause the accuracy of the predictive model to drift or degrade over time. For example, if the structure of the system data that is input into the predictive model changes, or if changes in the software cause certain keywords to change, the performance of the predictive model is affected. Of course, other types of changes can occur that can affect the predictive performance of the predictive model. SUMMARY

[0004] While it is known that fine-tuning or correction of the predictive model over a period of time can be required so that the model health (i.e., in terms of predictive performance) does not degrade or degrades only to a limited extent, there is no reliable method or technique that is able to automatically monitor the predictive model and provide an indication about the performance of the predictive model. Currently, monitoring of the predictive model is subjective and based on a subject matter expert (e.g., an engineer) reviewing the output of the predictive model with past service history and system logs to assess the current health of the model.

[0005] These issues of model drift and degradation, and current methods for evaluating the performance of predictive models, have led to consideration of how to automatically monitor predictive models (or at least significantly reduce the need for subject matter experts or others to manually review the performance of the models). With such automatic monitoring, in the event that a performance drift or degradation of a predictive model is identified, it is possible to identify (and also to implement) appropriate corrections to the predictive model.

[0006] According to a first aspect, there is provided a computer-implemented method of monitoring performance of a predictive computer-implemented model, PCIM, for monitoring a state of a first system. The PCIM receives, as input, observed values for a plurality of features relating to the first system, and the PCIM determines whether to raise a state alert based on the observed values. The method comprises obtaining reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features relating to the first system in a first time period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions comprising a respective reference probability distribution for each of the features determined from values of the respective feature in the first set of values; obtaining operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features relating to the first system in a second time period after the first time period; determining a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features determined from values of the respective feature in the second set of values; determining a drift measure for the PCIM, the drift measure representing a measure of performance drift of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions; and outputting the drift measure. Thus, the first aspect provides for automatic monitoring of a PCIM to identify when the PCIM is no longer operating correctly based on probability distributions of values of features of a system.

[0007] In some embodiments, the step of determining the drift measure comprises comparing, for each feature relating to the first system, one or more statistical measures of the reference probability distribution for the feature with one or more statistical measures of the operational probability distribution for the feature.

[0008] In these embodiments, the step of comparing can comprise, for each feature relating to the first system and for each statistical measure, determining a distance measure for the feature and the statistical measure from a value of the statistical measure for the reference probability distribution and a value of the statistical measure for the operational probability distribution.

[0009] In these embodiments, the one or more statistical measures can comprise any one or more of: a mean of the probability distribution, a standard deviation of the probability distribution, a density of the probability distribution, and one or more shape parameters defining a shape of the probability distribution.

[0010] In some embodiments, the first set of values for the plurality of features is a set of training values used to train the PCIM in the past, and the first time period is a time period before the PCIM is monitoring the state of the first system. These embodiments are advantageous in that the performance of the PCIM can be monitored in cases where values for the plurality of features during use of the PCIM are not available for analysis.

[0011] In these embodiments, the reference information for the PCIM can further comprise reference performance information indicating an expected reliability of the PCIM in issuing state alerts for the first system based on the set of training values, the operational information for the PCIM can further comprise operational performance information indicating an operational reliability of the PCIM in issuing state alerts for the first system in the second time period, and the drift measure can be further based on a comparison of the reference performance information and the operational performance information.

[0012] In alternative embodiments, the first set of values for the plurality of features is a set of values obtained during use of the PCIM, and the first time period is a time period in which the PCIM is monitoring the state of the first system. These embodiments are advantageous in that the performance of the PCIM can be monitored based on values for the plurality of features as they occur during use of the PCIM, and provide a better baseline for evaluating the performance or drift of the PCIM.

[0013] In these embodiments, the reference information for the PCIM can further comprise reference performance information indicating a reliability of the PCIM in issuing state alerts for the first system in the first time period, the operational information for the PCIM can further comprise operational performance information indicating an operational reliability of the PCIM in issuing state alerts for the first system in the second time period, and the drift measure can be further based on a comparison of the reference performance information and the operational performance information.

[0014] In these embodiments, each of the reference performance information and the operational performance information can comprise one or more of: a true positive rate, a false positive rate, a true negative rate, and a false negative rate.

[0015] In some embodiments, the method further comprises obtaining values of one or more other features related to the first system, the one or more other features comprising any of: presence of a log file for the first system, warranty status of a component of the first system, version of software or firmware used by the first system; and the drift measure is further based on the values of the one or more other features.

[0016] In some embodiments, the method further comprises analyzing the PCIM to identify the plurality of features related to the first system used by the PCIM. These embodiments are advantageous in that a PCIM can be evaluated to automatically identify features to use in an assessment of the PCIM.

[0017] In some embodiments, the method further comprises evaluating the drift measure to identify one or more of the features that contribute to the value of the drift measure; and analyzing the identified one or more features that contribute to the value of the drift measure to determine a correction to the operation of the PCIM to reduce the drift measure. These embodiments are advantageous in that a cause of a drift of a PCIM is identified and tied to the PCIM for which a correction to the drift is suggested.

[0018] In some embodiments, the method further comprises analyzing the determined drift measure to estimate a remaining life of the PCIM.

[0019] According to a second aspect, there is provided a computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method according to the first aspect or any embodiment thereof.

[0020] According to a third aspect, there is provided an apparatus for monitoring performance of a PCIM, the PCIM being for monitoring a state of a first system. The PCIM receives as input observed values for a plurality of features related to the first system, and the PCIM determines whether to issue a state alert based on the observed values. The apparatus comprises a processing unit configured to: obtain reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features related to the first system in a first time period; determine a set of reference probability distributions from the first set of values, the set of reference probability distributions comprising a respective reference probability distribution for each of the features determined from values of a respective feature in the first set of values; obtain operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features related to the first system in a second time period after the first time period; determine a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features determined from values of a respective feature in the second set of values; determine a drift measure for the PCIM, the drift measure representing a measure of a drift in performance of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions; and cause output of the drift measure. Thus, the third aspect provides for automatic monitoring of a PCIM to identify when the PCIM is no longer operating correctly based on probability distributions of values of features of a system.

[0021] In some embodiments, the processing unit is configured to determine the drift measure by comparing, for each feature related to the first system, one or more statistical measures of the reference probability distribution for the feature with one or more statistical measures of the operational probability distribution for the feature.

[0022] In these embodiments, the processing unit can be configured to compare, for each feature related to the first system and for each statistical measure, by determining a distance measure for the feature and the statistical measure from a value of the statistical measure for the reference probability distribution and a value of the statistical measure for the operational probability distribution.

[0023] In these embodiments, the one or more statistical measures can comprise any one or more of: a mean of a probability distribution, a standard deviation of a probability distribution, a density of a probability distribution, and one or more shape parameters defining a shape of a probability distribution.

[0024] In some embodiments, the first set of values ​​for the plurality of features is a set of training values ​​previously used to train the PCIM, and the first time period is the time period prior to the PCIM monitoring the state of the first system. The advantage of these embodiments is that they enable the monitoring of PCIM performance when it is not possible to obtain values ​​for the plurality of features during PCIM usage for analysis.

[0025] In these embodiments, the reference information for the PCIM may further include reference performance information, which indicates the expected reliability of the PCIM when issuing a status alarm for the first system based on the set of training values; the operational information for the PCIM may further include operational performance information, which indicates the operational reliability of the PCIM when issuing a status alarm for the first system during the second time period; and the drift metric may also be based on a comparison between the reference performance information and the operational performance information.

[0026] In an alternative embodiment, the first set of values ​​for the plurality of features is a set of values ​​obtained during the use of the PCIM, and the first time period is the time period during which the PCIM is monitoring the state of the first system. The advantage of these embodiments is that they enable monitoring of PCIM performance based on the values ​​of multiple features that occur during PCIM usage, and provide a better baseline for evaluating PCIM performance or drift.

[0027] In these embodiments, the reference information for the PCIM may further include reference performance information, which indicates the reliability of the PCIM when issuing a status alarm for the first system during the first time period; the operational information for the PCIM may further include operational performance information, which indicates the operational reliability of the PCIM when issuing a status alarm for the first system during the second time period; and the drift metric may also be based on a comparison between the reference performance information and the operational performance information.

[0028] In these embodiments, each of the reference performance information and the operational performance information may include one or more of the following: true positive rate, false positive rate, true negative rate, and false negative rate.

[0029] In some embodiments, the processing unit is further configured to: obtain values ​​of one or more other features related to the first system, the one or more other features including any one of the following: the existence of log files for the first system, the warranty status of components of the first system, the version of software or firmware used by the first system; and the drift metric is also based on the values ​​of the one or more other features.

[0030] In some embodiments, the processing unit is further configured to analyze the PCIM to identify the plurality of features used by the PCIM in relation to the first system. The advantage of these embodiments is that they enable the evaluation of the PCIM to automatically identify the features to be used in the evaluation of the PCIM.

[0031] In some embodiments, the processing unit is configured to: evaluate the drift metric to identify one or more features among the features that contribute to the value of the drift metric; and analyze the identified one or more features that contribute to the value of the drift metric to determine corrections for the operation of the PCIM, thereby reducing the drift metric. The advantage of these embodiments is that they identify the cause of the PCIM drift and pinpoint that cause to the PCIM for which correction of the drift is recommended.

[0032] In some embodiments, the processing unit is further configured to analyze the determined drift metric to estimate the remaining lifetime of the PCIM.

[0033] These and other aspects of the invention will become apparent and elucidated with reference to one or more embodiments described below. Attached Figure Description

[0034] Exemplary embodiments will now be described with reference to the following figures and by way of example only, in which:

[0035] Figure 1 The diagram illustrates the general principles involved in using predictive models in the monitoring and maintenance of systems;

[0036] Figure 2 This is a block diagram illustrating an apparatus according to an exemplary embodiment;

[0037] Figure 3 This is a block diagram illustrating PCIM monitoring models according to various embodiments and different types of information or data that can be used by the PCIM monitoring models to monitor the performance of PCIM.

[0038] Figure 4 It is a flowchart providing a high-level view of the process of monitoring PCIM according to various embodiments;

[0039] Figure 5The illustrations depict methods for monitoring PCIM according to various embodiments;

[0040] Figure 6 The illustrations depict methods for monitoring PCIM according to various embodiments;

[0041] Figure 7 It shows what can be used to determine Figure 4 Box 56 and Figure 5 Various inputs for PCIM drift in step 404; and

[0042] Figure 8 This is a flowchart illustrating a general method for monitoring the performance of PCIM according to various embodiments. Detailed Implementation

[0043] Figure 1 This illustration demonstrates the general principles of creating and using predictive models (PCIMs) in the monitoring and maintenance of systems. Predictive models can be used to monitor any type of system, such as healthcare-based imaging systems including magnetic resonance imaging (MRI), computed tomography (CT), image-guided therapy (IGT), etc. For ease of understanding, in this disclosure, the terms "predictive model," "predictive computer-implemented model," "predictive computer-implemented model," and "PCIM" all refer to models used to monitor the state of a system, such as an MRI scanner or a CT scanner. A PCIM can be any type of computer-implemented machine learning model, such as a support vector machine (SVM) model, a random forest model, or a logistic regression model.

[0044] Data 2 originates from or is provided by the system and includes values ​​for multiple characteristics. These characteristics (or "system characteristics") can relate to various operational or functional aspects of the system, such as error logs, measurement logs, measurements from one or more sensors, software versions, firmware versions, hardware components present in the system, etc. Data 2 may also include information about errors, failures, and / or other problems experienced by the system. Data 2 can be obtained from multiple sources, and it must encompass data from the system over a sufficiently long period to enable the formation of predictive models. For example, Data 2 could encompass data from the system over periods of hours, days, weeks, months, or years, and thus be considered historical data 2.

[0045] In the "Data Processing" box 4, data 2 is collected and processed to allow for data transformation that enables the construction or creation of a predictive model in the "Model Creation" box 6. The predictive model is created so that it can monitor (predict) the system's state based on the values ​​of system parameters and issue an alert if it predicts a problem requiring an alert. The accuracy of the predictive model is determined in the "Model Evaluation" box 8 (e.g., by using the predictive model to make state predictions using historical data 2). If the accuracy of the predictive model is within acceptable limits, it can be deployed (i.e., put into use) in the "Model Deployment" box 10. Thus, the predictive model begins monitoring the system's state and predicts problems related to system performance based on new values ​​for system characteristics. If a problem is predicted, the predictive model can issue an alert.

[0046] After a certain period of time, the performance of the predictive model can be evaluated (e.g., by “scoring” the output of the predictive model (“Model Scoring” box 12)). The output of the predictive model can be scored based on whether an alert has been issued and whether system users have reported problems with the system or whether the system itself has reported problems. As noted above, due to drift of the predictive model and / or various other reasons (e.g., due to changes in system characteristics (e.g., values ​​for certain system characteristics may no longer be available from the system) and / or due to the system characteristics used by the predictive model), the performance of the predictive model may not be the same as expected after “Model Evaluation” box 8 (which uses historical data 2). The “scoring” (output) of the predictive model determined in “Model Scoring” box 12, along with information about whether and when problems occurred in the system (including those directly reported by system users (operators), can be stored in database 14. The values ​​of system characteristics already evaluated by the predictive model can also be stored in database 14.

[0047] Given that the performance of predictive models can drift or degrade over time, it is crucial to provide a quantitative assessment of their performance. The technique proposed in this paper enables the monitoring of predictive model performance and the issuance of alerts regarding it. It will be appreciated that the proposed technique effectively provides a model for monitoring predictive model (PCIM). The model for monitoring PCIM performance is referred to herein as the "PCIM Monitoring Model" or "PMM".

[0048] In some cases, quantitative metrics of predictive model performance are available, such as true positives (TP), false positives (FP), late alerts (LA), and missed alerts (MA), and these metrics can be used to objectively evaluate the performance of predictive models because they indicate the reliability of the PCIM in issuing alerts and predicting system problems. A true positive (also referred to as “real positive” or “sensitivity”) is the number of correct alerts generated by the predictive model (e.g., 15 days after the predictive model issues an alert about the system, users of the system (e.g., customers) call for engineer support). A false positive is the number of times the predictive model incorrectly issues an alert; for example, the predictive model issues an alert, but the users of the system do not make any requests or complaints. A missed alert is the number of times a user of the system makes a request or complaint but the predictive model does not (and does not subsequently) generate an alert. Finally, a late alert is the number of times a user of the system makes a request or complaint before issuing an alert, but the predictive model subsequently generates an alert. These quantitative metrics (referred to herein as “performance information”) can be identified and stored to enable the evaluation of PCIM performance. Specifically, these quantitative metrics can be determined based on information about actual alarms issued by the PCIM and actual alarms issued by the system or PCIM users (e.g., service requests from system operators, operator logs of system failures, etc.). If the occurrence rate of TP generated by the PCIM decreases over time and / or the occurrence rate of FP, LA, and / or MA increases over time, this indicates that the PCIM performance has drifted and some action may need to be taken on the PICM.

[0049] However, direct information about the performance of predictive models (i.e., the occurrence rates of TP, FP, LA, and MA) may not be readily available. Therefore, the technique presented in this paper enables the acquisition of appropriate information about PCIM and allows for the evaluation of PCIM to determine whether its performance has drifted. Optionally, this evaluation can be performed in conjunction with performance information such as TP, FP, LA, and MA.

[0050] Therefore, PMM uses a quantitative approach to monitor PCIM performance. This means that subjective biases can be removed from the decision-making process regarding PCIM performance, thereby helping to deploy production-ready, stable PCIMs more quickly. In various embodiments, PMM can provide information about PCIM degradation (including estimates of when downtime and PCIM replacement may be necessary) and can indicate the appropriate corrections or adjustments needed to improve PCIM performance.

[0051] Before describing these technologies in more detail, Figure 2The document presents apparatus 22, which can be used to implement various embodiments of the techniques described herein, and is particularly capable of monitoring the performance of predictive computer-implemented models (PCIM). In some embodiments, apparatus 22 can also implement PCIM, i.e., apparatus 22 can receive observations of multiple system parameters related to the system, use PCIM to evaluate these values, and output status alerts for the system when required.

[0052] Device 22 is an electronic (e.g., computing) device that includes a processing unit 24 and a memory unit 26. The processing unit 24 is configured or adapted to control the operation of device 22 and implement the techniques described herein for monitoring the performance of PCIM.

[0053] Processing unit 24 can be configured to run or perform the methods described herein. Processing unit 24 can be implemented in a variety of ways using software and / or hardware to perform the various functions described herein. Processing unit 24 may include one or more microprocessors or digital signal processors (DSPs) that can be programmed using software or computer program code to perform desired functions and / or control components of processing unit 24 to achieve desired functions. Processing unit 24 can be implemented as a combination of dedicated hardware (e.g., amplifiers, preamplifiers, analog-to-digital converters (ADCs) and / or digital-to-analog converters (DACs)) for performing some functions and processors (e.g., one or more programmed microprocessors, controllers, DSPs, and associated circuitry) for performing other functions. Examples of components that may be used in various embodiments of this disclosure include, but are not limited to, conventional microprocessors, DSPs, application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0054] Processing unit 24 is connected to memory unit 26, which is capable of storing data, information, and / or signals used for operation and / or execution of the methods described herein by controlling device 22. In some embodiments, memory unit 26 stores computer-readable code executable by processing unit 24, such that processing unit 24, in conjunction with memory unit 26, performs one or more functions (including the methods described herein). Memory unit 26 can include any type of non-transient machine-readable medium, such as cache memory or system memory, including volatile and non-volatile computer memory, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), and electrically erasable PROM (EEPROM), and memory unit 26 can be implemented as memory chips, optical discs (e.g., compact discs (CDs), digital versatile discs (DVDs), or Blu-ray discs), hard disks, magnetic tape storage solutions, or solid-state devices (including memory sticks, solid-state drives (SSDs), memory cards), etc.

[0055] In some embodiments or implementations, memory unit 26 stores all the data required to perform the techniques described herein. In alternative embodiments, some or all of the data required for the techniques described herein is stored in a database or data storage unit 28 separate from device 22. In this case, device 22 (and in particular processing unit 24) can use interface circuitry 30 to access the data stored in data storage unit 28.

[0056] In some embodiments or implementations, memory unit 26 and / or data storage unit 28 are capable of storing historical data 2 used in the past for generating or training PCIM. In some embodiments and implementations, memory unit 26 and / or data storage unit 28 are capable of storing database 14, which includes a “score” determined for the predictive model in the “Model Scoring” box 12 and information about whether and when problems occur in the system, including those directly reported by users of the system and optionally also including values ​​of system features already evaluated by the predictive model.

[0057] Interface circuitry 30 enables data connectivity and / or data exchange with other devices, including one or more of the following: servers, databases (e.g., data storage unit 28), user equipment, systems for predictive model monitoring, and one or more sensors that acquire values ​​of multiple system parameters related to the system over time. This connection can be direct or indirect (e.g., via the Internet), thus interface circuitry 30 can connect device 22 to a network (e.g., the Internet) via any desired wired or wireless communication protocol. For example, interface circuitry 30 can operate using WiFi, Bluetooth, Zigbee, or any cellular communication protocol (including but not limited to GSM, UMTS, LTE, LTE-Advanced, etc.). In the case of a wireless connection, interface circuitry 30 (and therefore device 22) may include one or more suitable antennas for transmitting / receiving over a transmission medium (e.g., air). Alternatively, in the case of wireless connectivity, interface circuitry 30 may include a unit (e.g., a connector or plug) that enables interface circuitry 30 to be connected to one or more suitable antennas external to device 22 for transmitting / receiving over a transmission medium (e.g., air). Interface circuitry 30 is connected to processing unit 24 to enable information or data received by interface circuitry 30 to be provided to processing unit 24, and / or to enable interface circuitry 30 to transmit information or data (e.g., indications of the performance of a predictive model) from processing unit 24.

[0058] In some embodiments, device 22 includes a user interface 32, which includes one or more components that enable a user of device 22 (e.g., an engineer monitoring a system by a predictive model) to input information, data, and / or commands into device 22, and / or enable device 22 to output information or data to the user. User interface 32 may include any suitable input component(s), including but not limited to a keyboard, keys, one or more buttons, switches or dial pads, a mouse, a trackpad, a touchscreen, a stylus, a camera, a microphone, etc., and / or user interface 32 may include any suitable output component(s), including but not limited to a display screen, one or more lights or light elements, one or more speakers, a vibration element, etc.

[0059] Device 22 can be any type of electronic or computing device. For example, device 22 can be a server, computer, laptop, tablet, smartphone, smartwatch, etc., or a part thereof. In some embodiments, device 22 is located remotely from the system monitored by the predictive model. In some embodiments, device 22 is located remotely from the device or apparatus implementing the predictive model. Alternatively, device 22 can be local to or part of the system being monitored, and / or device 22 can be local to or be the device or apparatus implementing the predictive model.

[0060] It will be appreciated that actual implementations of device 22 may include Figure 2 Additional components are shown. For example, device 22 may also include a power source (e.g., a battery) or components that enable device 22 to be connected to an AC power source.

[0061] Figure 3 This is a block diagram illustrating a PCIM according to various embodiments and different types of information or data that can be used by the PCIM to monitor its performance. PMM 40 is shown as a single block, and further details of the operation of PMM 40 will be provided below with reference to the following figures. In all embodiments of the technology described herein, PMM 40 uses time series values ​​for system characteristics, which the PCIM uses to determine the state of the system. These time series values ​​are stored in an operation information database 42. PMM 40 can receive these time series values ​​when they are obtained (e.g., when measuring them and / or when inputting them into the PCIM), or PMM 40 can retrieve them at a later stage. In either case, the time series values ​​for the system characteristics in the operation information database 42 relate to the time period during which the PCIM is being used and the system is being monitored. System characteristics can relate to various operational or functional aspects of the system, such as error logs, measurement logs, measurement results from one or more sensors, software versions, firmware versions, hardware components present in the system, etc.

[0062] In some embodiments, PMM 40 receives information about alarms or problems predicted by PCIM during the time period covered by the time series values ​​stored in the Operational Information Database 42. This information is stored in the Alarm Information Database 44 and is referred to as "predicted alarm information" or "information about predicted alarms". The information in Database 44 may also include, or alternatively includes, information about alarms or problems raised by operators of the system during the time period covered by the time series values ​​stored in Database 42. Information about alarms or problems raised by operators or users is referred to herein as "actual alarm information" or "information about actual alarms". The information in Alarm Information Database 44 may also include, or alternatively includes, information about the reliability of alarms issued by PCIM 40 (e.g., with respect to the occurrence rate of TP, FP, LA, and / or MA during the time period covered by the time series values ​​stored in the Operational Information Database 42).

[0063] In some embodiments, PMM 40 also utilizes training data 2 previously used to train or create the PCIM. Training data 2 involves values ​​for system characteristics over a specific time period and may also include information about system problems (e.g., alarms, faults, etc.) and system problems raised or flagged by system operators or users (e.g., service calls, hardware component replacements, etc.). In some embodiments, training data 2 may also include information about the expected reliability of the trained PCIM in predicting system problems and generating alarms. This information may be any of the expected occurrence rates of TP, FP, LA, and / or MA.

[0064] It will be recognized that although training data 2, operation information database 42 and alarm information database 44 are shown as separate corresponding databases, it is possible to store any two or all of training data 2, operation information database 42 and alarm information database 44 in the same physical memory unit (e.g., memory unit 26 or data storage unit 28).

[0065] Figure 4 This is a flowchart providing a high-level view of the process for monitoring PCIM according to embodiments of the technology described herein. At least a portion of this process can be implemented by device 22. Box 52 represents the Predictive Model (PCIM). Information about PCIM 52 is provided to PMM 40, and the first box of PMM 40 is shown as box 54. The information about PCIM 52 can relate to PCIM 52 in a first time period and a second (subsequent) time period. The first time period can have any suitable length, such as one day, several days, one week, several weeks, one month, or several months. The second time period can also have any suitable length, such as one day, several days, one week, several weeks, one month, or several months.

[0066] The information about PCIM 52 provided to box 54 pertains to a second time period, which includes at least some time series values ​​from the time series values ​​of system characteristics stored in the operation information database 42. The time series values ​​for system characteristics within the second time period are referred to herein as "operation information" and are the values ​​of system characteristics relating to the updated or most recent operation time period of PCIM 52. Operation information can be received via box 54 when it is generated and entered into PCIM 52, or it can be retrieved from the operation information database 42 at a later time.

[0067] Information concerning PCIM 52 relating to the first time period is referred to herein as "reference information". In a preferred embodiment, the reference information is a portion of information stored in the operation information database 42 relating to a time period earlier than the second time period, in which case the first time period is the period during which PCIM 52 is operating and monitoring the system. In an alternative embodiment, the reference information is training data 2, in which case the first time period is the period prior to training and operating PCIM 52.

[0068] The information provided to box 54 regarding PCIM 52 may also include information about errors, malfunctions, and / or other problems experienced by the system during the first and second time periods, including information about alarms issued by PCIM 52 (i.e., predicted alarm information), information about queries or questions raised by users of the system (e.g., reports of malfunctions by system operators), orders for new hardware components by system operators, etc. (i.e., actual alarm information).

[0069] In box 54, PMM 40 determines the probability distribution for each system feature based on the values ​​of the system features in the reference information, and also determines the probability distribution for each system feature based on the values ​​of the system features in the operational information. These probability distributions represent the probability of observing a specific value of the corresponding system feature in the first / second time period, and can indicate the trend of the values ​​of the system features.

[0070] The second PMM box 56 uses a probability distribution to determine whether the performance of PCIM 52 has drifted from the expected performance, specifically by comparing the probability distributions for the first and second time periods. This comparison provides a drift metric indicating the amount of drift in the performance of PCIM 52. Box 56 makes it possible to identify changes in the statistical properties of the values ​​of the system characteristics input to PCIM 52.

[0071] In box 58 (which is optional), if drift is detected in PCIM 52 or sufficient drift is detected in PCIM 52, PMM 40 can determine one or more causes of the drift (e.g., a hardware component of the system malfunctions, the sensor provides inaccurate measurement results, the structure or configuration of the PCIM is no longer suitable for the current version of the system, etc.) and output an indication of one or more causes and a recommended action to correct the drift (e.g., replace the hardware component, calibrate the sensor, or replace the PCIM, etc.).

[0072] In some embodiments, predicted alarm information and actual alarm information can be used to evaluate the performance of PCIM 52. This can be particularly useful when the statistical properties of the system characteristics input to PCIM 52 have not changed significantly, since PCIM 52 was created and / or deployed first (e.g., when the values ​​input to PCIM 52 are within the same limits). Based on the predicted alarm information and actual alarm information, the following metrics indicative of PCIM performance can be calculated.

[0073] True Positives (TP): True positives are the number of correct alarms generated by PCIM 52. A predicted alarm is recognized as true if a customer or other person associated with the system (e.g., an operator or engineer) issues an alarm within the "prediction window" (i.e., the period during which PCIM 52 predicts a problem will occur) and the actual alarm issued indicates the same problem predicted by PCIM 52. Furthermore, after resolving the alarm issued by the customer or other person (e.g., this can be indicated by disabling service case logs in the computer system), the problem that caused the alarm must be resolved. This can be verified by checking the system's log files and the customer alarm database for relevant customer alarms. In other words, TP refers to the number of predicted alarms generated by the model that map to customer calls (actual alarms) within a specific (prediction) window.

[0074] False Positive (FP): An alarm predicted by PCIM 52 is considered false if it does not follow the actual alarm from the system's customers or other users within the prediction window.

[0075] Missed Alert (MA): Here, a client or other user of the system issues an actual alert, but PCIM 52 does not generate or predict the corresponding alert in the prediction window.

[0076] Late Alert (LA): Here, the system’s customers or other users issue an actual alert, but PCIM 52 only anticipates the alert after the actual alert has been issued. This hinders the following actions: proactively taking action to prevent problems from occurring or reducing the time spent resolving them.

[0077] Ideally, PCIM 52 should produce a high proportion of true positives and a much lower proportion of missed alerts and false positives.

[0078] Therefore, in some embodiments, to assess the health of the predictive model 52, the PMM 40 evaluates these metrics, as well as the number of predictive alerts generated by the PCIM 52 and the consistency of providing a large number of TPs. If the trends of TP, FP, MA, and / or LA exceed acceptable limits, this can indicate that the performance of the PCIM 52 is unsatisfactory and may require adjustment or replacement.

[0079] The trend for each PCIM 52 is unique and can be derived by either observing the PCIM 52 during the operation of the PCIM 52 monitoring system and setting the trend as a baseline for the PCIM 52, or fixing or pre-setting the baseline when the PCIM is created.

[0080] When a decrease in alarm counts is not anticipated, but is predicted by PCIM 52 to be decreasing or continuously decreasing (as in some predictive models where alarm counts decrease over time according to design), several scenarios need to be considered. First, the connection between PCIM 52 and the monitored system may be faulty, potentially causing the loss of system characteristic values ​​(e.g., accurate logs) from PCIM 52's perspective. Second, the system's software and / or firmware may have changed, which could alter the wording and / or content of the logs, potentially leading to PCIM 52 losing such logs. Third, the system or a part of the system (e.g., hardware components) may be out of warranty, and alarms related to that system or part may no longer be relevant. Fourth, the system that existed during PCIM 52's development may have reached the end of its lifespan. These factors are important and need to be evaluated periodically to ensure that PCIM 52 is still performing its designed functions.

[0081] Therefore, it is necessary to quantitatively measure or determine the aforementioned metrics and the probability distribution related to system characteristics within a first time period so that PMM 40 can determine whether the performance of PCIM 52 has degraded while monitoring the system. As noted above, in some embodiments, reference information related to PCIM 52 can include values ​​of system characteristics input to PCIM 52 during the operation of the PCIM 52 monitoring system, and optionally can also include information about actual alarms and predicted alarms. In these embodiments, this reference information is referred to as "True Situation" (GT) information.

[0082] Figure 5The flowchart in the document illustrates the monitoring of PCIM 52 according to various embodiments of the technology described herein. Figure 6 The process Figure 6 The monitoring of PCIM 52 according to various embodiments of the technology described herein is also illustrated. Figure 5 and Figure 6 The methods in the text generally correspond to each other (but in...) Figure 6 (The steps are shown in more detail in the text), and generally correspond to Figure 4 The flowchart is shown in the figure. The processing unit 24 in device 22 can, when appropriate, combine any one of the memory unit 26, interface circuitry 30, and user interface 32 to perform operations. Figure 5 or Figure 6 One or more steps of the method in any of the diagrams. Processing unit 24 may perform one or more of these steps in response to running computer program code that can be stored on a computer-readable medium (e.g., memory unit 26).

[0083] Figure 5 Box 401 and Figure 6 Box 501 corresponds to PCIM 52 and therefore does not form part of the illustrated method. Instead, these boxes are used to indicate the steps to which the outputs of PCIM 52 and values ​​of system characteristics are to be provided.

[0084] Once PCIM 52 has been verified and deployed ( Figure 1 (See box 10). PCIM 52 begins generating alarms. These alarms contain information about PCIM 52 and the system's support or the reason for issuing the alarm. Obtaining the values ​​of the system characteristics that would cause PCIM 52 to issue an alarm about the monitored system is crucial for establishing useful real-world information. Therefore, PMM 40 can identify these system characteristics or extract the names of the system characteristics from PCIM 52. Figure 6 Step 502 in the process, and when deploying PCIM 52, it is possible to obtain values ​​for these system characteristics over (at least) a certain period of time (the first time period). It is possible to obtain time-series values ​​over a sufficiently long period (e.g., one month or several months, for example, approximately 6 months), and it is assumed that PCIM 52 performs well (or at least sufficiently well) during the first time period. This also corresponds to... Figure 5 Step 403 in the process.

[0085] Once real-world information is collected, statistical measures can be determined (also...). Figure 5 In step 403 and in Figure 6 (In step 504). Examples of some statistical measures that can be determined include the mean, standard deviation, and frequency (density).

[0086] It can also determine the values ​​of the metrics TP, FP, MA, and LA within the first time period. This corresponds to... Figure 5 Step 402 and Figure 6 Step 503 in the process. These metrics represent the performance trend of PCIM 52. In some embodiments, it is possible to observe the number of new alarms issued by PCIM 52 per day and to set expectations for the number of new alarms. For example, there exists a PCIM 52 where the number of alarms / days decreases over time according to the design, but for some other PCIM 52s, this may indicate that the performance of PCIM 52 has drifted.

[0087] The part of collecting real-world information can be the calculation of the probability distribution of system characteristics (i.e., the probability distribution of the values ​​of system characteristics during the real-world data collection period). Figure 5 Step 403 and Figure 6 Step 504 in the diagram. Statistical measures can be determined based on these probability distributions.

[0088] In step 502, since real-world information is unavailable, PCIM 52 is analyzed to determine the system characteristics used by PCIM 52. These system characteristics can be identified by analyzing one or more computer files implementing PCIM 52. The values ​​of these system characteristics can then be obtained within a first time period.

[0089] In step 503, real-world information can be obtained or system characteristics for PCIM 52 can be identified and used to create GT information. In this step, information about the performance of PCIM 52 (e.g., in terms of actual alarms compared to predicted alarms or in terms of the occurrence rate of TP, FP, MA, and / or LA) can be obtained or calculated, thus enabling the extraction of PCIM 52 performance.

[0090] After real-world data collection, the monitoring of PCIM 52 can begin by tracking its performance using the established real-world information (and the real-world patterns represented in the ground truth information). Significant deviations from these patterns can be interpreted as model drift, and the next step is to identify the source of this drift. PCIM 52 performance is tracked by obtaining values ​​of system characteristics while PCIM 52 is operating. As noted above, these values ​​are obtained over a second time period. The probability distribution and relevant statistical measures are determined based on the values ​​obtained during this second time period.

[0091] As pointed out above, in Figure 5 Step 403 and Figure 6As shown in step 504, probability distributions are determined for the values ​​of one or more system characteristics, and statistical measures are determined based on these probability distributions. The probability distributions and statistical measures are determined separately for the values ​​obtained in the first time period and for the values ​​obtained in the second time period.

[0092] To find a statistical measure for a specific system characteristic, time series values ​​are the specific system characteristic over a relevant time period and are tested to provide the best fit for the values ​​of the system characteristic in terms of probability distribution. For each system characteristic, the distribution of the system characteristic's values ​​is tested against many known types of distributions (e.g., beta, log-normal, normal, gamma, Weibull, etc.), and the known type of distribution with the best fit is selected based on QQplot. The best fit of QQplot is considered to be the QQplot fit with the lowest least squares value between the actual data points in the distribution of values ​​and the fitted data points in the distribution of values. Once the distribution is fixed, the statistical measure defining that distribution is extracted. For example, given that the values ​​of a specific system characteristic have been determined to follow a beta distribution, the following calculations are then performed to determine the statistical measure for that system characteristic. A baseline for the system characteristic is formed based on the statistical measure determined from the system characteristic values ​​over a first time period.

[0093] Based on system characteristics The probability density function (PDF) is:

[0094] (1)

[0095] in, and These are shape parameters, which are tuned to utilize the distribution of values ​​of system features to obtain the best fit for the PDF; and are gamma functions. It is a variable The lower limit, and It is a variable The upper limit.

[0096] when and At that time, the PDF was:

[0097] (2)

[0098] The average value is:

[0099] (3)

[0100] The variance is:

[0101] (4)

[0102] Similarly, for all features of PCIM 52, these distributions and corresponding statistical measures (e.g., values ​​of the mean, gamma function, shape parameters, etc.) are calculated and stored. When PMM 40 receives a new set of system feature values ​​(e.g., values ​​of system features that have appeared since PCIM 52 was put into operation), distance metrics are calculated to find (e.g., in...) Figure 5 Step 403 and Figure 6 (As determined in step 504) is the deviation between the probability distribution / statistical measure of the system feature values ​​during the first time period and the probability distribution / statistical measure for the values ​​of a new set of received system features. In other words, it determines whether the probability distributions / statistical measures are similar to each other. This is in... Figure 5 Executed in step 404 and corresponding to Figure 6 Step 505 in the process.

[0103] A general distance metric can be defined as follows:

[0104] (5)

[0105] in," "This is a PDF of the values ​​obtained within the first time period (i.e., the actual situation information), and..." "This is a PDF of the values ​​obtained during the second time period (i.e., while PCIM 52 is operating). In " Different distance metrics are calculated for different values ​​of "". It's Manhattan distance. It is Euclidean distance, etc.

[0106] Figure 7 It shows that it can be used in Figure 4 Box 56 and Figure 5 In step 404, various inputs are determined to account for the drift of PCIM 52. These include distance values ​​for one or more system characteristics. (As shown in box 70) is used to determine drift. Optionally, quantitative measures of PCIM 52 performance (e.g., TP, FP, LA, and MA) are also used to determine PCIM 52 performance drift. These quantitative measures are represented by box 72. Optionally, information or values ​​for one or more other system characteristics can be used to determine drift, as shown in box 74. These other system characteristics can be the presence (or absence) of the system's log files, the warranty status of system components, the version of software or firmware used by the system, etc. These other system characteristics can be considered "meta-characteristics" because they relate to slowly changing characteristics of the system, rather than being considered as the more dynamic system characteristics mentioned above, such as those obtained by one or more sensors monitoring the system. Figure 5These other system features were obtained in step 406.

[0107] In box 56 / step 404, a regression model can be used to estimate or determine the drift. This regression model treats the above inputs as feature values ​​of a regression model and provides outputs in the form of drift values ​​representing the amount of drift of the PCIM 52 from its real-world performance. Box 56 / step 404 can provide continuous outputs for the drift values, thereby enabling continuous evaluation of the PCIM 52's performance.

[0108] If in Figure 5 Step 404 or Figure 6 If the drift of PCIM 52 is low in step 505 (e.g., below a threshold), it can be determined that PCIM 52 is operating or performing an operation as intended. Figure 6 Step 506). However, if the drift of PCIM52 is too high (e.g., above a threshold), it may be optionally possible to Figure 5 Step 405 and Figure 6 Steps 507 and 508 determine the cause of the drift and provide actions for handling PCIM 52 issues. Figure 5 Box 58, steps 406 and 407 and Figure 6 (Steps 509 and 510). It is also possible, or alternatively, to issue or send alerts to the PCIM 52 developer or other relevant parties indicating that the PCIM 52's performance is increasingly drifting. Alerts may also indicate an estimate of the remaining usable time of the PCIM 52 before repair, adjustment, or replacement is required.

[0109] Box 58 / Step 405 / Step 508 considers the drift value and the values ​​of one or more system features provided in Box 56 to assess the cause of the drift. Specifically, it is possible to identify one or more system features that have drifted, and when the cause of the drift (one or more) can be indicated, it is possible to provide the system feature (one or more) or the aspect of the system to which the system feature (one or more) relates. Optionally, it is possible to output the cause of the drift to the developer of PCIM 52 or other relevant parties.

[0110] In box 58 / step 407 / step 510, suggestions or recommendations can be provided for adjusting (e.g., fine-tuning) one or more system features identified as the cause of drift. These adjustments can relate to how PCIM 52 processes (one or more) system features(s). For example, if the problem is related to changes in a log file, the adjustments can relate to how PCIM 52 parses the system features. In some embodiments, PMM 40 can automatically implement these adjustments. For example, if changes to the log file are detected, PMM 40 can adjust or cause adjustments to the log file parsing structure. As another example, if the values ​​of (one or more) system features(s) are recorded in a different format, PMM 40 can rescale these values ​​to their original scale.

[0111] Figure 8 The flowchart illustrates a general method for monitoring the performance of the PCIM 52 according to the techniques described herein. The processing unit 24 in device 22 can, where appropriate, combine any of the memory unit 26, interface circuitry 30, and user interface 32 to perform [the task]. Figure 8 One or more steps in the method described above. Processing unit 24 may perform one or more steps in response to running computer program code that can be stored on a computer-readable medium (e.g., memory unit 26).

[0112] In the first step (step 801), reference information for PCIM 52 is obtained. As noted above, the reference information includes a set of values ​​for multiple system characteristics related to the system during the first time period.

[0113] Preferably, the first time period is the period during which PCIM 52 is being deployed and monitoring the system. In this case, the set of system characteristic values ​​(reference information) are the values ​​that are input into PCIM 52 and used by PCIM 52 to determine the status of the system. In some examples, the reference information may be the values ​​of the system characteristics obtained during the first 6 months of using PCIM 52, since it can be assumed that PCIM 52 is operating normally during that time.

[0114] In some embodiments, the reference information for PCIM further includes reference performance information indicating the reliability of PCIM in issuing status alarms for the first system during a first time period. The reference performance information may include one or more of the following: true positive rate, false positive rate, true negative rate, and false negative rate. Alternatively, the reference performance information may include predicted alarm information and actual alarm information, wherein the predicted alarm information is about alarms or issues predicted by PCIM 52 during the first time period, and the actual alarm information is about actual alarms issued or actual issues raised by users of the system during the first time period.

[0115] In an alternative embodiment, the reference information is a set of training values ​​for system features previously used to train PCIM 52. In this case, the first time period is the period prior to the deployment of PCIM 52 and spans the values ​​of the system features in the training set. In these embodiments, the reference information for PCIM 52 may also include reference performance information indicating the expected reliability of PCIM 52 when issuing status alerts for the system based on this set of training values. In other words, the reference performance information can indicate the reliability achieved by training PCIM 52. The reference performance information may include one or more of the following: true positive rate, false positive rate, true negative rate, and false negative rate.

[0116] In step 803, a set of reference probability distributions is determined based on the first set of values. This set of reference probability distributions includes a corresponding reference probability distribution for each system feature among the system features, and each probability distribution is determined based on the value of the corresponding system feature in the first set of values. This can be referenced as described above. Figure 5 Step 403 and Figure 6 The probability distribution is determined as described in step 504.

[0117] Next, in step 805, operational information for PCIM 52 is obtained. This operational information includes a set of values ​​for multiple system characteristics related to the system during a second time period following the first time period. The second time period is the period during which PCIM 52 is operating and monitoring the state of the system. The second time period may immediately follow the first time period, or (especially if the reference information is training data for PCIM 52) the second time period may be sometime after the first time period.

[0118] In embodiments where the reference information includes reference performance information, the operational information for the PCIM 52 may further include operational performance information indicating the operational reliability of the PCIM 52 when issuing status alarms for the system during a second time period. The operational performance information may include one or more of the following: true positive rate, false positive rate, true negative rate, and false negative rate. Alternatively, the operational performance information may include predicted alarm information and actual alarm information, wherein the predicted alarm information is information about alarms or problems predicted by the PCIM 52 during the second time period, and the actual alarm information is information about actual alarms issued or actual problems raised by users of the system during the second time period.

[0119] In step 807, a set of operational probability distributions is determined based on the set of values ​​obtained in step 805. This set of operational probability distributions includes the corresponding operational probability distribution for each system feature among the system features determined based on the values ​​of the corresponding system features in the second time period. (See above for reference.) Figure 5 Step 403 and Figure 6 The probability distribution is determined as described in step 504.

[0120] It will be appreciated that in some embodiments, steps 801 and 803 may be performed at a time prior to the second time period, thus performing steps 801 and 803 at a time before the performance of PCIM 52 is to be evaluated. In other embodiments, steps 801-807 may be performed when the performance of PCIM 52 is to be evaluated.

[0121] Next, in step 809, a drift metric is determined for PCIM 52, which represents a measure of the performance drift of PCIM 52 between the first time period and the second time period. The drift metric is based on a comparison of this set of reference probability distributions with this set of operational probability distributions. This can be referenced as described above. Figure 5 Step 404 and Figure 6 Determine the drift metric as described in steps 505-508.

[0122] In some embodiments, step 809 includes: for each system characteristic, comparing one or more statistical measures for a corresponding reference probability distribution with one or more statistical measures for a corresponding operating probability distribution. (See above regarding...) Figure 5 Step 403 and Figure 6 As indicated in step 504, the statistical metric can be any one or more of the following: the mean of the probability distribution, the standard deviation of the probability distribution, the density of the probability distribution, and one or more shape parameters defining the shape of the probability distribution. In some embodiments, the statistical metric is compared by determining a distance metric for a system feature based on: the value of a statistical metric for a reference probability distribution for the system feature, and the value of a statistical metric for an operational probability distribution for the system feature. The distance measurement can be determined as described above with reference to formula (5).

[0123] In embodiments where the reference information and operational information include performance information, the drift metric determined in step 809 can also be based on a comparison between the reference performance information and the operational performance information (e.g., a comparison between the TP occurrence rate in a first time period and the TP occurrence rate in a second time period).

[0124] In step 811, a drift metric is output. For example, the drift metric can be output to the operator or developer of PCIM 52. Alternatively or additionally, the drift metric can be output to a subsequent step in which the cause of the drift is determined.

[0125] In some embodiments, the method may further include obtaining values ​​for one or more other characteristics (meta-characteristics) related to the system, wherein the other characteristics include any of the following: the presence of log files for the system, the warranty status of system components, or the version of software or firmware used by the system. This corresponds to Figure 5 Step 406 in the above. In these embodiments, the drift metric is also based on the reference above. Figure 7 The value of one or more other characteristics described.

[0126] In some embodiments, PMM 40 may not initially have information about PCIM 52 or the system that PCIM 52 is monitoring. In such cases, PMM 40 needs to determine the system characteristics monitored by PCIM 52 in order to obtain reference information. Therefore, prior to step 801, the method may also include analyzing PCIM 52 to identify multiple system characteristics related to the system used by PCIM 52. This step may include analyzing computer files associated with PCIM 52 to identify system characteristics used as input to PCIM 52.

[0127] In some embodiments, after step 809, the method further includes evaluating a drift metric to identify one or more system features among those that contribute to the value of the drift metric. This corresponds to Figure 6 Step 508. It is possible to analyze one or more system characteristics that contribute to the value of the drift metric to determine corrections for the operation of PCIM 52, thereby reducing the drift metric. This corresponds to... Figure 5 Step 407. In some embodiments, the method may further include analyzing the determined drift metric to estimate the remaining lifetime of the PCIM 52.

[0128] Therefore, techniques are provided for automatically monitoring the performance of predictive models without requiring subject matter experts or others to manually review the performance of the predictive models. In some embodiments, appropriate corrections to the predictive models can be identified and implemented upon recognizing performance drift or degradation.

[0129] Those skilled in the art, through studying the accompanying drawings, disclosure, and claims, will be able to understand and implement variations of the disclosed embodiments in practice with respect to the principles and techniques disclosed herein. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude multiple. A single processor or other unit can implement the functions of several items recited in the claims. Although certain measures are recited in dissimilar dependent claims, this does not imply that combinations of these measures cannot be advantageously used. Computer programs can be stored or distributed on suitable media, such as optical storage media or solid-state media supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. No reference numerals in the claims should be construed as limiting the scope.

Claims

1. A computer-implemented method of monitoring performance of a predictive computer-implemented model (PCIM) for monitoring a state of a first system, wherein, The PCIM receives as input observed values for a plurality of features related to the first system, and determines whether to issue a state alert based on the observed values, wherein the method comprises: obtaining reference information for the PCIM, wherein the reference information for the PCIM comprises a first set of values for the plurality of features related to the first system in a first time period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions comprising a respective reference probability distribution for each of the features determined from values of the respective feature in the first set of values; obtaining operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features related to the first system in a second time period after the first time period; determining a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features determined from values of the respective feature in the second set of values; determining a drift measure for the PCIM, the drift measure representing a measure of performance drift of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions; and outputting the drift measure.

2. The method of claim 1, wherein, The step of determining the drift measure comprises, for each feature related to the first system, comparing one or more statistical measures of the reference probability distribution for the feature with one or more statistical measures of the operational probability distribution for the feature.

3. The method of claim 2, wherein, The step of comparing comprises, for each feature related to the first system and for each statistical measure, determining a distance measure for the feature and the statistical measure from values of the statistical measure for the reference probability distribution and values of the statistical measure for the operational probability distribution.

4. The method of claim 2 or 3, wherein, The one or more statistical measures comprise any one or more of: a mean of a probability distribution, a standard deviation of a probability distribution, a density of a probability distribution, and one or more shape parameters defining a shape of a probability distribution.

5. The method of any one of claims 1-3, wherein, The first set of values for the plurality of features is a set of training values used to train the PCIM in the past, and the first time period is a time period before the PCIM is monitoring the state of the first system.

6. The method of claim 5, wherein: the reference information for the PCIM further comprises reference performance information indicating an expected reliability of the PCIM in issuing a state alert for the first system based on the set of training values; the operational information for the PCIM further comprises operational performance information indicating an operational reliability of the PCIM in issuing a state alert for the first system in the second time period; and the drift measure is further based on a comparison of the reference performance information and the operational performance information.

7. The method of any one of claims 1-3, wherein, The first set of values for the plurality of features is a set of values obtained during use of the PCIM, and the first time period is a time period in which the PCIM is monitoring the state of the first system.

8. The method of claim 7, wherein: The reference information for the PCIM further includes reference performance information indicating a reliability of the PCIM in issuing state alerts for the first system in the first time period; The operational information for the PCIM further includes operational performance information indicating an operational reliability of the PCIM in issuing state alerts for the first system in the second time period; and The drift measure is further based on a comparison of the reference performance information and the operational performance information.

9. The method of claim 6 or 8, wherein, Each of the reference performance information and the operational performance information includes one or more of: a true positive rate, a false positive rate, a true negative rate, and a false negative rate.

10. The method of any one of claims 1-3, 6, and 8, wherein, The method further includes: obtaining values of one or more other features related to the first system, the one or more other features including any of: a presence of a log file for the first system, a warranty status of a component of the first system, a version of software or firmware used by the first system; and wherein the drift measure is further based on the values of the one or more other features.

11. The method of any one of claims 1-3, 6, and 8, wherein, The method further includes: analyzing the PCIM to identify the plurality of features related to the first system used by the PCIM.

12. The method of any one of claims 1-3, 6, and 8, wherein, The method further includes: evaluating the drift measure to identify one or more of the features that contribute to a value of the drift measure; and analyzing the identified one or more of the features that contribute to the value of the drift measure to determine a correction to an operation of the PCIM to reduce the drift measure.

13. The method of any one of claims 1-3, 6, and 8, wherein, The method further includes: analyzing the determined drift measure to estimate a remaining life of the PCIM.

14. A computer program product comprising a computer readable medium having computer readable code embodied therein, the computer readable code being configured such that, on execution by a suitable computer or processor, the computer or processor is caused to perform the method of any one of claims 1-13.

15. An apparatus for monitoring performance of a predictive computer-implemented model (PCIM) for monitoring a state of a first system, wherein, The PCIM receives as input observed values for a plurality of features related to the first system, and the PCIM determines whether to issue a state alert based on the observed values, wherein the apparatus comprises a processing unit configured to: obtain reference information for the PCIM, wherein the reference information for the PCIM includes a first set of values for the plurality of features related to the first system in a first time period; determine a set of reference probability distributions from the first set of values, the set of reference probability distributions including a respective reference probability distribution for each of the features determined from values of the respective feature in the first set of values; determine a set of operational probability distributions from the second set of values, the set of operational probability distributions including a respective operational probability distribution for each of the features determined from values of the respective feature in the second set of values; and determine a drift measure based on a comparison of the set of reference probability distributions and the set of operational probability distributions. obtaining operational information for the PCIM, wherein the operational information for the PCIM comprises a second set of values for the plurality of features related to the first system in a second time period after the first time period; determining a set of operational probability distributions from the second set of values, the set of operational probability distributions comprising a respective operational probability distribution for each of the features determined from values of a respective feature in the second set of values; determining a drift measure for the PCIM, the drift measure representing a measure of performance drift of the PCIM between the first time period and the second time period, wherein the drift measure is based on a comparison of the set of reference probability distributions and the set of operational probability distributions; and causing output of the drift measure.

Images