Monitoring performance of predictive computer-implemented models.

Abstract

According to one aspect, a computer-implemented method is provided for monitoring the performance of a predictive computer-implemented model (PCIM) used to monitor the condition of a first system, wherein the PCIM receives as input observations of a plurality of features related to the first system, and the PCIM determines whether to issue a condition alert based on the observations. The method includes the steps of: acquiring reference information for the PCIM, the reference information for the PCIM including a first set of values ​​for a plurality of features associated with a first system during a first period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions having a respective reference probability distribution for each feature determined from values ​​of the respective features in the first set of values; acquiring operational information for the PCIM, the operational information for the PCIM including a second set of values ​​for a plurality of features associated with the first system during a second period after the first period; determining a set of operational probability distributions from the second set of values, the set of operational probability distributions including a respective operational probability distribution for each feature determined from values ​​of the respective features in the second set of values; determining a drift measure for the PCIM representing a measure of drift in performance of the PCIM between the first period and the second period, the drift measure being based on a comparison between the set of reference probability distributions and the set of operational probability distributions; and outputting the drift measure.

Description

[Technical field]

[0001] The present disclosure relates to monitoring the performance of predictive computer-implemented models (PCIMs) used to monitor the state of a system, and more particularly to a computer-implemented method, apparatus and computer program product for monitoring the performance of a PCIM. [Background technology]

[0002] Predictive computer-implemented models, PCIMs (also referred to herein as "predictive models" and "forecasting models"), are becoming increasingly common across many platforms and systems, with the goal of identifying potential disruptions to a service or system early and resolving the issue with minimal disruption to end users of the service or system. In the context of medically based imaging systems, such as magnetic resonance imaging (MRI), computed tomography (CT), and image-guided therapy (IGT), predictive models have been built and used to alert stakeholders (e.g., monitoring and service engineers located remotely from the service or system) of system issues and failures that may be serviceable prior to failure to prevent and / or reduce system downtime. Machine learning and statistics-based models have been developed to achieve predictive maintenance of systems. Summary of the Invention [Problem to be solved by the invention]

[0003] Typically, these predictive models are built using historical data and patterns that data scientists, together with experts in the field, extract from the system's data logs. These models can determine a system's score daily based on new input data and send system alerts as needed. However, systems may evolve over time, e.g., there may be changes in aspects of the system hardware and / or subtle drifts or changes in usage patterns, software, firmware, etc., that can cause the accuracy of the predictive models to drift or degrade over time. For example, changes in the structure of the system data that is input to the predictive models or changes in certain keywords due to software changes can affect the performance of the predictive models. Of course, other types of changes may occur that can affect the predictive performance of the predictive models.

[0004] Although it is known that predictive models need to be fine-tuned or calibrated over a period of time so that there is no or limited degradation of the model's health (i.e., predictive performance), there is no reliable approach that can automatically monitor predictive models and provide metrics on the performance of the predictive models. Currently, predictive model monitoring is subjective and based on domain experts (e.g., engineers) reviewing the output of the predictive model with past service history and system logs to assess the current health of the model.

[0005] These issues with model drift and degradation and current approaches for assessing predictive model performance lead to consideration of how to automate the monitoring of predictive models (or at least substantially reduce the need for domain experts or others to manually review the model's performance). With such automated monitoring, it may be possible to identify (and implement) appropriate corrections to the predictive model when drift or degradation in the predictive model's performance is identified. [Means for solving the problem]

[0006] According to a first aspect, there is provided a computer-implemented method for monitoring performance of a predictive computer-implemented model PCIM used to monitor a condition of a first system, the PCIM receiving as inputs observed values ​​for a plurality of features related to the first system, the PCIM determining whether to issue a condition alert based on the observed values. The method includes the steps of obtaining reference information for a PCIM, the PCIM reference information including a first set of values ​​for a plurality of features associated with a first system during a first time period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions including a respective reference probability distribution for each feature determined from values ​​of the respective features in the first set of values; obtaining operational information for the PCIM, the PCIM operational information including a second set of values ​​for a plurality of features associated with the first system during a second time period after the first time period; determining a set of operational probability distributions from the second set of values, the set of operational probability distributions including a respective operational probability distribution for each feature determined from values ​​of the respective features in the second set of values; determining a drift measure for the PCIM indicative of a measure of drift in performance of the PCIM between the first time period and the second time period, the drift measure being based on a comparison between the set of reference probability distributions and a set of identical probability distributions; and outputting the drift measure. Thus, a first aspect provides automated monitoring of a PCIM to identify when the PCIM is malfunctioning based on a probability distribution of values ​​of a system feature.

[0007] In some embodiments, determining the drift measure comprises, for each feature associated with the first system, comparing one or more statistical measures for the reference probability distribution of that feature to one or more statistical measures for the operational probability distribution of that feature.

[0008] In these embodiments, the comparing step may include determining, for each feature and for each statistical measure associated with the first system, a distance measure for that feature and statistical measure from the value of the statistical measure with respect to the reference probability distribution and the value of the statistical measure with respect to the operational probability distribution.

[0009] In these embodiments, the one or more statistical measures may include any one or more of a measure of the probability distribution, a standard deviation of the probability distribution, a density of the probability distribution, and one or more shape parameters that define the shape of the probability distribution.

[0010] In some embodiments, the first set of values ​​for the plurality of features is a training set of values ​​used to train the PCIM, and the first time period is a time period before the PCIM monitors the state of the first system. These embodiments have the advantage that performance of the PCIM can be monitored when values ​​for the plurality of features are not available for analysis during use of the PCIM.

[0011] In these examples, the PCIM's baseline information may further include baseline performance information indicative of an expected reliability of the PCIM in issuing a status alert for the first system based on the values ​​of the training set, the PCIM's operational information may further include operational performance information indicative of the PCIM's operational reliability in issuing a status alert for the first system during the second time period, and the drift measure may further be based on a comparison of the baseline performance information and the operational performance information.

[0012] In alternative embodiments, the first set of values ​​for the multiple features is a set of values ​​obtained during use of the PCIM and the first time period is a time period during which the PCIM is monitoring a condition of the first system. These embodiments have the advantage that performance of the PCIM can be monitored based on values ​​of the multiple features that occurred during use of the PCIM, providing a better baseline for assessing performance or drift of the PCIM.

[0013] In these examples, the PCIM's baseline information may further include baseline performance information indicative of the reliability of the PCIM when issuing a first system status alert in a first time period, and the PCIM's operational information may further include operational performance information indicative of the operational reliability of the PCIM when issuing a first system status alert in a second time period, and the drift measure is further based on a comparison of the baseline performance information and the operational performance information.

[0014] In these examples, each of the baseline performance information and the operational performance information may include one or more of a true positive rate, a false positive rate, a true negative rate, and a false negative rate.

[0015] In some embodiments, the method further comprises obtaining values ​​of one or more other features associated with the first system, the one or more other features including any of the presence of a log file of the first system, a warranty status of a component of the first system, or a version of software or firmware used by the first system, and the drift measure is further based on the values ​​of the one or more other features.

[0016] In some embodiments, the method further comprises analysing the PCIM to identify a plurality of features associated with the first system for use by the PCIM. These embodiments have the advantage that the PCIM can be evaluated to automatically identify features to be used in the evaluation of the PCIM.

[0017] In some embodiments, the method further comprises evaluating the drift measure to identify one or more of the features that contributed to the value of the drift measure, and analysing the identified one or more features that contributed to the value of the drift measure to determine a correction to the operation of the PCIM to reduce the drift measure. These embodiments provide the advantage that causes of drift in the PCIM are identified and modifications to the PCIM are suggested to correct the drift.

[0018] In some embodiments, the method further comprises analysing the determined drift measure to estimate a remaining lifetime of the PCIM.

[0019] According to a second aspect, there is provided a computer program product including a computer readable medium having computer readable code embodied therein which, when executed by a suitable computer or processor, configures the computer or processor to perform a method according to the first aspect or any embodiment thereof.

[0020] According to a third aspect, there is provided an apparatus for monitoring performance of a PCIM used to monitor a condition of a first system, the PCIM receiving as input observed values ​​of a plurality of features associated with the first system, the PCIM determining whether to issue a condition alert based on the observed values, the apparatus having a processing unit, the processing unit comprising the steps of: acquiring reference information for the PCIM, the reference information for the PCIM having a first set of values ​​for a plurality of features associated with the first system in a first time period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions including respective reference probability distributions for each feature determined from values ​​of the respective features in the first set of values; acquiring operational information for the PCIM, the operational information for the PCIM representing a performance of the first system in a second time period after the first time period. The system is configured to perform the steps of: determining a set of operational probability distributions from the second set of values, the set of operational probability distributions including a respective operational probability distribution for each feature determined from values ​​of the respective features in the second set of values; determining a drift measure for the PCIM representative of a measure of drift in performance of the PCIM between a first time period and a second time period, the drift measure being based on a comparison between a set of reference probability distributions and a set of operational probability distributions; and outputting the drift measure. Thus, a third aspect provides for automated monitoring of a PCIM to identify when the PCIM is malfunctioning based on a probability distribution of values ​​of system features.

[0021] In some embodiments, the processing unit is configured to determine, for each feature related to the first system, the drift measure by comparing one or more statistical measures for a reference probability distribution of the feature with one or more statistical measures for a behavior probability distribution of the feature.

[0022] In these embodiments, the processing unit may be configured to perform the comparison by determining, for each feature and for each statistical measure associated with the first system, a distance measure and a statistical measure for that feature from the values ​​of the statistical measure of the reference probability distribution and the values ​​of the statistical measure of the operational probability distribution.

[0023] In these embodiments, the one or more statistical measures may include any one or more of: a measure of the probability distribution, a standard deviation of the probability distribution, a density of the probability distribution, and one or more shape parameters that define the shape of the probability distribution.

[0024] In some embodiments, the first set of values ​​for the plurality of features is a training set of values ​​used to train the PCIM, and the first time period is a time period before the PCIM monitors the state of the first system. These embodiments have the advantage that performance of the PCIM can be monitored when values ​​for the plurality of features are not available for analysis during use of the PCIM.

[0025] In these examples, the PCIM's baseline information may further include baseline performance information indicative of an expected reliability of the PCIM in issuing a status alert for the first system based on the values ​​of the training set, the PCIM's operational information may further include operational performance information indicative of the PCIM's operational reliability in issuing a status alert for the first system in the second time period, and the drift measure may be further based on a comparison of the baseline performance information and the operational performance information.

[0026] In alternative embodiments, the first set of values ​​for the plurality of features is a set of values ​​obtained during use of the PCIM and the first time period is a time period during which the PCIM is monitoring a condition of the first system. These embodiments have the advantage that performance of the PCIM can be monitored based on values ​​of the plurality of features that occurred during use of the PCIM, providing a better baseline for assessing performance or drift of the PCIM.

[0027] In these embodiments, the baseline information for the PCIM may further include baseline performance information indicative of the reliability of the PCIM in issuing a first system status alert in a first time period, and the operational information for the PCIM may further include operational performance information indicative of the operational reliability of the PCIM in issuing a first system status alert in a second time period, and the drift measure may further be based on a comparison of the baseline performance information and the operational performance information.

[0028] In these embodiments, each of the baseline performance information and the operational performance information may include one or more of a true positive rate, a false positive rate, a true negative rate, and a false negative rate.

[0029] In some embodiments, the processing unit is further configured to obtain values ​​of one or more other features associated with the first system, the one or more other features including any of the following: a presence of a log file of the first system, a warranty status of a component of the first system, or a version of software or firmware used by the first system, and the drift measure is further based on the values ​​of the one or more other features.

[0030] In some embodiments, the processing unit is further configured to analyze the PCIM to identify a plurality of features associated with the first system that are used by the PCIM. These embodiments have the advantage that the PCIM can be evaluated to automatically identify features to be used in the evaluation of the PCIM.

[0031] In some embodiments, the processing unit is configured to evaluate the drift measure to identify one or more of the features that contributed to the value of the drift measure, and to analyze the identified one or more features that contributed to the value of the drift measure to determine a correction to the operation of the PCIM to reduce the drift measure. These embodiments provide the advantage that a cause of drift of the PCIM is identified and a correction of the PCIM is suggested to correct the drift.

[0032] In some embodiments, the processing unit is further configured to analyze the determined drift measure to estimate a remaining lifetime of the PCIM.

[0033] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter.

[0034] Exemplary embodiments will now be described, by way of example only, with reference to the following drawings in which: [Brief description of the drawings]

[0035] [Figure 1] Diagram showing the general principles involved in using predictive models in system monitoring and maintenance. [Diagram 2] FIG. 1 is a block diagram illustrating an apparatus according to an embodiment. [Diagram 3] 1 is a block diagram illustrating a PCIM monitoring model and various types of information or data that can be used by the PCIM monitoring model to monitor the performance of a PCIM, according to various embodiments. [Figure 4] 1 is a flow diagram providing a high-level view of a PCIM monitoring process in accordance with various embodiments. [Diagram 5] 1 illustrates a method for monitoring a PCIM according to various embodiments. [Figure 6] 1 illustrates a method for monitoring a PCIM according to various embodiments. [Figure 7] 5. FIG. 6 illustrates various inputs that may be used to determine PCIM drift in block 56 of FIG. 4 and step 404 of FIG. [Figure 8] 4 is a flow chart illustrating a general method for monitoring performance of a PCIM, according to various embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] FIG. 1 illustrates the general principles involved in the generation and use of predictive models (PCIMs) in system monitoring and maintenance. Predictive models can be used to monitor any type of system, for example medical-based imaging systems including magnetic resonance imaging (MRI), computed tomography (CT), image-guided therapy (IGT), etc. For ease of understanding, in this disclosure, the terms "predictive model", "predictive model", "predictive computer-implemented model", and "PCIM" all refer to models used to monitor the condition of a system such as an MRI scanner or a CT scanner. A PCIM can be any type of computer-implemented machine learning model, such as a support vector machine (SVM) model, a random forest model, or a logistic regression model.

[0037] Data 2 is provided from or by the system and includes values ​​relating to a number of features. The features (or "system features") may relate to various operational or functional aspects of the system, such as error logs, measurement logs, measurements from one or more sensors, software versions, firmware versions, hardware components present in the system, etc. Data 2 may also include information regarding errors, failures and / or other problems experienced by the system. Data 2 may be obtained from multiple sources and relate to the system over a period of time long enough to allow a predictive model to be formed. For example, data 2 may relate to the system over a period of hours, days, weeks, months or years, in which case it may be considered historical data 2.

[0038] The data 2 are collated and processed in a "Data Processing" block 4, enabling data transformation so that a predictive model can be built or generated in a "Model Generation" block 6. A predictive model can be generated to monitor (predict) the state of the system from the values ​​of the system parameters and to generate an alert if the predictive model predicts that an alert is necessary. The accuracy of the predictive model is determined in a "Model Evaluation" block 8, for example by the predictive model making a state prediction using the historical data 2. If the accuracy of the predictive model is acceptable, the predictive model can be deployed (i.e. put into use) in a "Model Deployment" block 10. The predictive model then starts to monitor the state of the system and predicts problems with the system's performance based on new values ​​for the system features. The predictive model can generate an alert if a problem with the system is predicted.

[0039] The performance of the predictive model can be evaluated by "scoring" the output of the predictive model ("Model Scoring" block 12). The output of the predictive model can be scored based on whether an alert was issued, whether a user of the system reported a problem, or whether the system itself reported a problem. As mentioned above, the performance of the predictive model may not be the same as expected according to "Model Evaluation" block 8 (using historical data 2), due to drift of the predictive model and / or various other reasons, for example due to changes in the system features (e.g., values ​​of some system features may no longer be available from the system) and / or due to system features used by the predictive model. The "score" (output) for the predictive model determined in "Model Scoring" block 12 can be stored in a database 14 together with information on whether and when system problems occurred, which may include problems reported directly by users (operators) of the system. The values ​​of the system features evaluated by the predictive model can be stored in the database 14.

[0040] Given that predictive model performance may drift or deteriorate over time, it is important to be able to provide a quantitative assessment of predictive model performance. The techniques presented herein allow predictive model performance to be monitored and alerts regarding predictive model performance to be issued. It will be appreciated that the techniques presented herein effectively provide a model for monitoring predictive models (PCIMs). A model for monitoring PCIM performance is referred to herein as a "PCIM monitoring model" or "PMM."

[0041] In some cases, quantitative metrics of predictive model performance, such as True Positive (TP), False Positive (FP), Late Alert (LA), and Missed Alert (MA), are available and can be used to indicate the reliability of PCIM in issuing alerts and predicting problems for the system, thus allowing an objective evaluation of the predictive model's performance. True Positive (also called "True Positive" or "Sensitivity") is the number of correct alerts generated by the predictive model, e.g., the number of times a user of the system (e.g., a customer) called engineering support 15 days after the predictive model issued an alert for the system. False Positive is the number of alerts issued incorrectly by the predictive model, e.g., the number of times the predictive model issued an alert but no request or complaint was raised by the user of the system. Missed Alert is the number of times a user of the system raised a request or complaint but the predictive model did not (and ultimately did not) generate an alert. Finally, a Late Alert is the number of times that a request or complaint is raised by a user of the system before the alert is issued, but the predictive model subsequently generates an alert. These quantitative metrics (referred to herein as "performance information") may be determined and stored to enable evaluation of the performance of the PCIM. In particular, these quantitative metrics may be determined from information regarding actual alerts issued by the PCIM and actual alerts issued by users of the system or the PCIM (e.g., service requests by operators of the system, logging of system failures by operators, etc.). If the rate of TP by the PCIM decreases over time and / or the rates of FP, LA, and / or MA increase, this may indicate that the performance of the PCIM has drifted and some action may need to be taken with respect to the PCIM.

[0042] However, direct information on the performance of the predictive model (i.e., TP, FP, LA, MA rates) may not be readily available, and thus the techniques provided herein allow for obtaining appropriate information on the PCIM and for evaluating the PCIM to determine whether its performance has drifted. Optionally, this evaluation can be performed in conjunction with performance information such as TP, FP, LA, MA, etc.

[0043] In this manner, the PMM can monitor PCIM performance using a quantitative approach, i.e., remove subjective bias from the decision-making process regarding PCIM performance, and aid in the rapid deployment of stable, production-ready PCIMs. In various embodiments, the PMM can provide information regarding PCIM degradation, including an estimate of when the PCIM needs to be upgraded or replaced, and can indicate appropriate corrections or adjustments that need to be made to the PCIM to improve performance.

[0044] Before describing the techniques in more detail, Fig. 2 illustrates an apparatus 22 that may be used to implement various embodiments of the techniques described herein, and in particular to monitor the performance of a predictive computer-implemented model (PCIM). In some embodiments, the apparatus 22 may also implement a PCIM, i.e., the apparatus 22 may receive observations for a number of system parameters for the system, evaluate the values ​​using the PCIM, and output status alerts for the system as appropriate.

[0045] Device 22 is an electronic (e.g., computing) device having a processing unit 24 and a memory unit 26. Processing unit 24 is configured or adapted to implement the techniques described herein for controlling the operation of device 22 and monitoring the performance of the PCIM.

[0046] The processing unit 24 can be configured to execute or perform the methods described herein. The processing unit 24 can be implemented in various ways using software and / or hardware to perform various functions described herein. The processing unit 24 can have one or more microprocessors or digital signal processors that can be programmed using software or computer program code to perform the required functions and / or to control the components of the processing unit 24 to perform the required functions. The processing unit 24 can be implemented as a combination of dedicated hardware (e.g., amplifiers, preamplifiers, analog-to-digital converters (ADCs) and / or digital-to-analog converters (DACs)) to perform some functions and processors (e.g., one or more programmed microprocessors, controllers, DSPs and related circuits) to perform other functions. Examples of components that may be employed in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, DSPs, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0047] The processing unit 24 is connected to a memory unit 26 that can store data, information and / or signals for use by the processing unit 24 in controlling the operation of the device 22 and / or in executing or implementing the methods described herein. In some implementations, the memory unit 26 stores computer readable code executable by the processing unit 24 such that the processing unit 24, in conjunction with the memory unit 26, performs one or more functions, including the methods described herein. The memory unit 26 can include any type of non-transitory machine-readable medium, such as cache or system memory, including volatile and non-volatile, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), and the memory unit 26 can also be implemented in the form of a memory chip, an optical disk (e.g., compact disk (CD), digital versatile disk (DVD), Blu-ray disk), a hard disk, a tape storage solution, or a solid-state device, such as a memory stick, a solid state drive (SSD), a memory card, etc.

[0048] In some embodiments or implementations, memory unit 26 stores all data necessary for the techniques described herein to be performed. In alternative embodiments, some or all of the data necessary for the techniques described herein is stored in a database or data storage unit 28 separate from device 22. In this case, device 22, and in particular processing unit 24, can access the data stored in data storage unit 28 using interface circuitry 30.

[0049] In some embodiments or implementations, the memory unit 26 and / or the data storage unit 28 may store historical data 2 used to generate or train the PCIM. In some embodiments and implementations, the memory unit 26 and / or the data storage unit 28 may store a database 14 that includes the "scores" determined for the predictive models in the "model scoring" block 12, along with information about whether and when system problems occurred (including those problems reported directly by users of the system, and optionally the values ​​of system features evaluated by the predictive models).

[0050] The interface circuitry 30 is for enabling data connection and / or exchange with other devices, including any one or more of a server, a database (e.g., data storage unit 28), a user device, a system monitored by the predictive model, and one or more sensors that obtain time values ​​for a plurality of system parameters associated with the system. The connection may be direct or indirect (e.g., via the Internet), and thus the interface circuitry 30 may enable a connection between the device 22 and a network, such as the Internet, via any desired wired or wireless communication protocol. For example, the interface circuitry 30 may operate using WiFi, Bluetooth, Zigbee, or any cellular communication protocol, including, but not limited to, Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Long Term Evolution (LTE), LTE Advanced, etc. In the case of a wireless connection, the interface circuitry 30 (and thus the device 22) may have one or more suitable antennas for transmitting and receiving over a transmission medium (e.g., air). Alternatively, in case of a wireless connection, the interface circuit 30 may have means (e.g. a connector or plug) for enabling the interface circuit 30 to be connected to one or more suitable antennas external to the device 22 for transmitting and receiving via a transmission medium (e.g. air). The interface circuit 30 is connected to the processing unit 24 to enable information or data received by the interface circuit 30 to be provided to the processing unit 24 and / or information or data from the processing unit 24 (e.g. an indication of the performance of the predictive model) to be transmitted by the interface circuit 30.

[0051] In some embodiments, device 22 comprises a user interface 32 that includes one or more components that enable a user of device 22 (e.g., an engineer of the system being monitored by the predictive model) to input information, data and / or commands into device 22 and / or enable device 22 to output information or data to a user of device 22. User interface 32 may have any suitable input component, including a keyboard, a keypad, one or more buttons, switches or dials, a mouse, a trackpad, a touch screen, a stylus, a camera, a microphone, etc., and / or user interface 32 may have any suitable output component, including, but not limited to, a display screen, one or more optical devices or elements, one or more loudspeakers, vibrating elements, etc.

[0052] The device 22 may be any type of electronic or computing device. For example, the device 22 may be or may be part of a server, computer, laptop, tablet, smartphone, smartwatch, etc. In some implementations, the device 22 is remote from the system being monitored by the predictive model. In some implementations, the device 22 is remote from the device or device that implements the predictive model. Alternatively, the device 22 may be at or part of the system being monitored and / or the device 22 may be at or be the device or device that implements the predictive model.

[0053] It will be appreciated that a practical implementation of device 22 may include components in addition to those shown in Figure 2. For example, device 22 may also have a power source, such as a battery, or components to enable device 22 to be connected to a mains power source.

[0054] FIG. 3 is a block diagram illustrating the PMM and various types of information or data that the PMM can use to monitor the performance of the PCIM according to various embodiments. The PMM 40 is shown as a single block, and other details of the operation of the PMM 40 are provided below with reference to subsequent figures. In all embodiments of the technology described herein, the PMM 40 uses a time series of values ​​for system features that the PCIM uses to determine the state of the system. This time series of values ​​is stored in the operational information database 42. The time series of values ​​for the system features may be received by the PMM 40 as they are obtained (e.g., as they are measured and / or as they are entered into the PCIM) or they may be retrieved by the PMM 40 at a later stage. In either case, the time series of values ​​for the system features in the operational information database 42 relate to the period during which the PCIM is being used to monitor the system. The system features may relate to various operational or functional aspects of the system, such as error logs, measurement logs, measurements from one or more sensors, software versions, firmware versions, hardware components present in the system, etc.

[0055] In some embodiments, the PMM 40 receives information regarding alerts or problems predicted by the PCIM 40 during the time period to which the values ​​of the time series stored in the operational information database 42 relate. This information is stored in the alert information database 44 and is referred to as "predicted alert information" or "information regarding predicted alerts." The information in the database 44 may additionally or alternatively include information regarding alerts or problems raised by an operator or user of the system during the time period to which the values ​​of the time series stored in the database 42 relate. Information regarding alerts or problems raised by an operator or user is referred to herein as "actual alert information" or "information regarding actual alerts." The information in the alert information database 44 may additionally or alternatively include information regarding the reliability of alerts issued by the PCIM 40, such as information regarding the rates of TP, FP, LA and / or MA during the time period to which the values ​​of the time series stored in the operational data database 42 relate.

[0056] In some embodiments, the PMM 40 further utilizes training data 2 that was used to train or generate the PCIM. The training data 2 relates to values ​​of system features during a particular time period and may also include information about system problems, such as alerts, failures, and problems raised or flagged by an operator or user of the system, such as service calls, replacement of hardware components, etc. In some embodiments, the training data 2 may also include information about the reliability expected from the trained PCIM in predicting system problems and generating alerts. This information may be any of the expected rates of TP, FP, LA, and / or MA.

[0057] Although the training data 2, the operational information database 42 and the alert information database 44 are each shown as separate databases, it will be understood that any two or all of the training data 2, the operational information database 42 and the alert information database 44 may be stored within or on the same physical memory unit (e.g., the memory unit 26 or the data storage unit 28).

[0058] FIG. 4 is a flow diagram providing a high level view of a process for monitoring PCIMs according to an embodiment of the technology described herein. At least a portion of this process may be implemented by device 22. Block 52 represents a predictive model (PCIM). Information about PCIMs 52 is provided to PMM 40, a first block of which is shown as block 54. Information about PCIMs 52 may be associated with PCIMs 52 during a first period and a (subsequent) second period. The first period may have any suitable length, such as, for example, one day, several days, one week, several weeks, one month, or several months. The second period may also have any suitable length, such as, for example, one day, several days, one week, several weeks, one month, or several months.

[0059] The information of PCIM 52 relating to the second time period provided to block 54 includes at least a portion of the time series of values ​​of the system features stored in operational information database 42. The time series of values ​​of the system features during the second time period are referred to herein as "operational information" and are the values ​​of the system features associated with a more recent or most recent operational period of PCIM 52. The operational information may be received by block 54 at the time the operational information is generated and entered into PCIM 52, or may be obtained from operational information database 42 at a later time.

[0060] The information of the PCIM 52 relating to the first time period is referred to herein as "baseline information." In a preferred embodiment, the baseline information is a portion of the information stored in the operating information database 42 relating to a time period earlier than the second time period, where the first time period is a time period during which the PCIM 52 is operational and monitoring the system. In an alternative embodiment, the baseline information is training data 2, where the first time period is a time period before the PCIM 52 is trained and operational.

[0061] The information regarding PCIM 52 provided to block 54 may also include information regarding errors, failures and / or other problems experienced by the system during the first and second time periods, including information regarding alerts issued by PCIM 52 (i.e., predictive alert information), information regarding inquiries or problems raised by users of the system, such as reports of faults by operators of the system, orders for new hardware components by operators of the system, etc. (i.e., actual alert information).

[0062] In block 54, the PMM 40 determines a probability distribution for each of the system features from the values ​​of the system features in the baseline information and determines a probability distribution for each of the system features from the values ​​of the system features in the operational information, where the probability distributions represent the probability of observing a particular value of the respective system feature in the first / second time periods, respectively, and may indicate a trend in the values ​​of the system features.

[0063] The probability distribution is used by a second PMM block 56 to determine whether the performance of the PCIM 52 has drifted from the desired performance, in particular by comparing the probability distributions for the first and second time periods. This comparison provides a drift measure indicative of the amount of drift in the performance of the PCIM 52. This block 56 allows changes in the statistical characteristics of the values ​​of the system features input to the PCIM 52 to be identified.

[0064] In optional block 58, if drift or sufficient drift in the PCIM 52 is detected, the PMM 40 can determine one or more causes of the drift (e.g., a hardware component in the system has failed, a sensor is providing inaccurate measurements, the PCIM's structure or settings are no longer appropriate for the current version of the system, etc.) and output an indication of the one or more causes and recommended actions to correct the drift (e.g., replacing the hardware component, calibrating the sensor, or replacing the PCIM, etc.).

[0065] In some embodiments, the predicted and actual alert information can be used to evaluate the performance of the PCIM 52. This can be particularly useful when the statistical characteristics of the values ​​of the system features input into the PCIM 52 have not changed substantially (e.g., the values ​​input into the PCIM 52 are within the same limits) since the PCIM 52 was originally generated and / or deployed. From the predicted and actual alert information, the following metrics can be calculated that are indicative of the performance of the PCIM:

[0066] True Positive (TP): True positive is the number of correct alerts generated by PCIM 52. A predicted alert is considered true if a customer or other person (such as an operator or engineer) associated with the system raises an alert within the "prediction window" associated with the alert (i.e., the period during which PCIM 52 predicts that a problem with the system will occur), and if the actual alert raised indicates the same problem predicted by PCIM 52. Furthermore, after resolving the alert raised by the customer or other person (which can be viewed, for example, by closing a service case record for the computer system), the problem that led to the alert should already be resolved. This can be confirmed by reviewing the system's log files and the customer alert database for related customer alerts. In other words, TP relates to the number of predicted alerts generated by the mapping model that are matched to customer calls (actual alerts) within a certain (prediction) window.

[0067] False Positive (FP): If an alert raised by PCIM 52 is not followed within the prediction window by an actual alert from a customer or other user of the system, the alert predicted by PCIM 52 is considered false.

[0068] Missed Alert (MA): Here, an actual alert was raised by a customer or other user of the system, but the PCIM 52 did not generate or predict a corresponding alert within the prediction window.

[0069] Late Alert (LA): Here, an actual alert was raised by a customer or other user of the system, but the PCIM 52 did not predict the alert until the actual alert was raised, preventing proactive action from being taken to prevent the problem from occurring or to reduce the time it takes to resolve the problem.

[0070] Ideally, PCIM52 should produce a high rate of true positives and a much lower rate of missed alerts and false positives.

[0071] Thus, in certain embodiments, these metrics, along with the number of predictive alerts generated by the PCIM 52 and its consistency in providing bulk TP, are evaluated by the PMM 40 to assess the health of the predictive model 52. If there are unacceptable changes in the trends of TP, FP, MA and / or LA, this may indicate unsatisfactory performance of the PCIM 52, and the PCIM 52 may require adjustment or replacement.

[0072] The trend for each PCIM 52 is unique and can be derived by observing the PCIM 52 during its operation to monitor the system and setting this trend as the baseline for the PCIM 52, or the baseline can be determined or predetermined when the PCIM 52 is generated.

[0073] If there is a decrease or a continuing decrease in the number of alerts predicted by PCIM52 when no decrease is expected (some prediction models have a decrease in the number of alerts over time by design), there are several cases to consider. First, there may be a problem with the connection of PCIM52 to the system being monitored, in which case, from the perspective of PCIM52, the value of a system feature such as a correct log may be missed. Second, changes may have been made to the software and / or firmware of the system that may have changed the presentation and / or content of the logs that may cause PCIM52 to miss such logs. Third, a system or part of a system (such as a hardware component) may no longer be under warranty and alerts regarding that system or part of it may no longer be relevant. Fourth, a system that existed during the development of PCIM52 may have reached the end of its life. These factors are important and need to be evaluated from time to time to ensure that PCIM52 is performing the tasks it is designed to perform.

[0074] Thus, the above-mentioned metrics and probability distributions associated with the system features for the first time period need to be quantitatively measured or determined to enable the PMM 40 to make a determination as to whether the performance of the PCIM 52 monitoring the system is degraded. As mentioned above, in some embodiments, the criteria information associated with the PCIM 52 may include values ​​of the system features input to the PCIM 52 during operation of the PCIM 52 monitoring the system, and optionally information regarding actual and predicted alerts. In these embodiments, this criteria information is referred to as "ground truth" (GT) information.

[0075] The flow chart of Figure 5 illustrates monitoring of the PCIM 52 according to various embodiments of the techniques described herein. And the flow chart of Figure 6 illustrates monitoring of the PCIM 52 according to various embodiments of the techniques described herein. The methods of Figures 5 and 6 generally correspond to each other (albeit with the steps shown at a greater level of detail in Figure 6) and generally correspond to the flow diagram of Figure 4. One or more of the steps of the method of either Figure 5 or Figure 6 may be performed by a processing unit 24 in the device 22 in association with any of the memory unit 26, the interface circuitry 30, and the user interface 32, as appropriate. The processing unit 24 may perform one or more steps in response to execution of computer program code, which may be stored in a computer-readable medium, such as the memory unit 26.

[0076] Blocks 401 of Figure 5 and 501 of Figure 6 correspond to PCIM 52 and as such do not form part of the illustrated method, but instead are used to illustrate steps at which the output of PCIM 52 and values ​​of system features are provided.

[0077] Once the PCIM 52 is verified and deployed (block 10 in FIG. 1), the PCIM 52 starts generating alerts. These alerts contain information about the PCIM 52 and the values ​​of some system features of the system that support or provide the reason for this alert. In order to establish useful ground truth value information, it is important to obtain the values ​​of system features that may lead the PCIM 52 to issue an alert for the system being monitored. Thus, the PMM 40 can identify these system features or extract the names of the system features from the PCIM 52 (step 502 in FIG. 6) and obtain the values ​​of (at least) these system features over a period of time (a first period) while the PCIM 52 is deployed. The time series of values ​​can be obtained over a sufficiently long period, such as more than one month, for example about six months, and it is assumed that the performance of the PCIM 52 is good (or at least sufficient) during the first period. This also corresponds to step 403 in FIG. 5.

[0078] Once ground truth value information is collected, statistical measures can be determined (further in step 403 of FIG. 5 and step 504 of FIG. 6). Examples of some statistical measures that can be determined include mean, standard deviation, and frequency (density).

[0079] Values ​​for metrics TP, FP, MA, and LA can also be determined during the first time period. This corresponds to step 402 in FIG. 5 and step 503 in FIG. 6. These metrics represent trends in the performance of the PCIM 52. In some embodiments, the number of new alerts raised by the PCIM 52 each day can be observed and an expectation for the number of new alerts can be set. For example, some PCIMs 52, by design, have a decreasing number of alerts / day over time, while for some other PCIMs 52, this may indicate a drift in the performance of the PCIM 52.

[0080] Part of the collection of ground truth value information may be the calculation of probability distributions for system features (i.e., probability distributions of values ​​of system features over the ground truth value data collection period), as shown in step 403 of Figure 5 and step 504 of Figure 6. Statistical measures can be calculated from these probability distributions.

[0081] In step 502, because ground truth information is not available, the PCIM 52 is analyzed to determine system features used by the PCIM 52. These system features can be identified by analyzing a computer file that implements the PCIM 52. Values ​​for these system features can then be obtained over a first period of time.

[0082] In step 503, ground truth value information (GT information) is available or system features identified for the PCIM 52 can be identified and the identified system features can be used to generate ground truth information. In this step, the performance of the PCIM 52 can be extracted since information on the performance of the PCIM 52 is available or can be calculated, such as information on actual alerts vs. predicted alerts, or rates of TP, FP, MA and / or LA.

[0083] After collection of the ground truth values, the process of monitoring the PCIM 52 can begin by tracking the performance of the PCIM 52 with the established ground truth value information (and the ground truth value pattern represented in the GT information). Any significant deviation from the pattern can be understood as model drift, and the next step can be to identify the source of the drift. The performance of the PCIM 52 is tracked by obtaining values ​​of the system features when the PCIM 52 is operational. As described above, these values ​​are obtained during a second period of time. Probability distributions and associated statistical measures are determined from the values ​​obtained during the second period of time.

[0084] As described above, in step 403 of Figure 5 and step 504 of Figure 6, probability distributions may be determined for values ​​of one or more system features and statistical measures may be determined from these probability distributions. The probability distributions and statistical measures may be determined separately for values ​​obtained during a first time period and for values ​​obtained during a second time period.

[0085] To determine the statistical measures of a particular system feature, a time series of values ​​of the particular system feature for a relevant time period is obtained and a test is performed on the probability distribution that provides the best fit to the values ​​of the system feature. For each system feature, the distribution of the values ​​of the system feature is tested for many known types of distributions (beta, log-normal, normal, gamma, Weibull, etc.) and the known type of distribution that provides the best fit is selected based on the QQplot. The best fit of the QQ plot is considered to be the one that has the lowest least squares value between the actual data points in the distribution of values ​​and the fitted data points in the distribution of values. Once the distribution is established, the statistical measures that define the distribution are extracted. For example, suppose it is determined that the values ​​of a particular system feature follow a beta distribution. The following calculations are then performed to determine the statistical measures of this system feature. The statistical measures determined from the system feature values ​​over a first time period form the baseline of the system feature.

[0086] The probability density function (PDF) of a system feature x is: TIFF0007679831000001.tif11112, where α and β are shape parameters that are adjusted to obtain the best fit to the PDF with the distribution of values ​​of the system features, Γ is the Gamma function, a is a lower bound on the variable x, and b is an upper bound on the variable x.

[0087] For a=0 and b=1, the PDF is: TIFF0007679831000002.tif11104The mean is given by: TIFF0007679831000003.tif9103The variance is given by: TIFF0007679831000004.tif9104

[0088] Similarly, for every feature of PCIM 52, the distribution and corresponding statistical measures (mean, gamma function, values ​​of shape parameters, etc.) are calculated and stored. When a new set of values ​​of the system features is received by PMM 40 (e.g., values ​​of the system features that have occurred since PCIM 52 was activated), a distance metric is calculated to determine the deviation between the probability distribution / statistical measure of the values ​​of the system features during a first period of time (as shown in step 403 of FIG. 5 and step 504 of FIG. 6) and the probability distribution / statistical measure of the new set of received values ​​of the system features. That is, it is determined whether the probability distributions / statistical measures are similar to each other. This is performed in step 404 of FIG. 5 and corresponds to step 505 of FIG. 6.

[0089] A common distance measure can be defined as follows: TIFF0007679831000005.tif9100, where "y" is the PDF of the values ​​obtained over the first period (i.e., ground truth value information) and "z" is the PDF of the values ​​obtained over the second period (i.e., when PCIM52 is operational). Different values ​​of "p" calculate different distance metrics. For example, p=1 is Manhattan distance, p=2 is Euclidean distance, etc.

[0090] FIG. 7 illustrates various inputs that may be used to determine drift of the PCIM 52 in block 56 of FIG. 4 and step 404 of FIG. 5. Distance values ​​dp (represented by block 70) of one or more system features are used to determine drift. Optionally, quantitative measures of performance of the PCIM 52 (e.g., TP, FP, LA, and MA) are also used to determine drift of performance of the PCIM 52. The quantitative measures are represented by block 72. Optionally, information or values ​​of one or more other system features may be used to determine drift, as represented by block 74. These additional system features may include the presence or absence of a log file for the system, warranty status of components of the system, software and firmware versions used by the system, etc. These other system features may be considered "meta-features" since they relate to slowly changing features of the system rather than the more dynamic system features discussed above, such as those obtained by one or more sensors monitoring the system. These other system features are obtained in step 406 of FIG. 5.

[0091] In block 56 / step 404, drift can be estimated or determined by using a regression model that considers the above inputs as feature values ​​for the regression model and provides an output in the form of a drift value representative of the amount of drift of the PCIM 52 from ground truth performance. Block 56 / step 404 can provide a continuous output of drift values ​​that allows the performance of the PCIM 52 to be continually evaluated.

[0092] If the drift of the PCIM 52 is low (e.g., below a threshold) in step 404 of FIG. 5 or step 505 of FIG. 6, it can be determined that the PCIM 52 is operating or working as intended (step 506 of FIG. 6). However, if the drift of the PCIM 52 is too high (e.g., above a threshold), then optionally, the cause of the drift can be determined in step 405 of FIG. 5 and steps 507 and 508 of FIG. 6, and action can be provided to remedy the problem with the PCIM 52 (block 58 of FIG. 4, steps 406 and 407 of FIG. 5, and steps 509 and 510 of FIG. 6). Additionally or alternatively, an alert indicating that the performance of the PCIM 52 is drifting can be issued or sent to the developer of the PCIM 52 or other interested parties. The alert can also indicate an estimate of the remaining time that the PCIM 52 can be used before repair, adjustment, or replacement is required.

[0093] Block 58 / step 405 / step 508 evaluates the reasons for the drift, taking into account the drift values ​​provided by block 56 and the values ​​of the system features. In particular, the system features that have drifted can be identified and these system features, or aspects of the system to which the system features relate, can be provided as reasons for the drift. Optionally, the reasons for the drift can be output to the developer of PCIM 52 or other interested parties.

[0094] At block 58 / step 407 / step 510, suggestions or recommendations may be provided to adjust (e.g., tweak) one or more system features identified as causing drift. These adjustments may relate to how one or more system features are processed by PCIM 52. For example, if the issue is related to changes in a log file, the adjustments may relate to how the system feature is parsed by PCIM 52. In some embodiments, these adjustments may be performed automatically by PMM 40. For example, PMM 40 may adjust or trigger an adjustment to a log file parsing structure if it is found that the log file has been modified. As another example, if the values ​​of a system feature are recorded in a different format, PMM 40 may rescale the values ​​back to the original scale.

[0095] The flowchart of Figure 8 illustrates a general method for monitoring performance of PCIM 52 in accordance with the techniques described herein. One or more of the steps of the method of Figure 8 may be performed by processing unit 24 within device 22 in association with any of memory unit 26, interface circuitry 30, and user interface 32, as appropriate. Processing unit 24 may perform one or more steps in response to execution of computer program code, which may be stored in a computer-readable medium, such as memory unit 26.

[0096] In the first step, step 801, baseline information is obtained for the PCIM 52. As described above, the baseline information includes a set of values ​​for a number of system features associated with the system during a first period of time.

[0097] Preferably, the first period is the period during which the PCIM 52 is deployed and monitoring the system, where a set of values ​​of system features (baseline information) are entered into the PCIM 52 and used by the PCIM 52 to determine the state of the system. In some examples, the baseline information may be values ​​of system features obtained during the first six months that the PCIM 52 is in use, since it can be assumed that the PCIM 52 is operating correctly during this period.

[0098] In some embodiments, the PCIM's baseline information further includes baseline performance information indicative of the PCIM's reliability in issuing status alerts for the first system during the first time period. The baseline performance information may include one or more of a true positive rate, a false positive rate, a true negative rate, and a false negative rate. Alternatively, the baseline performance information may include predicted alert information, which is information regarding alerts or problems predicted by the PCIM 52 during the first time period, and actual alert information, which is information regarding actual alerts or problems raised by users of the system during the first time period.

[0099] In alternative embodiments, the baseline information is a training set of values ​​of the system features used to train the PCIM 52. In this case, the first period is a period prior to deployment of the PCIM 52 and spans the values ​​of the system features in the training set. In these embodiments, the baseline information for the PCIM 52 may further include baseline performance information indicative of an expected reliability of the PCIM 52 in issuing status alerts for the system based on the values ​​of the training set. In other words, the baseline performance information may indicate the reliability that the PCIM 52 was trained to achieve. The baseline performance information may include one or more of a true positive rate, a false positive rate, a true negative rate, and a false negative rate.

[0100] In step 803, a set of reference probability distributions is determined from the first set of values. The set of reference probability distributions includes a respective reference probability distribution for each of the system features, each probability distribution determined from values ​​of the respective system feature in the first set of values. The probability distributions may be determined as described above with reference to step 403 of Figure 5 and step 504 of Figure 6.

[0101] Next, in step 805, operational information of the PCIM 52 is obtained. The operational information of the PCIM 52 includes a set of values ​​of a plurality of system features for the system during a second time period after the first time period. The second time period is a time period during which the PCIM 52 is operating and monitoring the state of the system. The second time period may be immediately after the first time period, or (particularly if the reference information is training data for the PCIM 52) the second time period may be some time after the first time period.

[0102] In embodiments in which the baseline information includes baseline performance information, the operational information of PCIM 52 may further include operational performance information indicative of the operational reliability of PCIM 52 in issuing status alerts for the system during the second time period. The operational performance information may include one or more of a true positive rate, a false positive rate, a true negative rate, and a false positive rate. Alternatively, the operational performance information may include predicted alert information, which is information regarding alerts or problems predicted by PCIM 52 during the second time period, and actual alert information, which is information regarding actual alerts or problems raised by users of the system during the second time period.

[0103] In step 807, a set of operational probability distributions is determined from the set of values ​​obtained in step 805. The set of operational probability distributions includes a respective operational probability distribution for each system feature determined from the values ​​of the respective system features in the second time period. The probability distributions may be determined as described above with reference to step 403 of Figure 5 and step 504 of Figure 6.

[0104] It will be appreciated that in some embodiments, steps 801 and 803 may be performed prior to the second time period, and thus prior to evaluating the performance of PCIM 52. In other embodiments, steps 801-807 may be performed when the performance of PCIM 52 may be evaluated.

[0105] Next, in step 809, a drift measure is determined for the PCIM 52 representing a measure of the drift in performance of the PCIM 52 between the first time period and the second time period. The drift measure is based on a comparison of the set of reference probability distributions to the set of operational probability distributions. The drift measure may be determined as described above with reference to step 404 of Figure 5 and steps 505-508 of Figure 6.

[0106] In some embodiments, step 809 includes comparing, for each system feature, one or more statistical measures of the respective reference probability distribution to one or more statistical measures of the respective operational probability distribution. As described above with respect to step 403 of FIG. 5 and step 504 of FIG. 6, the statistical measures may be any one or more of the mean of the probability distribution, the standard deviation of the probability distribution, the density of the probability distribution, and one or more shape parameters that define the shape of the probability distribution. In some embodiments, the statistical measures are compared by determining a distance measure for a system feature from the value of the statistical measure of the reference probability distribution for that system feature and the value of the statistical measure of the operational probability distribution for that system feature. The distance measure may be determined as described above with reference to equation (5).

[0107] In embodiments in which the baseline information and operational information include performance information, the drift measure determined in step 809 may further be based on a comparison of the baseline performance information to the operational performance information (e.g., a comparison of the TP rate in a first period of time to the TP rate in a second period of time).

[0108] In step 811, the drift measure is output. For example, the drift measure can be output to an operator or developer of PCIM 52. Additionally or alternatively, the drift measure can be output to a subsequent step where the reason for the drift is determined.

[0109] In some embodiments, the method may further include obtaining values ​​of one or more other features (meta-features) for the system, including any of the presence of a log file for the system, the warranty status of a component of the system, or the version of software or firmware used by the system. This corresponds to step 406 in Figure 5. In these embodiments, the drift measure is further based on values ​​of the one or more other features, as described above with reference to Figure 7.

[0110] In some embodiments, PMM 40 initially has no information about PCIM 52 or the system being monitored by PCIM 52, in which case PMM 40 needs to determine the system features being monitored by PCIM 52 in order to obtain baseline information. Thus, prior to step 801, the method may further include analyzing PCIM 52 to identify a number of system features related to the system that will be used by PCIM 52. This step may include analyzing a computer file associated with PCIM 52 to identify system features to be used as input to PCIM 52.

[0111] In some embodiments after step 809, the method further comprises evaluating the drift measure to identify one or more system features that contributed to the value of the drift measure. This corresponds to step 508 of Figure 6. The one or more system features identified as contributing to the value of the drift measure may be analyzed to determine corrections for operation of the PCIM 52 to reduce the drift measure. This corresponds to step 407 of Figure 5. In an embodiment, the method may further include analyzing the determined drift measure to estimate a remaining life of the PCIM 52.

[0112] Thus, techniques are provided for automatically monitoring the performance of a predictive model without the need for a subject matter expert or other person to manually review the performance of the predictive model. In certain embodiments, when drift or deterioration in the performance of a predictive model is identified, appropriate corrections to the predictive model can be identified and implemented.

[0113] Variations to the disclosed embodiments can be understood and implemented by those skilled in the art in implementing the principles and techniques described herein, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used to advantage. A computer program can be stored or distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. Any reference signs in the claims should not be interpreted as limiting their scope.

Claims

1. 1. A computer-implemented method for monitoring performance of a predictive computer-implemented model (PCIM) used to monitor a condition of a first system, the PCIM receiving as inputs observations of a plurality of features related to the first system, the PCIM determining whether to issue a condition alert based on the observations, the computer-implemented method comprising: obtaining PCIM reference information, the PCIM reference information including a first set of values ​​of a plurality of features for the first system during a first time period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions including a respective reference probability distribution for each feature determined from values ​​of each feature in the first set of values; obtaining operational information of the PCIM, the PCIM operational information including a second set of values ​​of a plurality of features for the first system at a second time period after the first time period; determining a set of operation probability distributions from the second set of values, the set of operation probability distributions including a respective operation probability distribution for each feature determined from values ​​of the respective features in the second set of values; determining a drift metric for the PCIM representative of a measure of drift in performance of the PCIM between the first time period and the second time period, the drift metric being based on a comparison of the set of reference probability distributions and the set of operational probability distributions; outputting said drift measure; The method according to claim 1,

2. 2. The method of claim 1 , wherein determining the drift measure comprises, for each feature associated with the first system, comparing one or more statistical measures of the reference probability distribution for that feature to one or more statistical measures of the operation probability distribution for that feature.

3. 3. The method of claim 2, wherein the comparing step comprises determining, for each feature and each statistical measure for the first system, a distance measure for that feature and that statistical measure from a value of the statistical measure in the reference probability distribution and a value of the statistical measure in the operational probability distribution.

4. A method as described in claim 2 or 3, wherein the one or more statistical measures compared include one or more of the mean, standard deviation, density, and one or more shape parameters defining the distribution shape of the reference probability distribution and the operating probability distribution.

5. 5. The method of claim 1, wherein the first set of values ​​for the plurality of features are values ​​of a training set used to train the PCIM, and the first period of time is a period of time before the PCIM monitors a state of the first system.

6. the baseline information for the PCIM further comprises baseline performance information indicative of an expected reliability of the PCIM in issuing a status alert for the first system based on values ​​of the training set; the operational information of the PCIM further includes operational performance information indicative of a reliability of operation of the PCIM in issuing a status alert of the first system during the second time period; the drift measure is further based on a comparison of the baseline performance information to the operational performance information; The method of claim 5.

7. 5. The method of claim 1, wherein the first set of values ​​for the plurality of features is a set of values ​​obtained during use of the PCIM, and the first period of time is a period of time during which the PCIM is monitoring a state of the first system.

8. the PCIM's baseline information further includes baseline performance information indicative of the PCIM's reliability in issuing a status alert for the first system during the first time period; the operation information of the PCIM further includes operational performance information indicating operational reliability of the PCIM when issuing a status alert of the first system during the second period; the drift measure is further based on a comparison of the baseline performance information to the operational performance information; The method of claim 7.

9. The method of claim 6 or 8, wherein each of the baseline performance information and the operational performance information comprises one or more of a true positive rate, a false positive rate, a true negative rate, and a false negative rate.

10. The method further comprises obtaining values ​​of one or more other features related to the first system, the one or more other features including any of the following: the existence of a log file of the first system, a warranty status of a component of the first system, and a version of software or firmware used by the first system; The method of claim 1 , wherein the drift measure is further based on values ​​of the one or more other features.

11. 11. The method of claim 1, further comprising analyzing the PCIM to identify a plurality of features related to the first system that are used by the PCIM.

12. The method further comprises: evaluating the drift measure to identify one or more of the features that contributed to a value of the drift measure; analyzing one or more features that contributed to the identified drift measure value to determine a correction to operation of the PCIM to reduce the drift measure; 12. The method of claim 1 , comprising:

13. The method further comprises:

13. The method of claim 1, further comprising analysing the determined drift measure to estimate a remaining life of the PCIM.

14. A computer readable medium having computer readable code which, when executed by a computer or processor, is configured to cause the computer or processor to perform a method according to any one of claims 1 to 13.

15. 1. An apparatus for monitoring performance of a predictive computer-implemented model (PCIM) used to monitor a condition of a first system, the PCIM receiving as input observed values ​​of a plurality of features related to the first system, the PCIM determining whether to issue a condition alert based on the observed values, the apparatus comprising: obtaining PCIM reference information, the PCIM reference information including a first set of values ​​for the plurality of features for the first system at a first time period; determining a set of reference probability distributions from the first set of values, the set of reference probability distributions including a respective reference probability distribution for each feature determined from values ​​of each feature in the first set of values; obtaining operational information of the PCIM, the operational information of the PCIM including a second set of values ​​of the plurality of features for the first system at a second time period after the first time period; determining a set of operation probability distributions from the second set of values, the set of operation probability distributions including a respective operation probability distribution for each feature determined from values ​​of the respective features in the second set of values; determining a drift metric for the PCIM representative of a measure of drift in performance of the PCIM between the first time period and the second time period, the drift metric being based on a comparison of the set of reference probability distributions and the set of operational probability distributions; providing an output of said drift measure; 23. An apparatus having a processing unit for executing the steps of:

Images