Data monitoring based on machine learning

Abstract

An overall performance metric of the computer system may be determined for each bin of the set of analysis bins. If one or more bins of the set of analysis bins do not have at least a predetermined minimum number of records, a new set of analysis bins may be redefined by combining analysis bins of the set of analysis bins. For each bin of the redefined set of bins, a machine learning (ML) performance metric of the ML model may be calculated. Using the ML performance metric of the redefined bin, an ML performance metric may be estimated for the set of analysis bins. The computer system may be configured based on the correlation between the calculated overall performance metric and the ML performance metric across the set of analysis bins.

Description

[Technical field]

[0001] The present disclosure relates to the field of digital computer systems, and more particularly to methods for controlling the operation of computer systems. [Background technology]

[0002] Machine learning models are increasingly used in data monitoring. However, machine learning models can be inaccurate for several reasons, such as bias in the training data due to one or more of preconceptions in the labels, under- / over-sampling, or generating models with undesired biases. Machine learning monitoring does not always identify these biases. Summary of the Invention

[0003] As described by this disclosure, various embodiments provide methods, systems, and computer program products for controlling operation of a computer system. In one aspect, the disclosure relates to controlling operation of a computer system, the computer system configured to perform data transactions and evaluate characteristics of the data transactions using a machine learning (ML) model.

[0004] A set of analysis bins is determined. The analysis bins represent a set of values ​​of attributes of records of data transactions. An overall performance metric of the computer system is calculated. The overall performance metric is indicative of transaction execution performance of the computer system using, for each analysis bin of the set of analysis bins, records of transactions having attribute values ​​represented by the analysis bin. A new set of analysis bins is redefined by combining the analysis bins of the set of analysis bins if one or more analysis bins of the set of analysis bins do not have at least a predetermined minimum number of records. For each analysis bin of the redefined set of analysis bins, a machine learning performance metric of the ML model is calculated using records having attribute values ​​represented by each analysis bin. The ML performance metric of the redefined set of analysis bins is used to estimate an ML performance metric within each bin of the set of analysis bins. If each analysis bin of the set of analysis bins has at least a minimum number of records, the computer system is configured to enable a positive correlation between the overall performance metric and the ML performance metric of the data transactions further executed based on a correlation across the set of analysis bins between the calculated overall performance metric and the ML performance metric.

[0005] In the following, embodiments of the present disclosure are described in more detail, by way of example only, and with reference to the following drawings, in which: [Brief description of the drawings]

[0006] [Figure 1] 1 is a flowchart of a method for controlling the operation of a computer system according to an example of the present disclosure. [Figure 2A] 1 is a flowchart of a method for defining analysis bins for calculating metrics according to an example of the present disclosure. [Figure 2B] 1 is a diagram illustrating analysis and metric values ​​according to an example of the present disclosure. [Figure 3A]1 is a flowchart of a method for defining analysis bins for calculating metrics according to an example of the present disclosure. [Figure 3B] 1 is a diagram illustrating analysis bins according to an example of the present disclosure. [Figure 4] FIG. 1 illustrates a computerized system suitable for implementing one or more method steps associated with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements beyond the technology found in the market, or to enable other skilled in the art to understand the embodiments disclosed herein.

[0008] The continuous increase in data has led to significant investments in artificial intelligence (AI) solutions to help extract insights from data, but choosing the right AI services to provide reliable and accurate system configurations can be challenging.

[0009] The present disclosure may enable a machine learning system (MLS) to evaluate, refine, or update an AI solution so that the computer system can operate more efficiently (e.g., using less memory, providing more accurate results, using fewer iterations of the machine learning model). For example, a computer system may leverage an MLS to prevent unnecessary redundant processing actions caused by an unadapted AI solution.

[0010] When using artificial intelligence solutions to monitor data transactions, it may be important to identify the technical benefits that can be obtained from investments in improving existing machine learning models. Therefore, correlations between metric values ​​of machine learning models and overall indicator values ​​may be advantageously used to provide meaningful recommendations. In particular, the computer system configuration of the MLS may not only rely on specific AI metrics, but also on their impact on the overall process. The MLS may be configured to measure the overall impact (e.g., using the overall indicator value). Furthermore, the MLS of the present disclosure may be advantageous in cases of relatively rare data (e.g., lack of available data) such that the metric value of the machine learning model cannot be determined (computation of the AI ​​metric requires a large amount of data). The MLS of the present disclosure may provide an approximation of the metric value of the machine learning model to solve the problem of insufficient data.

[0011] According to some embodiments, the MLS may perform an update if the correlation is negative or zero correlation, and performing the update may include any one of retraining the ML model, adding an additional ML model to provide a composite evaluation of the characteristic, or replacing the ML model with another ML model, and configuring the computer system includes using the performed update to evaluate further transactions.

[0012] For example, after adding additional ML models, an ML performance metric may be calculated for each of the ML models and the resulting values ​​may be combined. The combination may be, for example, a weighted sum or average of the values. The weighted sum may, for example, use weights associated with the ML models. The weights may, for example, be user-defined.

[0013] According to some embodiments of MLS, if there is no correlation between the overall performance metric and the ML performance metric in a set of analysis bins, then the ML model may be replaced with another ML model.

[0014] According to some embodiments of MLS, if the correlation between the overall performance metric and the ML performance metric in a set of analysis bins is a positive correlation, this indicates an improvement of the ML model, which may include retraining the ML model.

[0015] The ML model may, for example, have been trained or adapted to a given type of data (e.g., data for a given region, set, area, etc.). Retraining may be performed, for example, by increasing the size of a training set previously used to initially train the ML model, and the retraining is performed using the increased size of the training set. In another example, the retraining may be performed using a new training set including the latest data. This may update the ML model so that it can be used for accurate monitoring of data transaction processing.

[0016] In one example, the computer system may be configured to perform calculations of ML performance metrics as part of a given monitoring process of data transactions in the computer system. For example, the computer system may issue an alert or stop execution if the value of the ML performance metric is suspicious. This performed update may enable the computer system to improve monitoring of further data transactions, for example, by preventing false alarms triggered by non-adapted ML models. This may save computer system resources consumed by unnecessary false alarms.

[0017] According to some embodiments, the attribute is the occurrence time of the data transaction, each bin of the set of analysis bins represents a time interval, one of the redefined bins is obtained by merging two or more consecutive bins of the set of analysis bins, and the estimation is performed by modeling the variation of the ML performance metric as a function of the redefined bins and using the model to determine the value of the ML metric in the set of analysis bins. This can be seamlessly integrated with existing systems since most of the monitoring systems perform monitoring of data as a function of time. This can have the further advantage of identifying problems in advance and reacting in a timely manner. For example, a problem will last at most for the duration of the set of analysis bins since the computer system will be configured shortly after that duration.

[0018] According to some embodiments, the modeling includes fitting the distribution of the ML performance metric across the redefined bins. The fitting includes regression analysis, such as linear regression, to estimate a relationship that best fits the data points according to certain mathematical criteria. This may allow for systematic and accurate estimation of the value of the ML performance metric. Accurate estimation of the metric value may allow for reliable control / operation of the computer system. This may allow for the calculation of the metric value of the machine learning model on larger subsets covering longer time intervals, in a manner similar to that of a moving average calculation. The calculated metric may then be used to calculate more granular results, for example using cubic spline approximations.

[0019] According to some embodiments, each analysis bin of the set of analysis bins may represent a set of values ​​of different attributes of records of the data transactions, the set of values ​​being values ​​of clusters of records formed using the different attributes, and one of the redefined bins is obtained by merging two or more bins of the set of analysis bins whose associated clusters have a predetermined distance from each other.

[0020] For example, each bin B_i of the set of analysis bins may be associated with a respective attribute Att_i. Data records describing data transactions performed by a computer system during a given time period (e.g., transactions for the past month) may be partitioned based on the value of attribute Att_i, and one cluster may be created for each distinct attribute Att_i, such that each bin of the set of analysis bins is associated with a respective cluster of records. The clusters may be combined such that a resulting set of combined clusters is associated with a respective redefined bin, e.g., each redefined bin may be associated with a respective set of combined clusters. Each set of combined clusters may have sufficient data to allow computation of a metric value of a machine learning model for them. The combined clusters of each set may have a cluster center-to-center distance that is less than a defined distance. The combined clusters of each set may have a minimum distance between cluster centers. In another example, the set of combined clusters may be user-defined. This may enable a sophisticated approach to data slicing based on similarity of input records. This may enable flexible monitoring of transactions using different attributes.

[0021] According to some embodiments, the estimation of the ML performance metric may include: For each bin of the set of analysis bins, the ML performance metric of the cluster j associated with that bin is defined as follows: sum(wi*mi) / sum(wi), where mi is the ML metric of the set of combined clusters i, and wi is calculated as follows: meanDtoJ / maxD*nPinJ / nPinCls, where meanDtoJ is the average distance between the centers of the set of combined clusters i and the centers of cluster j, maxD is the maximum distance between the centers of the set of combined clusters i, nPinJ is the number of data points in cluster j, and nPinCls is the number of data points in the set of combined clusters i. This allows for an accurate calculation. The calculated metric may then be used to calculate more granular results using a weighted arithmetic mean.

[0022] According to some embodiments, the method may be performed at run-time on a computer system. This is advantageous for real-time monitoring of data. Monitoring of machine learning models in production environments may be based on data analysis of evaluation payloads performed in real-time to calculate metrics such as fairness scores, accuracy loss (drift metrics), etc. The machine learning model metric values ​​and overall process indicator values ​​may be compiled to enable correlation discovery based on time-based data splitting, clustering, or other data slicing methods.

[0023] According to some embodiments, the method may be repeated for a further set of analysis bins using a controlled computer system. For example, the set of analysis bins may be the current set of analysis bins covering a current period, e.g., this week. This may allow for further monitoring of transaction data for the next period following the current period. This may allow for uninterrupted monitoring of data transactions.

[0024] According to some embodiments, the MLS includes further iterations for a different ML performance metric, for example, some steps other than determining the set of analysis bins and calculating the overall performance metric may be repeated for another ML performance metric.

[0025] According to some embodiments, the analysis bins are of equal size.

[0026] According to some embodiments, the calculation of the overall or ML performance metric further comprises normalizing the calculated metric.

[0027] These embodiments may enable analysis that is scalable with the amount of data and the number of bins.

[0028] According to some embodiments, the MLS further includes collecting records of data transactions associated with each analysis bin of the set of analysis bins to perform calculations on the collected records.

[0029] According to some embodiments, the overall performance metric is a key performance indicator (KPI), which may include one or more metrics to provide context for the performance of the computer system.

[0030] According to some embodiments, the ML performance metric is one of the predictive accuracy of the ML model and a fairness score.

[0031] FIG. 1 is a flow chart of a method of controlling the operation of a computer system according to an example of the present disclosure. The computer system may be configured to, for example, execute or perform a data transaction. A data transaction may be a set of actions that together perform a task. A data transaction may perform a task, for example, debiting or crediting an account or requesting an inventory list. A data transaction may be described by one or more data records. A data record is a collection of related data items, such as the name, birth date, and class of a particular user who requested the data transaction. A record represents an entity, which may refer to a user, object, transaction, or concept, about which information is stored in the record. The terms "data record" and "record" are used interchangeably. Data records may be stored in a graph database as entities with relationships, where each record may be assigned to a node or vertex of the graph with properties that are attribute values, such as name, birth date, etc. A data record may be a record in a relational database, in another example.

[0032] A data transaction may be evaluated to determine its characteristics or properties. The evaluation may indicate, for example, whether the data transaction is anomalous, an insecure transaction, etc. The evaluation may be performed, for example, using a trained ML model. The ML model may be trained, for example, on historical telecommunications asset failure data, including, for example, sensor data, to predict asset failures before the asset failures cause outages. However, information technology operations need to ensure that the ML model is accurately predicting failures, but the data is highly complex. In another example, the ML model may be trained on historical successful and unsuccessful forecast override data. The trained ML model may help a demand planner adjust its demand forecasts. However, the trained ML model may need to be monitored for its accuracy, for example, over a period of time, so that the AI-powered application can check that it is always producing results that are as accurate as those produced by knowledge workers. In a further example, the ML model may be trained on historical transaction data to identify suspicious patterns. Trained models may need to be monitored to help banks keep up with ever-changing regulations, allowing economic crime analysts to understand the reasoning behind the model's alert analysis so they can make decisions about which alerts to dismiss and which to escalate.

[0033] In operation 101, a set of analysis bins (referred to as "InitSet" for purposes of clarity) may be determined. An analysis bin represents a set of values ​​of an analysis attribute of a record of a data transaction. The set of analysis bins may or may not be of equal width or size. The analysis attribute may be, for example, the occurrence time of the data transaction. In this case, the set of analysis bins may cover, for example, a time range, for example, a time range of one month, and each of the analysis bins may cover a respective time range, for example, a time range of the first week of the month. In another example, the analysis attribute may be the age of a user who requested the data transaction. In this case, the set of analysis bins may cover, for example, ages between 18 and 100 years, and each of the analysis bins may cover a respective time range, for example, a time range of 80 to 100 years. For ease of explanation, it is assumed that the set of analysis bins InitSet includes 10 bins B1 to B10.

[0034] A data transaction performed by a computer system may be associated with a respective analysis bin of a set of analysis bins. Continuing with the above example, all transactions triggered by a user having an age between 80 and 100 may be associated with analysis bin [80,100]. This means that each analysis bin X of the set of analysis bins may be associated with a data record, each of the data records having a value for an analysis attribute that falls within analysis bin X.

[0035] In one example, the set of analysis bins can be user-defined, e.g., in operation 101, a user input can be received, the user input indicating a set of analysis bins. In another example, a plurality of sets of analysis bins can be predefined (e.g., pre-stored), and determining the set of analysis bins in operation 101 can include selecting (e.g., randomly) one set of analysis bins from among the plurality of predefined sets of analysis bins. In one example, the set of analysis bins can be determined such that a number of transactions associated with each analysis bin of the set of analysis bins is greater than a predefined transaction count threshold. This transaction count threshold can be sufficient to perform an overall performance analysis, e.g., to evaluate an overall performance metric.

[0036] In operation 103, an overall performance metric of the computer system may be calculated for each bin of the set of analysis bins. The calculation may be performed using a record of transactions having attribute values ​​represented by the analysis bin. The overall performance metric may be, for example, an average transaction duration. Then, for each analysis bin X of the set of analysis bins, a transaction duration may be determined for each transaction of the transactions associated with analysis bin X. Then, an average of the determined transaction durations may be calculated and assigned to analysis bin X. In another example, the overall performance metric may be a number of failed transactions. Then, for each analysis bin X of the set of analysis bins, a number of failed transactions for that bin X may be determined.

[0037] A data record describing a transaction may be of one or more types. For example, a transaction may be associated with an overall record describing the overall properties / attributes of the transaction and another ML record describing the results of running an ML model on the transaction. The overall record may include the overall attributes. The ML record may include the ML attributes. The ML record may be a record in a payload logging table. The overall record and the ML record may be linked together by a transaction ID that belongs to both records. In another example, a single type of record may be used to describe a transaction, e.g., the single type of record may include attributes of both the overall record and the ML record. This single record may include empty values ​​for the ML attributes if no ML models are run on the transaction of that single record.

[0038] It may be determined whether the number of records for one or more analysis bins of the set of analysis bins is less than a predetermined minimum number of records (query operation 105). A number of records below the minimum number of records may be sufficient for calculating the overall performance metric in operation 103, but may not be sufficient for performing ML performance monitoring within a given bin. In the case of two different types of records, the query operation 105 may be performed on the ML records. For example, the query operation 105 may determine whether the number of ML records for one or more analysis bins of the set of analysis bins is less than a predetermined minimum number of records. In the case of a single type of record, the query operation 105 may determine whether the number of records having a non-empty ML attribute value is less than a predetermined minimum number of records. For example, the query operation 105 may be performed as follows: Each analysis bin X of the set of analysis bins InitSet may be processed to determine whether the number of records whose analysis attribute value falls within bin X is less than a predetermined minimum number of records.

[0039] If it is determined (query operation 105) that the number of records for one or more analysis bins of the set of analysis bins InitSet is less than a predetermined minimum number of records, operations 107-111 are performed, otherwise operation 113 is performed. For example, consider the case where bins B2 and B5 are determined to have a number of records less than a predetermined minimum number of records.

[0040] In operation 107, a new or different set of analysis bins (referred to as "NewSet" for the sake of clarity) may be determined or redefined. Continuing with the above example of InitSet, the redefined set of bins NewSet may include n bins rB1 to rBn, where n<10. Operation 107 may be performed, for example, by combining the analysis bins of the set of analysis bins InitSet. Continuing with the above example, bins B2 and B3 of InitSet may be combined to form a new bin rB2, and bins B4 and B5 of InitSet may be combined to form a new bin rB3, since only B2 and B5 have a number of records below a predetermined minimum number of records. This results in a redefined set NewSet of 8 bins rB1 to rB8, where rB1 is B1, rB4 is B6, rB5 is B7, rB6 is B8, rB7 is B9, and rB8 is B10, i.e. rB2 and rB3 are redefined. This may utilize the existing set of analysis bins, InitSet, to define the new set of bins, which may save resources since accumulated records for bins that have not been changed can be reused. In another example, the new set of bins, NewSet, may be defined independently from the set of analysis bins, InitSet, of operation 101, by determining a new width for the new set of bins, NewSet, such that the number of records in each bin of the new set of bins is greater than a predetermined minimum number of records.

[0041] For each bin rB of the redefined set of bins NewSet, in operation 109, an ML performance metric of the ML model may be calculated using records having attribute values ​​represented by each bin rB. The ML performance metric may be, for example, a prediction accuracy of the ML model. For example, each record of the ML records may include an ML attribute that describes an ML prediction accuracy used to evaluate the data transactions of the record. In operation 109, for each bin of the redefined set of bins NewSet, the accuracies of the bin's ML records may be averaged to provide a value of the bin's ML performance metric. In another example, the ML performance metric may be a fairness score.

[0042] Continuing with the above example, operation 109 may result in eight values ​​of the ML performance metric, each value associated with a respective bin of the redefined set of bins, NewSet. However, the overall performance metric has been evaluated ten times for the analysis bins of InitSet. This may result in a suboptimal correlation analysis between the two metrics. To address that, in operation 111, the ML performance metric may be estimated in each bin of the set of analysis bins, InitSet, using the ML performance metric of the redefined set of bins, NewSet. For example, knowing eight values ​​of the ML performance metric in the bins of NewSet, ten values ​​of the ML performance metric may be derived for the bins of InitSet. Continuing with the above example, the ML performance metric for bins rB1, rB4, rB5, rB6, rB7, and rB8 of NewSet may be the same as for bins B1, B6, B7, B8, B9, and B10, respectively. The ML performance metric may be estimated for bins B2-B5 of the InitSet by combining (or extrapolating) the metric values ​​of the surrounding bins, e.g., B1, rB1, rB2, and B6. Another example of performing the estimation is shown in Figures 2-3.

[0043] If each bin B of the set of analysis bins InitSet has a number of records greater than a predetermined minimum number of records, operation 113 may be performed as follows: For each bin B of the set of bins InitSet, an ML performance metric for the ML model may be calculated in operation 113 using records having attribute values ​​represented by each bin B.

[0044] After performing operation 111 or operation 113, each bin of the set of bins InitSet has a pair of values ​​of the ML performance metric and the overall performance metric. This allows the values ​​of the two metrics to be compared bin by bin. In particular, the behavior of the variation of the ML performance metric in the set of bins InitSet can be compared with the behavior of the overall performance metric in the set of bins InitSet. This can allow for an accurate correlation analysis of the two metrics, and therefore the correlation can be used reliably by the method. For example, based on the correlation across the set of analysis bins InitSet between the calculated overall performance metric and the ML performance metric, the computer system can be configured in operation 115 to allow further transactions to have a positive correlation between the overall performance metric and the ML performance metric. The configuration can be based on the correlation between the overall performance metric and the ML performance metric. For example, if the correlation is negative, this can indicate that the trained ML model is not suitable for the use case being used. For example, the trained ML model will perform well on data from a given domain, such as telecommunications. But for other domains, it may not provide the required accuracy. In another example, information indicating a correlation between the overall performance metric and the ML performance metric after performing act 111 or act 113 may be provided, for example, to a user. The information may be used by the user, for example, as monitoring information for the computer system.

[0045] The correlation between the two metrics may have the following characteristics: In one example, the correlation between the two metrics may be a strong positive correlation. For example, a degradation in the ML performance metric degrades a particular KPI, e.g., a 2% degradation in the model fairness score degrades the credit amount approval KPI by 5%. This indicates that resource investment in certain areas of model quality may be important. In response, the system may be configured to further refine the ML model to avoid degradation of the model fairness score.

[0046] In one example, a particular ML performance metric improves without impacting KPIs. This indicates, for example, that a 5% increase in model accuracy has no impact on clicks. Such insight clearly indicates that the investment in model accuracy may not be worthwhile and a new ML model may be used instead of that ML model.

[0047] In one example, there is no correlation (or very little correlation) between any ML performance metric and the KPIs. This may indicate a serious problem with the ML model, and the results of that model are completely ignored in the process. This may raise an alarm to rethink the decision-making process and the configuration of the computer system.

[0048] Thus, depending on the correlation between the ML performance metric and the overall performance metric, the computer system may be configured accordingly. The configuration may be performed to enable a positive correlation between the two metrics for the next transaction executed. For example, the ML performance metric calculated for future transactions may have an improved value that is consistent with the overall performance metric. The present disclosure may consider the combined effect of multiple monitoring metrics.

[0049] For example, operation 115 may automatically trigger retraining of the ML model. Retraining may be performed using new data corresponding to the current use case of the computer system. In another example, retraining may be performed by augmenting a previously used training set to improve the accuracy of the trained model with certain inputs from payload analysis to meet the goals and adapt to new data. In another example, the ML model may be adapted without the need to perform retraining, for example, by updating minimum sample size and thresholds to generate more data on the currently trained model without incurring additional processing costs. This may avoid intensive CPU usage when the underlying data is not changing.

[0050] FIG. 2A is a flowchart of a method for defining analysis bins for calculating metrics according to an example of the present disclosure.

[0051] At operation 201, a set of analysis bins B1-B10 may be provided. The set of analysis bins may cover a time range [tS, tE]=

[0010] , as shown in FIG. 2B. The set of analysis bins includes 10 bins B1-B10 of width 1, as shown in FIG. 2B. The data in each bin B1-B10 may not be sufficient to calculate an ML performance metric.

[0052] In operation 203, a new set of bins rB1 to rB5 is (re)defined by combining two consecutive bins of the set of 10 bins B1 to B10. This may be performed, for example, because data in 10 bins may not be sufficient to calculate an ML performance metric. This results in 5 new bins rB1 to rB5 of width 2. For example, a new bin rB1 can be obtained by combining bins B1 and B2, a new bin rB2 can be obtained by combining bins B3 and B4, a new bin rB3 can be obtained by combining bins B5 and B6, a new bin rB4 can be obtained by combining bins B7 and B8, and a new bin rB5 can be obtained by combining bins B9 and B10.

[0053] 2B further shows, for each bin of the new set of analysis bins rB1-rB5, a data point 220 representing the value of the ML performance metric in each of the five bins rB1-rB5. To estimate the value of the ML performance metric in each of the ten bins B1-B10, a fitting 222 (or modeling) of the distribution of values ​​of the ML performance metric may be performed. The fitting 222 may be a cubic spline approximation, which may be used to estimate or approximate a value 224 of the ML performance metric in each of the ten bins.

[0054] FIG. 3A is a flowchart of a method for defining analysis bins for calculating metrics according to an example of the present disclosure.

[0055] In operation 301, a set of analysis bins may be provided. The set of analysis bins may cover five clusters, as shown in FIG. 3B. The set of analysis bins includes five bins each associated with a respective cluster 320.1-5. Each cluster of clusters 320.1-5 may include records having similar values ​​of a respective attribute, for example, cluster 320.1 may include records of users having ages between 20 and 40 years old, cluster 320.2 may include records of users from a given region or country, etc.

[0056] In operation 303, a new set of bins is defined by combining two or more bins of the set of five bins. This may be performed, for example, because the data in the five bins may not be sufficient to calculate an ML performance metric. This results in three new bins 322.1-322.3, each associated with a respective set of combined clusters (e.g., the set of combined clusters may be referred to as combined clusters). For example, new bin 322.1 represents the set of combined clusters 320.1, 320.2, and 320.3, new bin 322.2 represents the set of combined clusters 320.4, 320.2, and 320.3, and new bin 322.3 represents the set of combined clusters 320.5, 320.4, and 320.3. The clusters may be combined, for example, based on the distance between them. The distance may be calculated using one or more attributes of the records of the clusters. An ML performance metric may be calculated for the three bins 322.1-322.3. Then, the following formula may be used to obtain the ML performance metric for each bin 320.1-320.5:

[0057] For each bin j of the set of analysis bins 320.1-320.5, the ML performance metric may be estimated as follows: sum(wi*mi) / sum(wi), where i is the index of the set of combined clusters, varying from 1 to 3 in this example, mi is the ML performance metric for the set of combined clusters i, and wi is calculated as follows: meanDtoJ / maxD*nPinJ / nPinCls, where meanDtoJ is the average distance between the centers of the set of combined clusters i and the centers of cluster j, maxD is the maximum distance between the centers of the set of combined clusters i, nPinJ is the number of data points in cluster j, and nPinCls is the number of data points in the set of combined clusters i. In some embodiments, meanDtoJ may represent the following: the average distance between the original centers in the combined clusters and the center of cluster j. In some embodiments, maxD may represent the following: the maximum distance between the original centers. In some embodiments, nPinJ may represent the following: the number of data points in cluster j. In some embodiments, nPinCls may represent the following: the maximum distance between the original centers.

[0058] FIG. 4 represents a general computerized system 400 suitable for implementing at least some of the method steps associated with the present disclosure. As will be appreciated, the methods described herein are at least partially non-interactive and automated by a computerized system such as a server or embedded system. However, in some embodiments, the methods described herein may be implemented in a (partially) interactive system. These methods may be further implemented in software 412, 422 (including firmware 422), hardware (processor) 405, or a combination thereof. In some embodiments, the methods described herein are implemented in software as executable programs and executed by a dedicated or general-purpose digital computer, such as a personal computer, a workstation, a minicomputer, or a mainframe computer. Thus, the most general system 400 includes a general-purpose computer 401.

[0059] In some embodiments, with respect to the hardware architecture, as shown in FIG. 4, a computer 401 includes a processor 405, a memory (main memory) 410 coupled to a memory controller 415, and one or more input / output (I / O) devices (or peripherals) 10, 445 communicatively coupled via a local I / O controller 435. The I / O controller 435 may be, but is not limited to, one or more buses or other wired or wireless connections as known in the art. The I / O controller 435 may have additional elements such as controllers, buffers (caches), drivers, repeaters, and receivers to enable communication, which are omitted for simplicity. Additionally, the local interface may include address, control, and / or data connections to enable appropriate communication between the aforementioned components. As described herein, the I / O devices 10, 445 may generally include any generalized cryptographic or smart card known in the art.

[0060] Processor 405 is a hardware device for executing software, particularly software stored in memory 410. Processor 405 may be any custom or commercially available processor, a central processing unit (CPU), a coprocessor among several processors associated with computer 401, a semiconductor-based microprocessor (in the form of a microchip or chip set), a microprocessor, or generally any device for executing software instructions.

[0061] The memory 410 may include any one or combination of volatile memory elements (e.g., RAM such as DRAM, SRAM, SDRAM, etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), programmable read only memory (PROM)). It is noted that the memory 410 may have a distributed architecture, where various components are located remotely from each other but may be accessed by the processor 405.

[0062] The software in memory 410 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions, particularly functions included in the embodiments of the present disclosure. In the example of Figure 4, the software in memory 410 includes instructions 412, e.g., instructions for managing a database, such as a database management system.

[0063] The software in memory 410 will also typically include a suitable operating system (OS) 411, which essentially controls the execution of other computer programs, such as software 412, in some cases, for implementing the methods described herein.

[0064] The methods described herein may be in the form of a source program 412, an executable program 412 (object code), a script, or any other entity that includes a set of instructions 412 to be executed. In the case of a source program, the program must be translated by a compiler, assembler, interpreter, etc., to operate properly in conjunction with the OS 411, which may or may not be contained within the memory 410. Additionally, the methods may be written as an object-oriented programming language having classes of data and methods, or a procedural programming language having routines, subroutines, and / or functions.

[0065] In some embodiments, a keyboard 450 and a mouse 455 may be coupled to the input / output controller 435. Other output devices, such as I / O device 445, may include, but are not limited to, input devices, such as, for example, a printer, a scanner, a microphone, and the like. Finally, I / O device 10, 445 may further include, but are not limited to, devices that communicate both input and output, such as, for example, a network interface card (NIC) or a modem (for accessing other files, devices, systems, or networks), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, and the like. I / O device 10, 445 may be any generalized cryptographic card or smart card known in the art. System 400 may further include a display controller 425 coupled to the display 430. In some embodiments, system 400 may further include a network interface for coupling to a network 465. Network 465 may be an IP-based network for communication between computer 401 and any external servers, clients, and the like over a broadband connection. The network 465 transmits and receives data between the computer 401 and the external system 30, which may be involved in performing some or all of the steps of the methods discussed herein. In some embodiments, the network 465 may be a managed IP network managed by a service provider. The network 465 may be implemented in a wireless manner using wireless protocols and technologies such as, for example, WiFi, WiMax, etc. The network 465 may also be a packet-switched network, such as a local area network, a wide area network, a metropolitan area network, an Internet network, or other similar types of network environments.Network 465 can be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), an intranet or other suitable network system and includes equipment for receiving and transmitting signals.

[0066] If computer 401 is a PC, workstation, intelligent device, etc., the software in memory 410 may further include a basic input / output system (BIOS) 422. The BIOS is a set of basic software routines that initializes and tests hardware at start-up, starts the OS 411, and supports the transfer of data between hardware devices. The BIOS is stored in ROM so that the BIOS can be executed when computer 401 is booted.

[0067] When computer 401 is operating, processor 405 is configured to execute software 412 stored in memory 410, to communicate data to and from memory 410, and to generally control the operation of computer 401 in accordance with the software. The methods and OS 411 described herein are read by processor 405, possibly buffered within processor 405, and then executed, in whole or in part, but typically the latter.

[0068] 4, the methods may be stored on any computer-readable medium, such as storage 420, for use by or in connection with any computer-related system or method. Storage 420 may include disk storage, such as HDD storage.

[0069] The present invention may be a system, method, or computer program product, or a combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium(s) having computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0070] A computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes the following: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical coding devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. A computer-readable storage medium as used herein should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted through a conductor.

[0071] The computer readable program instructions described herein may be downloaded from the computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission cables, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in the computer readable storage medium in the respective computing / processing device.

[0072] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and the like, and procedural programming languages ​​such as the “C” programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0073] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0074] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus to generate a machine such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. These computer readable program instructions may also be stored in a computer readable storage medium capable of instructing a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner, such that the computer readable storage medium on which the instructions are stored comprises an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0075] The computer readable program instructions may also be loaded into a computer, programmable data processing apparatus, or other device to generate a computer-implemented process by causing the computer, other programmable apparatus, or other device to perform a series of operational steps such that the instructions, which execute on the computer, other programmable apparatus, or other device, implement the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0076] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts and block diagrams may represent a module, segment, or portion of instructions that includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be performed as one step, may be performed concurrently, substantially concurrently, in a partially or fully overlapping manner, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a dedicated hardware-based system that performs the specified functions or executes or executes a combination of dedicated hardware and computer instructions.

[0077] This subject matter may include the following provisions:

[0078] 1. A method of controlling operation of a computer system, the computer system being configured to execute data transactions and evaluate properties of the data transactions using a machine learning (ML) model, the method comprising: determining a set of analysis bins, the analysis bins representing sets of values ​​for an attribute of a record of a data transaction; calculating, for each bin of the set of analysis bins, an overall performance metric for the computer system, the overall performance metric being indicative of a performance of execution of transactions by the computer system using records of transactions having attribute values ​​represented by the bin; If one or more bins of the set of analysis bins does not have at least a predetermined minimum number of records, redefining a new set of analysis bins by combining analysis bins of the set of analysis bins; For each bin of the redefined set of bins, calculating a machine learning performance metric for the ML model using records having attribute values ​​represented by each bin; using the ML performance metric for the redefined set of bins to estimate an ML performance metric within each bin of the set of analysis bins; calculating an ML performance metric within each bin of the set of analysis bins, where each bin has at least a minimum number of records; and configuring the computer system to enable a positive correlation between the overall performance metric and the ML performance metric of the further executed data transaction based on a correlation across the set of analysis bins between the calculated overall performance metric and the ML performance metric. A method comprising:

[0079] 2. The method of claim 1, further comprising performing an update including any of the following if the correlation is negative or zero correlation: retraining the ML model, adding an additional ML model to enable combination of evaluations of the properties, or replacing the ML model with another ML model, and configuring the computer system includes using the performed update to evaluate further transactions.

[0080] 3. The method of claim 1, wherein if the correlation between the overall performance metric and the ML performance metric in the set of analysis bins is positive, then improving the ML model by retraining the ML model using a larger training dataset.

[0081] 4. The method of any of clauses 1 to 3, wherein the attribute is a time of occurrence of a data transaction, each bin of the set of analysis bins represents a time interval, and one of the redefined bins is obtained by merging two or more of the temporally consecutive bins of the set of analysis bins, and estimating is performed by modeling the variation of the ML performance metric as a function of the redefined bins and using the ML model to determine values ​​of the ML performance metric in the set of analysis bins.

[0082] 5. The method of clause 4, wherein the modeling includes fitting a distribution of the ML performance metric across the redefined bins.

[0083] 6. The method according to any of clauses 1 to 3, wherein each analysis bin of the set of analysis bins represents a set of values ​​of different attributes of records of data transactions, the sets of values ​​being values ​​of clusters of records formed using the different attributes, and one of the redefined bins is obtained by merging two or more bins of the set of analysis bins whose associated clusters have a predetermined distance from each other.

[0084] 7. The method of claim 6, wherein estimating the ML performance metric includes: for each bin in the set of analysis bins, an ML performance metric for cluster j associated with that bin is defined as: sum(wi*mi) / sum(wi), where mi is the ML metric for the set of combined clusters i, and wi is calculated as: meanDtoJ / maxD*nPinJ / nPinCls, where meanDtoJ is the average distance between the centers of the set of combined clusters i and the center of cluster j, maxD is the maximum distance between the centers of the set of combined clusters i, nPinJ is the number of data points in cluster j, and nPinCls is the number of data points in the set of combined clusters i.

[0085] 8. A method according to any one of clauses 1 to 7, when executed on a computer system.

[0086] 9. The method according to any of clauses 1 to 8, wherein the method is repeated for a further set of analysis bins using a controlled computer system.

[0087] 10. The method of any of clauses 1 to 9, further comprising repeating the method for different ML performance metrics.

[0088] 11. The method of any of clauses 1 to 10, wherein the analysis bins are of equal size.

[0089] 12. The method of any of clauses 1 to 11, wherein the calculation of the overall performance metric or the ML performance metric further comprises normalizing the calculated metric.

[0090] 13. The method of any of clauses 1 to 12, further comprising collecting records of data transactions associated with each analysis bin of the set of analysis bins to perform calculations on the collected records.

[0091] 14. The method of any of clauses 1 to 13, wherein the overall performance metric is a key performance indicator KPI.

[0092] 15. The method of any of clauses 1 to 14, wherein the ML performance metric is one of a predictive accuracy of the ML model and a fairness score.

[0093] The description of various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to explain the principles of the embodiments, practical applications or technical improvements beyond the technology found in the market, or to enable other skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method implemented in a computer system configured to perform data transactions and evaluate characteristics of the data transactions using a machine learning (ML) model, the method comprising: determining a first set of analysis bins based on a predetermined attribute of the attributes of the records of the data transaction, the first set of analysis bins being a set of the records of the data transaction having values ​​of the predetermined attribute; calculating, for each analysis bin of the first set of analysis bins, an overall performance metric of the computer system based on the records of the data transactions associated with each of the analysis bins, the overall performance metric being indicative of a performance of transaction execution by the computer system; in response to one or more of the analysis bins of the first set of analysis bins not having at least a predetermined minimum number of records; redefining the first set of analysis bins to determine a second set of analysis bins, the second set of analysis bins including combining the analysis bins that do not have at least the predetermined minimum number of records with other analysis bins; calculating, for each analysis bin of the second set of analysis bins, a machine learning (ML) performance metric of the ML model based on the records associated with each analysis bin; estimating the ML performance metric for each of the analysis bins of the first set of analysis bins using the ML performance metric for each of the analysis bins of the second set of analysis bins; calculating the ML performance metric for each of the analysis bins of the first set of analysis bins in response to each of the analysis bins of the first set of analysis bins having at least the minimum number of records; updating the ML model based on a correlation between the overall performance metric and the ML performance metric across the first set of analysis bins to enable a positive correlation between the overall performance metric and the ML performance metric of further executed data transactions; A method comprising:

2. 2. The method of claim 1 , wherein updating the ML model comprises adding an additional ML model to enable combined evaluation of the characteristics in response to the correlation between the overall performance metric and the ML performance metric across the first set of analysis bins being a negative or zero correlation.

3. 2. The method of claim 1 , wherein updating the ML model comprises, in response to the correlation between the overall performance metric and the ML performance metric across the first set of analysis bins being a positive correlation, improving the ML model by retraining the ML model using a larger training data set.

4. the attribute being a time of occurrence of the data transaction; each of the analysis bins of the first set of analysis bins represents a time interval, and the analysis bins of the second set of analysis bins are obtained by combining two or more of the analysis bins of the first set of analysis bins that are consecutive in time; the estimating is performed by modeling the variation of the ML performance metric as a function of the second set of analysis bins; The method of claim 1 , wherein the ML performance metric is calculated using the ML model.

5. The method of claim 4 , wherein the modeling comprises fitting a distribution of the ML performance metric across the second set of analysis bins.

6. 1. A method implemented in a computer system configured to perform data transactions and evaluate characteristics of the data transactions using a machine learning (ML) model, the method comprising: determining a first set of analysis bins based on a plurality of attributes of the records of the data transaction, the first set of analysis bins comprising a plurality of clusters based on the plurality of attributes, each of the clusters comprising a set of the records of the data transaction having values ​​of the attributes; calculating, for each analysis bin of the first set of analysis bins, an overall performance metric of the computer system based on the records of the data transactions associated with each of the analysis bins, the overall performance metric being indicative of a performance of transaction execution by the computer system; redefining the first set of analysis bins to determine a third set of analysis bins, the third set of analysis bins including combining two or more of the analysis bins having a predetermined distance from each other between the associated clusters; calculating, for each analysis bin of the third set of analysis bins, a machine learning (ML) performance metric for the ML model based on the records associated with each analysis bin; estimating the ML performance metric for each of the analysis bins of the first set of analysis bins using the ML performance metric for each of the analysis bins of the third set of analysis bins; updating the ML model based on a correlation between the overall performance metric and the ML performance metric across the first set of analysis bins to enable a positive correlation between the overall performance metric and the ML performance metric of further executed data transactions; A method comprising:

7. 7. The method of claim 1, wherein the method is repeated for a set of further analysis bins based on records of the data transactions executed on the computer system using the updated ML model.

8. The method of claim 1 , further comprising repeating the method for different said ML performance metrics.

9. The method of claim 1 , wherein the ML performance metric is selected from the group consisting of: a predictive accuracy of the ML model, and a fairness score.

10. 1. A computer system, comprising: Memory, Processor and Equipped with configured to perform a data transaction and evaluate a characteristic of the data transaction using a machine learning (ML) model; The processor is communicatively coupled to the memory, the processor comprising: determining a first set of analysis bins based on a predetermined attribute of the attributes of the records of the data transaction, the first set of analysis bins being a set of the records having values ​​of the predetermined attribute of the attributes of the records of the data transaction; calculating, for each analysis bin of the first set of analysis bins, an overall performance metric of the computer system based on the records of the data transactions associated with each of the analysis bins, the overall performance metric being indicative of a performance of transaction execution by the computer system; if one or more of the analysis bins of the first set of analysis bins does not have at least a predetermined minimum number of records; redefining the first set of analysis bins to determine a second set of analysis bins, the second set of analysis bins including combining the analysis bins that do not have at least the predetermined minimum number of records with other analysis bins; For each analysis bin of the second set of analysis bins, calculating a machine learning (ML) performance metric of an ML model based on the records of the data transactions associated with each analysis bin of the second set of analysis bins; estimating the ML performance metric for each of the analysis bins of the first set of analysis bins using the ML performance metric for each of the analysis bins of the second set of analysis bins; calculating the ML performance metric for each of the analysis bins of the first set of analysis bins if each of the analysis bins of the first set of analysis bins has at least the minimum number of records; updating the ML model based on a correlation between the overall performance metric and the ML performance metric across the first set of analysis bins to enable a positive correlation between the overall performance metric and the ML performance metric of further executed data transactions; Run the system.

11. 1. A computer system, comprising: Memory, Processor and Equipped with configured to perform a data transaction and evaluate a characteristic of the data transaction using a machine learning (ML) model; The processor is communicatively coupled to the memory, the processor comprising: determining a first set of analysis bins based on a plurality of attributes of the records of the data transaction, the first set of analysis bins comprising a plurality of clusters based on the plurality of attributes, each of the clusters comprising a set of the records of the data transaction having values ​​of the attributes; calculating, for each analysis bin of the first set of analysis bins, an overall performance metric of the computer system based on the records of the data transactions associated with the analysis bin, the overall performance metric being indicative of a performance of transaction execution by the computer system; redefining the first set of analysis bins to determine a third set of analysis bins, the third set of analysis bins including combining two or more of the analysis bins having a predetermined distance from each other between the associated clusters; for each analysis bin of the third set of analysis bins, calculating a machine learning (ML) performance metric of the ML model based on the records associated with the analysis bin; estimating the ML performance metric for each of the analysis bins of the first set of analysis bins using the ML performance metric for each of the analysis bins of the third set of analysis bins; updating the ML model based on a correlation between the overall performance metric and the ML performance metric across the first set of analysis bins to enable a positive correlation between the overall performance metric and the ML performance metric of further executed data transactions; Run the system.

12. A computer program product causing a computer to carry out a method according to any one of claims 1 to 9.

Images