Data monitoring based on machine learning

Abstract

An overall performance metric of the computer system can be determined for each bin in a set of analysis bins. In a case where one or more bins in the set of analysis bins do not have at least a predefined minimum number of records, a new set of analysis bins can be redefined by merging the analysis bins in the set of analysis bins. For each bin in the redefined set of bins, a machine learning (ML) performance metric of an ML model can be calculated. The ML performance metric of the redefined bin can be used to estimate an ML performance metric for the set of analysis bins. The computer system can be configured based on a correlation over the set of analysis bins between the calculated overall performance metric and the ML performance metric.

Description

Background Art

[0001] The present disclosure relates to the field of digital computer systems, and more particularly, to a method for controlling the operation of a computer system.

[0002] Machine learning models are increasingly used for data monitoring. However, machine learning models can be inaccurate for several reasons—for example, bias in the training data due to one or more of the following: biased labels, undersampling / oversampling, or generating models with undesirable biases. Monitoring machine learning may not always identify these biases. Summary of the Invention

[0003] Various embodiments provide a method, system, and computer program product for controlling the operation of a computer system as described herein. In one aspect, the present invention relates to controlling the operation of a computer system configured to perform data transactions and evaluate attributes of the data transactions using a machine learning (ML) model.

[0004] Determine a set of analysis bins. The analysis bins represent a set of attribute values of data transaction records. Calculate an overall performance index of the computer system. The overall performance index indicates transaction execution attributes of the computer system for each analysis bin in the set of analysis bins using transaction records having attribute values represented by the analysis bins. In the case where one or more analysis bins in the set of analysis bins do not have at least a predefined minimum number of records, redefine a new set of analysis bins by connecting the analysis bins in the set of analysis bins. For each analysis bin in the redefined set of analysis bins, calculate a machine learning performance index of an ML model using records having attribute values represented by each analysis bin. Use the ML performance index of the redefined analysis bins to estimate the ML performance index in each bin in the set of analysis bins. In the case where each analysis bin in the set of analysis bins has at least a minimum number of records, calculate the ML performance index in each analysis bin in the set of analysis bins. The computer system is configured to achieve a positive correlation between the overall performance index and the ML performance index for further executed data transactions based on a correlation between the calculated overall performance index and the ML performance index over the set of analysis bins. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Embodiments of the present disclosure are explained in more detail below, by way of example only, with reference to the accompanying drawings, in which:

[0006] Figure 1 is a flowchart of a method for controlling the operation of a computer system according to an example of the present disclosure.

[0007] Figure 2A is a flowchart of a method for defining analytical bins for calculating metrics according to an example of the present disclosure.

[0008] Figure 2B is a diagram illustrating analysis and metrics according to an example of the present disclosure.

[0009] Figure 3A is a flow chart of a method for defining analytical bins for calculating metrics according to examples of the present disclosure.

[0010] Figure 3B is a diagram illustrating an analysis pod according to an example of the present disclosure.

[0011] Figure 4 It represents a computerized system suitable for implementing one or more method steps involved in the present disclosure. DETAILED DESCRIPTION

[0012] The description of various embodiments of the present disclosure will be presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0013] The continued growth of data is driving significant investment in artificial intelligence (AI) solutions to help extract insights from that data. However, selecting the right AI services to provide trusted and accurate system configurations can be challenging.

[0014] The present disclosure may enable a machine learning system (MLS) to evaluate, improve, or update AI solutions, enabling computer systems to operate more efficiently (e.g., using less memory, providing more accurate results, using fewer machine learning model iterations). For example, a computer system may utilize MLS to prevent unnecessary additional operations caused by an unsuitable AI solution.

[0015] When using artificial intelligence solutions to monitor data transactions, it may be important to identify the technical benefits that can be obtained from investments in improving existing machine learning models. To this end, the correlation between machine learning model metric values and overall indicator values can be advantageously used to provide meaningful recommendations. Specifically, the computer system configuration of the MLS may rely not only on a specific AI metric, but also on its impact on the entire process. The MLS can be configured to measure the overall impact (e.g., using overall indicator values). In addition, the MLS of the present disclosure can be advantageous in the case of relatively rare data (e.g., lack of available data), and the machine learning model metric values cannot be determined using the relatively rare data (the calculation of the AI metric may require a large amount of data). The MLS of the present disclosure can provide an approximation of the machine learning model metric values to solve the problem of insufficient data.

[0016] According to some embodiments, where the correlation is negative or zero, the MLS may perform an update that may include any of: retraining the ML model, adding an additional ML model for providing a combined evaluation of the attributes, or replacing the ML model with another ML model, wherein the configuration of the computer system includes using the performed update to evaluate the further transaction.

[0017] For example, after adding additional ML models, an ML performance metric can be calculated for each ML model, and the resulting values can be combined. The combination can be, for example, a weighted sum or average of the values. The weighted sum can, for example, use weights associated with the ML models. The weights can, for example, be user-defined.

[0018] According to some embodiments of the MLS, in the event that there is no correlation between the overall performance metric and the ML performance metrics in the set of analysis bins, the ML model may be replaced with another ML model.

[0019] According to some embodiments of the MLS, the correlation between the overall performance metric in the analysis set and the ML performance metric is a positive correlation, which may indicate improving the ML model and may include retraining the ML model.

[0020] The ML model may, for example, have already been trained or adapted for a given type of data (e.g., data for a given region, collection, domain, etc.). For example, retraining may be performed by increasing the size of a training set previously used to initially train the ML model, wherein the retraining is performed using the increased size training set. In another example, retraining may be performed using a new training set that includes the latest data. This may update the ML model so that it can be used for accurate monitoring of data transaction processing.

[0021] In one example, a computer system can be configured to perform calculations of ML performance indicators as part of a given monitoring process for data transactions at the computer system. For example, if the value of the ML performance indicator is suspicious, the computer system can provide an alert or halt operations. Updates to this performance can enable the computer system to improve monitoring of other data transactions by, for example, preventing false warning alerts that could be triggered by an unadapted ML model. This can save computer system resources that might otherwise be consumed by unnecessary warning alerts.

[0022] According to some embodiments, the attribute is the time at which a data transaction occurred, wherein each bin in the set of analysis bins represents a time interval, wherein a redefined bin of a redefined bin is obtained by merging two or more consecutive bins in the set of analysis bins, wherein the estimation is performed by modeling the change in the ML performance indicator based on the redefined bins, and using a model for determining the value of the ML metric in the set of analysis bins. This can be seamlessly integrated with existing systems because most monitoring systems can perform monitoring of data based on time. This can further have the advantage of proactively identifying problems and reacting in a timely manner. For example, a problem may last at most for the duration of the set of analysis bins because the computer system can be configured immediately after that duration.

[0023] According to some embodiments, modeling includes fitting a distribution of an ML performance metric over a redefined interval. Fitting involves regression analysis, such as linear regression, which estimates the relationship that most closely fits the data points according to a specific mathematical criterion. This can enable systematic and accurate estimation of the ML performance metric value. Accurate estimation of the metric value can enable reliable control / operation of the computer system. This can enable calculation of machine learning model metric values over larger subsets covering longer time intervals, in a manner similar to that used in rolling mean calculations. The calculated metric can then be used to calculate finer-grained results using, for example, a cubic spline approximation.

[0024] According to some embodiments, each analysis bin in the analysis bin set may represent a set of values of different attributes of records of data transactions, wherein the set of values is the values of a cluster of records formed using different attributes, wherein the redefined bin in the redefined bin is obtained by merging two or more bins whose associated clusters in the analysis bin set have a predefined distance to each other.

[0025] For example, each bin B_i in the set of analysis bins can be associated with a corresponding attribute Att_i. Data records describing data transactions performed by a computer system during a predefined time period (e.g., transactions from the last month) can be split based on the value of the attribute Att_i, and a cluster can be created for each different attribute Att_i, so that each bin in the set of analysis bins is associated with a corresponding cluster of records. Those clusters can be joined so that the resulting set of merged clusters can be associated with the corresponding redefined bins, for example, each redefined bin can be associated with the corresponding set of merged clusters. Each set of merged clusters can have enough data to enable machine learning model metrics to be calculated on them. The merged clusters of each group can have a distance between the centers of the clusters that is less than a defined distance. The merged clusters of each group can have a minimum distance between the centers of the clusters. In another example, the set of merged clusters can be user-defined. This can enable complex methods of data slicing based on similarity of the input records. This can enable flexible monitoring of transactions using different attributes.

[0026] According to some embodiments, the estimation of the ML performance metric may include: for each bin in the set of analysis bins, the ML performance metric of cluster j associated with the bin is defined as follows: sum(wi*mi) / sum(wi), where mi is the ML metric of the merged set of clusters i, and wi is calculated as: meanDtoJ / maxD*nPinJ / nPinCls, where meanDtoJ is the average distance between the center of the merged set of clusters i and the center of cluster j, maxD is the maximum distance between the centers of the merged set of clusters i, nPinJ is the number of data points in cluster j, and nPinCls is the number of data points in the merged set of clusters i. This allows for precise calculations. A more granular result can then be calculated by using the metric calculated using a weighted arithmetic mean.

[0027] According to some embodiments, the method can be performed at runtime on a computer system. This can be advantageous for real-time monitoring of data. Monitoring of machine learning models in a production environment can be based on scoring of payload data analysis performed at runtime to calculate metrics such as fairness scores, accuracy degradation (drift metrics), etc. Machine learning model metrics and overall process metric values can be set together to allow for time-based data segmentation, clustering, or other data slicing methods to discover correlations.

[0028] According to some embodiments, the method can be repeated for further sets of analysis bins using the controlled computer system. For example, the set of analysis bins can be the current set of analysis bins covering the current time period (e.g., the current week). This can enable further monitoring of transaction data for the next time period following the current time period. This allows for uninterrupted monitoring of data transactions.

[0029] According to some embodiments, the MLS includes further repetitions for different ML performance indicators. For example, in addition to determining the set of analysis bins and calculating the overall performance indicator, certain steps can be repeated for another ML performance indicator.

[0030] According to some embodiments, the analysis bins are bins of equal size.

[0031] According to some embodiments, the calculation of the overall performance indicator or the ML performance indicator further comprises normalizing the calculated metrics.

[0032] These embodiments may enable analysis that scales with the amount of data and the number of bins.

[0033] According to some embodiments, the MLS further includes collecting records of data transactions associated with each analysis silo in the set of analysis silos for performing calculations on the collected records.

[0034] According to some embodiments, the overall performance indicator is a key performance indicator (KPI).A key performance indicator may include a plurality of one or more metrics used to provide context for the performance of a computer system.

[0035] According to some embodiments, the ML performance metric is one of: accuracy and fairness score of the ML model's predictions.

[0036] Figure 1 is a flow chart of a method for controlling the operation of a computer system according to an example of the present disclosure. The computer system may, for example, be configured to perform or run a data transaction. A data transaction may be a set of operations that together perform a task. A data transaction may, for example, perform a task, such as entering an account debit or credit, or requesting an inventory list. A data transaction may be described by one or more data records. A data record is a collection of related data items: such as the name, date of birth, and category of a particular user requesting the data transaction. A record represents an entity, where an entity refers to a user, object, transaction, or concept about which information is stored in a record. The terms "data record" and "record" are used interchangeably. These data records may be stored as entities with relationships in a graph database, where each record may be assigned to a node or vertex of the graph, where attributes are property values: such as name, date of birth, etc. In another example, the data record may be a record of a relational database.

[0037] Data transactions can be evaluated to determine their characteristics or attributes. This evaluation can, for example, indicate whether the transaction data is abnormal, unsafe, or the like. This evaluation can be performed, for example, using a trained ML model. For example, an ML model can be trained on historical telecommunications asset failure data (e.g., including sensor data) to predict asset failures before they cause outages. However, information technology operations need to ensure that ML models accurately predict failures, but the data is complex. In another example, an ML model can be trained based on historical, successful, and unsuccessful forecast coverage data. A trained ML model can help demand planners make adjustments to their forecasted demand. However, a trained ML model may need to be monitored, for example, for accuracy over time, so that AI-powered applications can be verified to consistently produce results as accurate as knowledge workers. In a further example, an ML model can be trained on historical transaction data to identify suspicious patterns. A trained model may need to be monitored to help banks keep pace with evolving regulations and allow financial crime analysts to understand the reasoning behind their model's alert analysis so they can make decisions about which alerts to dismiss and which to escalate.

[0038] In operation 101, a set of analysis bins (named "InitSet" for clarity) may be determined. An analysis bin represents a set of values of analysis attributes of records of data transactions. The set of analysis bins may or may not have equal width or size. The analysis attribute may, for example, be the time at which the data transaction occurred. In this case, the set of analysis bins may cover, for example, a time range of one month, and each of the analysis bins may cover a corresponding time interval, such as the first week of the month, etc. In another example, the analysis attribute may be the age of the user requesting the data transaction. In this case, the set of analysis bins may cover, for example, ages between 18 and 100 years old, and each of the analysis bins may cover a corresponding time range, such as 80 to 100 years old. To simplify the description, it is assumed that the analysis bin set InitSet includes 10 bins B1 to B10.

[0039] Data transactions executed by a computer system can be associated with corresponding analysis bins in the set of analysis bins. Based on the above example, all transactions triggered by users between the ages of 80 and 100 can be associated with analysis bins [80, 100]. This means that each analysis bin X in the set of analysis bins can be associated with a data record, where each data record has a value for an analysis attribute that falls within analysis bin X.

[0040] In one example, the set of analysis bins can be user-defined. For example, user input can be received in operation 101, where the user input indicates the set of analysis bins. In another example, multiple sets of analysis bins can be predefined (e.g., pre-stored), where determining the set of analysis bins in operation 101 can include selecting (e.g., randomly) one of the multiple predefined sets of analysis bins. In one example, the set of analysis bins can be determined such that the number of transactions associated with each analysis bin in the set of analysis bins is above a predefined threshold number of transactions. This threshold number of transactions can be sufficient to perform an overall performance analysis, for example, to evaluate an overall performance metric.

[0041] In operation 103, an overall performance indicator of the computer system can be calculated for each bin in the set of analysis bins. The calculation can be performed using transaction records with attribute values represented by the analysis bins. The overall performance indicator can be, for example, an average transaction duration. To this end, for each analysis bin X in the set of analysis bins, a transaction duration can be determined for each transaction associated with analysis bin X. Furthermore, an average value of the determined transaction durations can be calculated and assigned to analysis bin X. In another example, the overall performance indicator can be the number of failed transactions. To this end, for each analysis bin X in the set of analysis bins, the number of failed transactions for this bin X can be determined.

[0042] Data records describing transactions can be of one or more types. For example, a transaction can be associated with a master record describing the transaction's master attributes / properties and another ML record describing the results of executing an ML model on the transaction. The master record can include master attributes. An ML record can include ML attributes. An ML record is a record in the payload logging table. The master and ML records can be linked to each other via a transaction ID belonging to both records. In another example, a single type record can be used to describe a transaction. For example, a single type record can include attributes from both the master and ML records. If an ML model is not executed for the transaction associated with that single record, that single record can include null values for the ML attributes.

[0043] It may be determined (query operation 105) whether the number of records for one or more analysis bins of the group analysis is less than a predefined minimum number of records. A number of records below the minimum number of records may not be sufficient to perform ML performance monitoring in a given bin, whereas it may be sufficient to calculate the overall performance indicator in operation 103. In the case of two different types of records, the query operation 105 may be performed on the ML records. For example, it may be determined in the query operation 105 whether the number of ML records for one or more analysis bins of the group analysis is less than a predefined minimum number of records. In the case of a single type of record, it may be determined in the query operation 105 whether the number of records with non-empty ML attribute values is less than a predetermined minimum number of records. For example, the query operation 105 may be performed as follows. Each analysis bin X in the group analysis InitSet may be processed to determine whether the number of records whose analysis attribute values fall into that bin X is less than a predefined minimum number of records.

[0044] If it is determined (query operation 105) that the number of records for one or more analysis bins for the set of analyses InitSet is less than the predefined minimum number of records, operations 107 to 111 may be performed, otherwise operation 113 may be performed—assuming, for example, that bins B2 and B5 are determined to have a number of records less than the predefined minimum number of records.

[0045] In operation 107, a new or another set of analysis bins (named 'NewSet' for clarification purposes) may be determined or redefined. Following the example of InitSet above, the redefined bin set NewSet may include n bins rB1 to rBn, where n < 10. Operation 107 may be performed, for example, by connecting the analysis bins of the set of analysis bins InitSet. Following the example above, since only B2 and B5 have a number of records below a predefined minimum number of records, bins B2 and B3 of InitSet may be merged to form a new bin rB2, and bins B4 and B5 of InitSet may be merged to form a new bin rB3. This may result in 8 redefined bin sets rB1 to rB8 of NewSet, where rB1 is B1, rB4 is B6, rB5 is B7, rB6 is B8, rB7 is B9, and rB8 is B10, i.e., rB2 and rB3 are redefined. This may utilize the existing set of analysis bins InitSet in order to define a new set of bins. This can save resources because the accumulated records of unchanged bins can be reused. In another example, a new bin set NewSet can be defined independently of the set of analysis bins InitSet of operation 101, and by determining a new width of the new bin set NewSet, the number of records in each bin of the new set can be higher than a predefined minimum number of records.

[0046] For each bin rB in the redefined bin set NewSet, an ML performance indicator of the ML model may be calculated in operation 109 using records having attribute values represented by each bin rB. The ML performance indicator may be, for example, a prediction accuracy of the ML model. For example, each record in the ML records may include an ML attribute describing the accuracy of the ML prediction of the data transaction for the evaluation record. In operation 109, for each bin of the redefined bin NewSet, the precision of the ML records for the bin may be averaged to provide a value of the ML performance indicator for the bin. In another example, the ML performance indicator may be a fairness score.

[0047] Following the above example, operation 109 can generate 8 values of the ML performance indicator, each associated with a corresponding bin of the redefined bin set NewSet. However, the total performance indicator has been evaluated 10 times for the analysis bins of InitSet. This may result in a suboptimal correlation analysis between the two metrics. To address this issue, in operation 111, the ML performance indicator of the redefined bin set NewSet can be used to estimate the ML performance indicator within each bin of the analysis bin set InitSet. For example, given the 8 values of the ML performance indicator in the bins of NewSet, 10 values of the ML performance indicator can be derived for the bins of InitSet. Based on the above example, the ML performance indicators of bins rB1, rB4, rB5, rB6, rB7, and rB8 of NewSet can be the same for the corresponding bins B1, B6, B7, B8, B9, and B10. By combining the metric values of the surrounding bins (e.g., B1, rB1, rB2, and B6) (or by extrapolating the values of the surrounding bins), the ML performance indicator can be estimated for bins B2 to B5 of InitSet. Other examples of performing the estimation are shown in FIG. 2 to FIG. 3 .

[0048] When each bin B in the analysis bin set InitSet has a record number higher than a predefined minimum record number, operation 113 may be performed as follows: For each bin B in the set of bins InitSet, an ML performance indicator of the ML model may be calculated in operation 113 using the records having the attribute value represented by each bin B.

[0049] After performing operation 111 or operation 113, each bin in the bin set InitSet may have a pair of values for the ML performance indicator and the total performance indicator. This enables bin-by-bin comparison of the values of the two metrics. Specifically, the behavior of the ML performance indicator along the bin set InitSet can be compared with the behavior of the total performance indicator along the bin set InitSet. This enables accurate correlation analysis of the two metrics, and therefore, the present method can reliably use this correlation. For example, based on the correlation between the calculated total performance indicator and the ML performance indicator on the analysis bin set InitSet, the computer system can be configured in operation 115 so that further transactions can have a positive correlation between the total performance indicator and the ML performance indicator. The configuration can be based on the correlation between the overall and ML performance indicators. For example, if the correlation is negative, this can indicate that the trained ML model is insufficient for the use case being used. For example, the trained ML model can process data from a given field, such as telecommunications, well. However, for other fields, it may not provide the required accuracy. In another example, after performing operation 111 or operation 113, information indicating the correlation between the overall performance indicator and the ML performance indicator may be provided to the user, for example, and may be used by the user as monitoring information for the computer system.

[0050] The correlation between two metrics can have the following characteristics. In one example, the correlation between the two metrics can be a strong positive correlation. For example, a decline in an ML performance metric drives a decline in a specific KPI. For example, a 2% drop in the model's fairness score drives a 5% drop in the amount of credit awarded for that KPI. This indicates that the quality of resource investment in a specific area of the model may be important. The system can be configured accordingly to further improve the ML model to avoid a decline in the model's fairness score.

[0051] In one example, a specific ML performance metric improved without impacting the KPI. This means, for example, that a 5% improvement in model accuracy had no impact on click counts. This insight clearly indicates that the investment in model accuracy may not be worthwhile and that a new ML model can be used to replace the ML model.

[0052] In one example, there was no correlation at all (or very little correlation) between any ML performance metric and the KPI. This could indicate a serious problem with the ML model, where the model results were completely ignored in the process. This could trigger an alert to re-examine the decision-making process and the configuration of the computer system.

[0053] As a result, based on the correlation between the ML performance metric and the overall performance metric, the computer system can be configured accordingly. This configuration can be performed so that it achieves a positive correlation between the two metrics in order to execute the next trade. For example, the ML performance metric calculated for a future trade can have an improved value consistent with the overall performance metric. The present disclosure can also consider the combined effects of multiple monitoring metrics.

[0054] For example, operation 115 can automatically trigger retraining of the ML model. Retraining can be performed using new data corresponding to the current usage of the computer system. In another example, constraints can be enforced by augmenting a previously used training set to improve the accuracy of the trained model using specific input from payload analysis, thereby meeting the target and adapting to the new data. In another example, the ML model can generate more data on the currently trained model without having to retrain, for example by updating the minimum sample size and thresholds, without incurring additional processing costs. This can avoid intensive CPU usage when the underlying data has not changed.

[0055] Figure 2A is a flowchart of a method of defining analytical bins for computing metrics according to an example of the present disclosure.

[0056] In operation 201, a set of analysis bins B1 to B10 may be provided. Figure 2B As shown, the analysis bin set can cover the time range [tS, tE] =

[010] . The set of analysis bins includes 10 bins B1 to B10 with a width of 1, as shown in FIG. Figure 2B The data in each of bins B1 to B10 may not be sufficient to calculate the ML performance metric.

[0057] In operation 203, a new group of bins rB1 to rB5 is (re)defined by combining two consecutive bins from the set of 10 bins B1 to B10. This may be done, for example, because the data in 10 bins may not be sufficient to calculate the ML performance indicator. This results in 5 new bins rB1 to rB5 of width 2. For example, new bin rB1 may be obtained by merging bins B1 and B2, new bin rB2 may be obtained by merging bins B3 and B4, new bin rB3 may be obtained by merging bins B5 and B6, new bin rB4 may be obtained by merging bins B7 and B8, and new bin rB5 may be obtained by merging bins B9 and B10.

[0058] Figure 2BFurther shown are data points 220 for each bin of the new set of analysis bins rB1-rB5, representing the value of the ML performance indicator in each of the five bins rB1 through rB5. To estimate the value of the ML performance indicator in each of the ten bins B1 through B10, a fitting 222 (or modeling) of the distribution of values of the ML performance indicator can be performed. The fitting 222 (which is a cubic spline approximation) can be used to estimate or approximate the value 224 of the ML performance indicator in each of the ten bins.

[0059] Figure 3A is a flow chart of a method for defining analytical bins for calculating metrics according to examples of the present disclosure.

[0060] In operation 301, a set of analysis bins may be provided. The set of analysis bins may cover five clusters, such as Figure 3B As shown. The analysis bin set includes five bins, each associated with a corresponding cluster 320.1-5. Each of the clusters 320.1-5 may include records with similar values for a corresponding attribute. For example, cluster 320.1 may include records for users between the ages of 20 and 40, and cluster 320.2 may include records for users in a given region or country, etc.

[0061] In operation 303, a new set of bins is defined by combining two or more bins from the five bin sets. This may be performed, for example, because the data in the five bins may not be sufficient to calculate an ML performance indicator. This produces three new bins 322.1 to 322.3, each associated with a respective merged set of clusters (e.g., the merged set of clusters may be referred to as merged clusters). For example, new bin 322.1 represents the merged set of clusters 320.1, 320.2, and 320.3, new bin 322.2 represents the merged set of clusters 320.4, 320.2, and 320.3, and new bin 322.3 represents the merged set of clusters 320.5, 320.4, and 320.3. These clusters may be connected, for example, based on the distance between them. The distance may be calculated using one or more attributes of the records of the clusters. An ML performance indicator may be calculated for the three bins 322.1 to 322.3. And to obtain the ML performance indicator for each bin 320.1 to 320.5, the following formula may be used.

[0062] For each bin j in the set of analysis bins 320.1 to 320.5, the ML performance metric can be estimated as follows: sum(wi*mi) / sum(wi), where i is the index of the merged cluster set, which ranges from 1 to 3 in this example, where mi is the ML performance metric for the merged cluster set i and wi is calculated as: meanDtoJ / maxD*nPinJ / nPinCls, where meanDtoJ is the average distance between the centers of the merged cluster set i and the centers of cluster j, maxD is the maximum distance between the centers of the merged cluster set i, nPinJ is the number of data points in cluster j, and nPinCls is the number of data points in the merged cluster set i. In some embodiments, meanDtoJ can represent the following: the average distance between the original centers in the merged cluster and the center of cluster j. In some embodiments, maxD can represent the following: the maximum distance between the original centers. In some embodiments, nPinJ can represent the following: the number of data points in cluster j. In some embodiments, nPinCls can represent the following: the maximum distance between the original centers.

[0063] Figure 4 401. A general purpose computerized system 400 is shown that is suitable for implementing at least a portion of the method steps involved in the present disclosure. It should be understood that the methods described herein are at least partially non-interactive and automated by a computerized system such as a server or embedded system. However, in some embodiments, the methods described herein can be implemented in (partial) interactive systems. These methods can further be implemented in software 412, 422 (including firmware 422), hardware (processor) 405, or a combination thereof. In some embodiments, the methods described herein are implemented in software as an executable program and are executed by a dedicated or general purpose digital computer (such as a personal computer, workstation, minicomputer, or mainframe computer). Therefore, the most general purpose system 400 includes a general purpose computer 401.

[0064] In some embodiments, in terms of hardware architecture, such as Figure 4As shown, computer 401 includes a processor 405, a memory (main memory) 410 coupled to a memory controller 415, and one or more input and / or output (I / O) devices (or peripheral devices) 10, 445 communicatively coupled via a local input / output controller 435. The input / output controller 435 can be, but is not limited to, one or more buses or other wired or wireless connections as known in the art. The input / output controller 435 can have additional elements omitted for simplicity, such as controllers, buffers (cache memories), drivers, repeaters, and receivers to enable communication. Further, the local interface can include address, control, and / or data connections to enable appropriate communication between the above components. As described herein, the I / O devices 10, 445 can generally include any universal cryptographic card or smart card known in the art.

[0065] Processor 405 is a hardware device for executing software, particularly software stored in memory 410. Processor 405 can be any custom or commercially available processor, a central processing unit (CPU), a secondary processor among several processors associated with computer 401, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, or generally any device for executing software instructions.

[0066] The memory 410 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM, such as DRAM, SRAM, SDRAM, etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM)). Note that the memory 410 may have a distributed architecture in which different components are located remotely from each other but can be accessed by the processor 405.

[0067] The software in the memory 410 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions (especially the functions involved in the embodiments of the present disclosure). Figure 4 In the example of , the software in memory 410 includes instructions 412, for example, instructions to manage a database such as a database management system.

[0068] The software in memory 410 will typically also include a suitable operating system (OS) 411. The OS 411 essentially controls the execution of other computer programs, such as software 412 that may be used to implement the methods as described herein.

[0069] The methods described herein may be in the form of a source program 412, an executable program 412 (object code), a script, or any other entity that includes a set of instructions to be executed 412. If it is a source program, the program may need to be translated via a compiler, assembler, interpreter, etc., which may or may not be included in the memory 410, in order to operate properly in conjunction with the OS 411. Furthermore, the methods may be written in an object-oriented programming language with data and method classes, or in a procedural programming language with routines, subroutines, and / or functions.

[0070] In some embodiments, a keyboard 450 and a mouse 455 may be coupled to the input / output controller 435. Other output devices such as I / O devices 445 may include input devices such as, but not limited to, printers, scanners, microphones, and the like. Finally, I / O devices 10, 445 may further include devices that transmit both input and output, such as, but not limited to, network interface cards (NICs) or modulators / demodulators (for accessing other files, devices, systems, or networks), radio frequency (RF) or other transceivers, telephone interfaces, bridges, routers, and the like. I / O devices 10, 445 may be any common cryptographic card or smart card known in the art. System 400 may further include a display controller 425 coupled to display 430. In some embodiments, system 400 may also include a network interface for coupling to a network 465. Network 465 may be an IP-based network for communicating between computer 401 and any external servers, clients, and the like via a broadband connection. Network 465 transmits and receives data between computer 401 and external systems 30, which may be involved in performing some or all of the steps of the methods discussed herein. In some embodiments, network 465 may be a managed IP network managed by a service provider. Network 465 may be implemented wirelessly (e.g., using wireless protocols and technologies such as WiFi, WiMax, etc.). Network 465 may also be a packet-switched network, such as a local area network, a wide area network, a metropolitan area network, an internet network, or other similar types of network environments. Network 465 may be a fixed wireless network, a wireless local area network (LAN), a wireless wide area network (WAN), a personal area network (PAN), a virtual private network (VPN), an intranet, or other suitable network system, and include equipment for receiving and transmitting signals.

[0071] If the computer 401 is a PC, workstation, smart device, etc., the software in the memory 410 may also include a basic input and output system (BIOS) 422. The BIOS is a collection of basic software routines that initialize and test hardware at startup, start the OS 411, and support data transfer between hardware devices. The BIOS is stored in ROM so that it can be executed when the computer 401 is activated.

[0072] When the computer 401 is running, the processor 405 is configured to execute software 412 stored in the memory 410, transfer data to and from the memory 410, and generally control the operation of the computer 401 in accordance with the software. The methods and OS 411 described herein (in whole or in part, but typically the latter) are read by the processor 405, possibly buffered within the processor 405, and then executed.

[0073] When the systems and methods described herein are implemented in software 412, such as Figure 4 As shown, these methods can be stored on any computer-readable medium, such as storage 420, for use by or in conjunction with any computer-related system or method. Storage 420 can include disk storage, such as HDD storage.

[0074] The present invention may be a system, method and / or computer program product of any possible degree of technical detail integration. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon for causing a processor to execute various aspects of the present invention.

[0075] Computer readable storage medium can be a tangible device that can retain and store the instructions used by the instruction execution device.Computer readable storage medium can be, for example but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the above.The non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as punch card or the protrusion structure in the groove with the instruction recorded thereon and any suitable combination of the above.Computer readable storage medium as used herein should not be interpreted as temporary signal itself, such as radio wave or other free propagation electromagnetic wave, electromagnetic wave propagated by waveguide or other transmission media (for example, light pulse passing through fiber optic cable) or electric signal emitted by wire.

[0076] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or downloaded to an external computer or external storage device. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0077] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, the configuration data of integrated circuit or source code or object code written in any combination of one or more programming languages, these programming languages include object-oriented programming languages (such as Smalltalk, C++ etc.) and process programming languages (such as " C " programming languages or similar programming languages). The computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, performed as an independent software package, partly on the user's computer, partly on a remote computer or fully on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer by any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an internet service provider through the internet). In certain embodiments, the electronic circuit comprising for example programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (PLA) can make the electronic circuit personalized to perform computer-readable program instructions by utilizing the state information of computer-readable program instructions, so as to perform various aspects of the present invention.

[0078] The present invention will be described below with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0079] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in the flowchart and / or block diagram or multiple blocks. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to operate in a specific manner. Thus, the computer-readable storage medium having the instructions stored therein includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in the flowchart and / or block diagram or multiple blocks.

[0080] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in the flowchart and / or block diagram or multiple boxes.

[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to different embodiments of the present invention. To this end, each box in the flowchart or block diagram may represent a module, segment or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may not occur in the order marked in the figure. For example, the two boxes shown in succession can actually be completed as a step, performed simultaneously, substantially simultaneously, in a partially or completely time-overlapping manner, or the boxes can sometimes be performed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0082] This topic may include the following clauses.

[0083] 1. A method for controlling the operation of a computer system, the computer system being configured to perform data transactions and using a machine learning (ML) model to evaluate characteristics of the data transactions, the method comprising: determining a set of analysis bins, wherein an analysis bin represents a set of attribute values of the data transaction records;

[0084] calculating an overall performance index of the computer system, the overall performance index indicating transaction execution performance of the computer system for each analysis bin in the set of analysis bins using records of transactions having attribute values represented by the analysis bins; and in the event that one or more bins in the set of analysis bins do not have at least a predefined minimum number of records:

[0085] Redefine a new analysis bin set by merging the analysis bins of the analysis bin set,

[0086] For each bin in the redefined set of bins, calculating a machine learning performance metric of the ML model using records having attribute values represented by each bin;

[0087] estimating the ML performance indicator in each bin of the analysis bin set using the ML performance indicator of the redefined bin set;

[0088] calculating the ML performance indicator in each bin in the set of analysis bins if each bin in the set of analysis bins has at least the minimum number of records;

[0089] Based on the correlation across the set of analysis bins between the calculated overall performance indicator and the ML performance indicator, the computer system is configured to achieve a positive correlation between the overall performance indicator and the ML performance indicator for further executed data transactions.

[0090] 2. The method of clause 1, further comprising: in a case where the correlation is a negative correlation or a zero correlation, performing an update comprising any one of: retraining the ML model, adding an additional ML model for implementing a combination of evaluations of the attributes, replacing the ML model by another ML model, wherein the configuration of the computer system comprises using the performed update to evaluate the further transaction.

[0091] 3. The method of clause 1, wherein the correlation between the overall performance metric and the ML performance metric in the analysis set is a positive correlation, improving the ML model by retraining the ML model using a larger training dataset.

[0092] 4. A method according to any of the preceding clauses, wherein the attribute is the time at which the data transaction occurred, wherein each bin in the set of analysis bins represents a time interval, wherein the redefined bin of the redefined bin is obtained by merging two or more temporally consecutive bins in the set of analysis bins, wherein the estimating is performed by modeling the change of the ML performance indicator according to the redefined bin, and the value of the ML performance indicator in the set of analysis bins is determined using the ML model.

[0093] 5. The method of clause 4, wherein the modeling comprises fitting a distribution of the ML performance metric over the redefined bins.

[0094] 6. The method of any of the preceding clauses 1-3, wherein each analysis bin in the set of analysis bins represents a set of values of different attributes of records of data transactions, wherein the set of values is the values of a cluster of records formed using the different attributes, and wherein the redefined bin is obtained by merging two or more bins in the set of analysis bins whose associated clusters have a predefined distance from each other.

[0095] 7. A method according to clause 6, wherein estimating the ML performance indicator includes: for each bin in the set of analysis bins, the ML performance indicator of cluster j associated with the bin is defined as follows: sum(wi*mi) / sum(wi), where mi is the ML metric of the set i of joined clusters, and wi is calculated as: meanDtoJ / maxD*nPinJ / nPinCls, where meanDtoJ is the average distance between the center of the set i of merged clusters and the center of cluster j, mayD is the maximum distance between the centers of the set i of merged clusters, nPinJ is the number of data points in cluster j, and nPinCls is the number of data points in the set i of connected clusters.

[0096] 8. The method of any of the preceding clauses, performed at runtime on the computer system.

[0097] 9. The method according to any of the preceding clauses, wherein the method is repeated for a further set of analysis bins using a controlled computer system.

[0098] 10. The method of any of the preceding clauses, further comprising repeating the method for different ML performance metrics.

[0099] 11. The method according to any of the preceding clauses, wherein the analysis bins are bins of equal size.

[0100] 12. The method according to any of the preceding clauses, wherein the calculation of the overall performance indicator or the ML performance indicator further comprises normalizing the calculated indicator.

[0101] 13. The method of any of the preceding clauses, further comprising collecting records of the data transactions associated with each analysis silo in the set of analysis silos for performing the calculation on the collected records.

[0102] 14. The method according to any of the preceding clauses, wherein the overall performance indicator is a key performance indicator (KPI).

[0103] 15. The method of any of the preceding clauses, wherein the ML performance metric is one of: accuracy and fairness score of the ML model's predictions.

[0104] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to explain the principles of the embodiments, practical applications, or technical improvements to technologies found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for controlling the operation of a computer system configured to perform data transactions and evaluate characteristics of the data transactions using a machine learning (ML) model, the method comprising: Determining an analysis bin set, where an analysis bin represents a set of record attribute values of the data transaction; calculating an overall performance index of the computer system, the overall performance index indicating transaction execution performance of the computer system for each analysis bin in the set of analysis bins using records of transactions having attribute values represented by the analysis bins; In response to one or more analysis bins in the set of analysis bins not having at least a predefined minimum number of records: redefine a new analysis bin set by merging analysis bins of the analysis bin set; For each analysis bin in the redefined set of analysis bins, calculating a machine learning performance metric of the ML model using records having attribute values represented by each analysis bin of the redefined set; estimating the ML performance indicator in each analysis bin in the set of analysis bins using the ML performance indicator of the redefined set of analysis bins; responsive to each analysis bin in the set of analysis bins having at least a minimum number of records, calculating the ML performance indicator in each analysis bin in the set of analysis bins; Based on the correlation across the set of analysis bins between the calculated overall performance metric and the ML performance metric, the computer system is configured to achieve a positive correlation between the overall performance metric and the ML performance metric for further executed data transactions.

2. The method of claim 1 , in response to the correlation between the total performance indicator in the analysis set and the ML performance indicator being a negative correlation or a zero correlation, performing an update comprising adding an additional ML model that enables combining the evaluation of the characteristics.

3. The method of claim 1 , in response to the correlation between the overall performance metric and the ML performance metric in the analysis set being a positive correlation, improving the ML model by retraining the ML model using a larger training dataset.

4. The method according to claim 1, wherein The attribute is the time of occurrence of the data transaction, wherein each analysis bin in the set of analysis bins is a representation of a time interval, wherein a redefined bin in the set of analysis bins is obtained by merging two or more temporally consecutive bins in the set of analysis bins, wherein the estimation is performed by modeling the variation of the ML performance metric as a function of the redefined set of analysis bins, and The method further determines a value of the ML performance indicator in the set of analysis bins using the ML model.

5. The method of claim 4, wherein the modeling comprises fitting a distribution of the ML performance metric on the redefined bins.

6. The method according to claim 1, wherein Each analysis bin in the set of analysis bins represents a set of values of different attributes of records of the data transaction, wherein the set of values is the values of a record cluster formed using the different attributes, and wherein the redefined bins in the redefined bins are obtained by merging two or more bins in the set of analysis bins whose associated clusters have a predetermined distance from each other. The method of claim 1 , executed at runtime on the computer system.

8. The method according to claim 1, wherein The method is repeated for additional sets of analysis bins using the controlled computer system.

9. The method of claim 1 , further comprising repeating the method for different ML performance metrics.

10. The method according to claim 1, wherein The analysis bins are bins of equal size.

11. The method according to claim 1, wherein The calculation of the overall performance indicator or the ML performance indicator further comprises normalizing the calculated indicator. 12 . The method of claim 1 , further comprising collecting records of the data transaction associated with each analysis silo in the set of analysis silos for performing calculations on the collected records. The method according to claim 1 , wherein the overall performance indicator is a key performance indicator.

14. The method of claim 1, wherein the ML performance metric is selected from the group consisting of an accuracy and a fairness score of the ML model's predictions.

15. A system for controlling the operation of a computer system, the system comprising: Memory; as well as a processor communicatively coupled to the memory, the processor configured to: Determining a set of analysis bins, where an analysis bin represents a set of values of record attributes of a data transaction; calculating an overall performance index of the computer system, the overall performance index indicating transaction execution performance of the computer system for each analysis bin in the set of analysis bins using records of transactions having attribute values represented by the analysis bins; In case one or more bins in the set of analysis bins does not have at least a predefined minimum number of records: redefine the new analysis bin set by merging the analysis bins of the analysis bin set, For each bin in the set of redefined bins, computing a machine learning (ML) performance metric of the ML model using records having attribute values represented by each bin; using the ML performance indicator of the redefined set of bins to estimate the ML performance indicator in each bin of the set of analysis bins; computing the ML performance indicator in each bin in the set of analysis bins if each bin in the set of analysis bins has at least a minimum number of records; Based on the correlation across the set of analysis bins between the calculated overall performance metric and the ML performance metric, the computer system is configured to achieve a positive correlation between the overall performance metric and the ML performance metric for further executed data transactions.

16. A computer program product, comprising: one or more computer-readable storage media; as well as program instructions collectively stored on the one or more computer-readable storage media, the program instructions being configured to: Determine a set of analysis bins, where an analysis bin represents a set of record attribute values of a data transaction; calculating an overall performance index of the computer system, the overall performance index indicating transaction execution performance of the computer system for each analysis bin in the set of analysis bins using records of transactions having attribute values represented by the analysis bins; In case one or more bins in the set of analysis bins does not have at least a predefined minimum number of records: redefine a new set of analysis bins by merging analysis bins of the set of analysis bins, For each bin in the redefined set of bins, calculating a machine learning (ML) performance metric of the ML model using records having attribute values represented by each bin; estimating the ML performance indicator for each analysis bin in the set of analysis bins using the ML performance indicator of the redefined set of analysis bins; and calculating the ML performance indicator in each bin in the set of analysis bins if each bin in the set of analysis bins has at least a minimum number of records. Based on the correlation across the set of analysis bins between the calculated overall performance indicator and the ML performance indicator, the computer system is configured to achieve a positive correlation between the overall performance indicator and the ML performance indicator for further executed data transactions.

Images