Methods, systems, and computer program products for ensemble learning with rejection

Abstract

Methods, systems, and computer program products for ensemble learning are provided. An example system includes at least one processor configured to: (i) generate a rejection region for each baseline model of a set of baseline models; (ii) generating a global rejected region based on the rejected region of each baseline model; (iii) training an integrated machine learning model; (iv) updating a set of baseline machine learning models based on the baseline model predicted performance metrics for each baseline machine learning model; and (iv) repeating (i)-(iv) until there is a single baseline model in the set of baseline models or the predicted performance or global acceptance of an integrated model meets a threshold.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is related to U.S. Patent Provisional Application Serial No. 63 / 503,294 filed on May 19, 2023, the disclosure of which is hereby incorporated by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to ensemble learning and, in non-limiting embodiments or aspects, to methods, systems, and computer program products for ensemble learning with rejection to improve performance and confidence in classification tasks. Background Art

[0004] Recent studies have found that selective ensemble learning (e.g., dynamic ensemble selection, etc.) shows better prediction performance for classification tasks than traditional static ensemble learning. However, available methods have some limitations that affect practical implementation, such as high computational cost and / or limitations in ranking and clustering of baseline machine learning models, especially for class-imbalanced data. In addition, existing methods can make predictions for all data without measuring the model confidence about different feature patterns. Summary of the invention

[0005] Thus, provided are improved methods, systems, and computer program products for integrated learning.

[0006] According to non-limiting embodiments or aspects, provided is a method comprising: (i) for each baseline machine learning model in a set of baseline machine learning models, utilizing at least one processor: training the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; for the baseline machine learning model, generating a rejection region associated with at least one data type of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; and utilizing the baseline machine learning model to process a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model; and (ii) utilizing at least one processor to process a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model based on the plurality of first predictions for the plurality of first training samples. The method further comprises the steps of: (i) generating a global rejection region associated with one or more data types of a plurality of different data types by using a rejection region associated with the model; and (iii) training an integrated machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples of a plurality of second training samples outside the global rejection region; (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model; and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein the training of the integrated machine learning model generates a rejection region for the plurality of second training samples outside the global rejection region. (iv) updating, using at least one processor, the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) repeating (i)-(iv) using at least one processor until there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the ensemble model prediction performance metric satisfies a threshold ensemble model prediction performance, a ratio of the plurality of second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0007] In some non-limiting embodiments or aspects, for each baseline machine learning model in a set of baseline machine learning models, generating, for the baseline machine learning model, a rejection region associated with at least one of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples comprises optimizing according to the following equation:

[0008]

[0009] in

[0010] and the objective function defined by any other constraints of interest;

[0011] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region; N rej is the number of first samples that are rejected because their scores are within the rejection region; N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0012] In some non-limiting embodiments or aspects, t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0013] In some non-limiting embodiments or aspects, for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model, and wherein training the integrated machine learning model using at least one processor is further based on (d) multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0014] In some non-limiting embodiments or aspects, each baseline machine learning model in a set of baseline machine learning models comprises a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and wherein the rejection region of each baseline machine learning model comprises q–1 bounds defining the rejection region.

[0015] In some non-limiting embodiments or aspects, a metamodel is used to integrate a set of baseline machine learning models into an integrated machine learning model.

[0016] In some non-limiting embodiments or aspects, the method further comprises: (vi) obtaining a current sample using at least one processor; (vii) determining whether the current sample is within a global rejection region using at least one processor; (viii) in response to determining that the current sample is outside the global rejection region, automatically processing the current sample using an integrated machine learning model using at least one processor to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically marking the current sample as unable to receive a credible prediction from the integrated classifier using at least one processor.

[0017] According to some non-limiting embodiments or aspects, provided is a system comprising: at least one processor configured to: (i) for each baseline machine learning model in a set of baseline machine learning models, train the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; for the baseline machine learning model, generate a rejection region associated with at least one data type among the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; and process a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model ; (ii) generating a global rejection region associated with one or more of a plurality of different data types based on the rejection region associated with each baseline machine learning model; (iii) training an integrated machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples from a plurality of second training samples outside the global rejection region; (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model; and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein the training integrated machine learning model generates a global rejection region for the global rejection region. (i) a subset of ensemble predictions of another subset of second training samples in a plurality of second training samples outside the region, and wherein the ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) repeating (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one ensemble model prediction performance metric in the ensemble model prediction performance metric satisfies a threshold ensemble model prediction performance, a ratio of the plurality of second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0018] In some non-limiting embodiments or aspects, at least one processor is configured to, for each baseline machine learning model in a set of baseline machine learning models, generate, for the baseline machine learning model, a rejection region associated with at least one of a plurality of different data types based on a plurality of first predictions for a plurality of first training samples by optimizing an objective function defined according to the following equation and any other constraints of interest:

[0019]

[0020] in

[0021] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region; N rej is the number of first samples that are rejected because their scores are within the rejection region; N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0022] In some non-limiting embodiments or aspects, t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0023] In some non-limiting embodiments or aspects, for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model, and wherein at least one processor is further configured to train the integrated machine learning model based on (d) the multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0024] In some non-limiting embodiments or aspects, each baseline machine learning model in a set of baseline machine learning models comprises a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and wherein the rejection region of each baseline machine learning model comprises q–1 bounds defining the rejection region.

[0025] In some non-limiting embodiments or aspects, a metamodel is used to integrate a set of baseline machine learning models into an integrated machine learning model.

[0026] In some non-limiting embodiments or aspects, at least one processor is further configured to: (vi) obtain a current sample; (vii) determine whether the current sample is within a global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically process the current sample using an integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically mark the current sample as unable to receive a credible prediction from the integrated classifier.

[0027] According to some non-limiting embodiments or aspects, provided is a computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: (i) for each baseline machine learning model in a set of baseline machine learning models, train the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples comprise a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; and for the baseline machine learning model, based on a plurality of first predictions for the plurality of first training samples, The method further comprises the steps of: processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using a first prediction to generate a rejection region associated with at least one data type among a plurality of different data types; and processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each of the plurality of rejection flags indicates a plurality of whether the corresponding second sample in the second sample is within the rejection region of the baseline machine learning model; (ii) generating a global rejection region associated with one or more of the multiple different data types based on the rejection region associated with each baseline machine learning model; (iii) training an integrated machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of the second training samples in the multiple second training samples outside the global rejection region; (b) a plurality of rejection flags for the multiple second samples associated with each baseline machine learning model; and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein the training set (i) generating a subset of ensemble predictions for another subset of second training samples in a plurality of second training samples outside a global rejection region, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) repeating (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one ensemble model prediction performance metric in the ensemble model prediction performance metric satisfies a threshold ensemble model prediction performance, a ratio of the plurality of second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0028] In some non-limiting embodiments or aspects, the program instructions, when executed by at least one processor, cause the at least one processor, for each baseline machine learning model in a set of baseline machine learning models, to generate, for the baseline machine learning model, a rejection region associated with at least one data type among a plurality of different data types based on a plurality of first predictions for a plurality of first training samples by optimizing an objective function defined according to the following equation and any other constraints of interest:

[0029]

[0030] in

[0031] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region; N rej is the number of first samples that are rejected because their scores are within the rejection region; N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0032] In some non-limiting embodiments or aspects, t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0033] In some non-limiting embodiments or aspects, for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model, and wherein the program instructions, when executed by at least one processor, further cause the at least one processor to train the integrated machine learning model based on (d) the multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0034] In some non-limiting embodiments or aspects, a metamodel is used to integrate a set of baseline machine learning models into an integrated machine learning model.

[0035] In some non-limiting embodiments or aspects, the program instructions, when executed by at least one processor, further cause the at least one processor to: (vi) obtain a current sample; (vii) determine whether the current sample is within a global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically process the current sample using an integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically mark the current sample as unable to receive a credible prediction from the integrated classifier.

[0036] Additional non-limiting embodiments or aspects are set forth in the following numbered clauses:

[0037] Clause 1. A method comprising: (i) for each baseline machine learning model in a set of baseline machine learning models, using at least one processor: training the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; for the baseline machine learning model, generating a rejection region associated with at least one data type of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; and processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model; (ii) using at least one processor, based on the rejection region associated with each baseline machine learning model (iii) using at least one processor to train an ensemble machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples from the plurality of second training samples outside the global rejection region; (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model; and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein the training ensemble machine learning model generates predictions for the second training samples from the plurality of second training samples outside the global rejection region. a subset of ensemble predictions for another subset of training samples, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating, using at least one processor, the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) repeating, using at least one processor, (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one ensemble model prediction performance metric in the ensemble model prediction performance metric satisfies a threshold ensemble model prediction performance, a ratio of a plurality of second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0038] Clause 2. The method of clause 1, wherein for each baseline machine learning model in a set of baseline machine learning models, generating, for the baseline machine learning model, a rejection region associated with at least one of a plurality of different data types based on a plurality of first predictions for a plurality of first training samples comprises optimizing according to the following equation:

[0039]

[0040] in

[0041] and the objective function defined by any other constraints of interest;

[0042] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region; N rej is the number of first samples that are rejected because their scores are within the rejection region; N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0043] Clause 3. The method of clause 1 or 2, wherein t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0044] Clause 4. A method as described in any of clauses 1-3, wherein for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of a rejection region of the baseline machine learning model, and wherein training the integrated machine learning model using at least one processor is further based on (d) multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0045] Clause 5. A method as described in any of clauses 1-4, wherein each baseline machine learning model in a set of baseline machine learning models comprises a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and wherein the rejection region of each baseline machine learning model comprises q–1 bounds defining the rejection region.

[0046] Clause 6. A method as described in any of Clauses 1-5, wherein the metamodel is used to integrate a set of baseline machine learning models into an integrated machine learning model.

[0047] Clause 7. The method as described in any of Clauses 1-7 further includes: (vi) utilizing at least one processor to obtain a current sample; (vii) utilizing at least one processor to determine whether the current sample is within a global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, utilizing at least one processor to automatically process the current sample using an integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, utilizing at least one processor to automatically mark the current sample as unable to receive a credible prediction from the integrated classifier.

[0048] Clause 8. A system comprising: at least one processor configured to: (i) for each baseline machine learning model in a set of baseline machine learning models, train the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; for the baseline machine learning model, generate a rejection region associated with at least one data type of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; and process a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model; and (ii) based on The method comprises the steps of: (i) training an ensemble machine learning model based on the set of baseline machine learning models to generate a global rejection region associated with one or more data types of a plurality of different data types based on a rejection region associated with each baseline machine learning model; (iii) training an ensemble machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples in a plurality of second training samples outside the global rejection region; (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model; and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein the training ensemble machine learning model generates a subset of ensemble predictions for another subset of second training samples in a plurality of second training samples outside the global rejection region, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) repeating (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one ensemble model prediction performance metric in the ensemble model prediction performance metric satisfies a threshold ensemble model prediction performance, a ratio of the plurality of second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0049] Clause 9. The system of clause 8, wherein the at least one processor is configured to, for each baseline machine learning model in a set of baseline machine learning models, generate, for the baseline machine learning model, a rejection region associated with at least one of the plurality of different data types based on a plurality of first predictions for a plurality of first training samples by optimizing an objective function defined according to the following equation and any other constraints of interest:

[0050]

[0051] in

[0052] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region; N rej is the number of first samples that are rejected because their scores are within the rejection region; N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0053] Clause 10. A system as described in clause 8 or 9, wherein t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0054] Clause 11. A system as described in any of clauses 8-10, wherein for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of a rejection region of the baseline machine learning model, and wherein at least one processor is further configured to train the integrated machine learning model based on (d) multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0055] Clause 12. A system as described in any of clauses 8-11, wherein each baseline machine learning model in a set of baseline machine learning models comprises a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and wherein the rejection region of each baseline machine learning model comprises q–1 bounds defining the rejection region.

[0056] Clause 13. A system as described in any of Clauses 8-12, wherein the metamodel is used to integrate a set of baseline machine learning models into an integrated machine learning model.

[0057] Clause 14. A system as described in any of clauses 8-13, wherein at least one processor is further configured to: (vi) obtain a current sample; (vii) determine whether the current sample is within a global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically process the current sample using an integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically mark the current sample as unable to receive a credible prediction from the integrated classifier.

[0058] Clause 15. A computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: (i) for each baseline machine learning model in a set of baseline machine learning models, train the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; and for the baseline machine learning model, generate a plurality of first predictions for the plurality of first training samples in accordance with the plurality of different data types. a rejection region associated with at least one of the data types; and processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each of the plurality of rejection flags indicates a corresponding one of the plurality of second samples whether the second sample is within the rejection region of the baseline machine learning model; (ii) generating a global rejection region associated with one or more of the multiple different data types based on the rejection region associated with each baseline machine learning model; (iii) training an integrated machine learning model based on the set of baseline machine learning models based on: (a) another subset of the second training samples among the multiple second training samples that are outside the global rejection region; (b) a plurality of rejection flags for the multiple second samples associated with each baseline machine learning model; and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein the integrated machine learning model is trained (i) generating a subset of ensemble predictions for another subset of second training samples in a plurality of second training samples outside a global rejection region, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) repeating (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one ensemble model prediction performance metric in the ensemble model prediction performance metric satisfies a threshold ensemble model prediction performance, a ratio of the plurality of second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0059] Clause 16. The computer program product of clause 15, wherein the program instructions, when executed by at least one processor, cause the at least one processor, for each baseline machine learning model in a set of baseline machine learning models, to generate, for the baseline machine learning model, a rejection region associated with at least one data type among a plurality of different data types based on a plurality of first predictions for a plurality of first training samples by optimizing an objective function defined according to the following equation and any other constraints of interest:

[0060]

[0061] in

[0062] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region; N rej is the number of first samples that are rejected because their scores are within the rejection region; N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0063] Clause 17. A computer program product as described in clause 15 or 16, wherein t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0064] Clause 18. A computer program product as described in any of clauses 15-17, wherein for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of a rejection region of the baseline machine learning model, and wherein the program instructions, when executed by at least one processor, further cause the at least one processor to train the integrated machine learning model based on (d) the multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0065] Clause 19. A computer program product as described in any of Clauses 15-18, wherein the metamodel is used to integrate a set of baseline machine learning models into an integrated machine learning model.

[0066] Clause 20. A computer program product as described in any of Clauses 15-19, wherein the program instructions, when executed by at least one processor, further cause the at least one processor to: (vi) obtain a current sample; (vii) determine whether the current sample is within a global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically process the current sample using an integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically mark the current sample as unable to receive a credible prediction from the integrated classifier.

[0067] These and other features and characteristics of the present disclosure, as well as methods of operation and functions of the related structural elements and combinations of parts and economies of manufacture, will become more apparent when considering the following description and appended claims with reference to the accompanying drawings, all of which form a part of this specification, wherein like reference numerals designate corresponding parts in the various figures. It is to be expressly understood, however, that the drawings are for purposes of illustration and description only and are not intended as a definition of the limits of the disclosed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Additional advantages and details are explained in more detail below with reference to a non-limiting exemplary embodiment shown in the schematic drawings, in which:

[0069] Figure 1 is a schematic diagram of an electronic payment processing network according to some non-limiting embodiments or aspects;

[0070] Figure 2 According to some non-limiting embodiments or aspects Figure 1 A schematic diagram of example components of one or more devices Figure 3A and Figure 3B is a flow chart of a method for integrated learning according to some non-limiting embodiments or aspects;

[0071] Figure 4 Shows the process of a typical dynamic ensemble selection (DES) method in the reasoning phase;

[0072] Figure 5 is a flow chart illustrating the logic behind classifier-based rejection and confidence-based rejection;

[0073] Figure 6 is a diagram showing how to use example scores for binary classification to determine rejection regions;

[0074] Figure 7 is a graph showing the relationship between the basis and the global rejection region;

[0075] Figure 8illustrates example input features for metamodeling according to non-limiting embodiments or aspects of the present disclosure;

[0076] Fig. 9 illustrates a backward baseline machine learning model selection process for ensemble learning according to non-limiting embodiments or aspects of the present disclosure;

[0077] Fig.10 shows an example relationship between ensemble machine learning model performance, global acceptance rate, and the number of baseline machine learning models;

[0078] Fig.11 is a table containing the description of the experimental dataset;

[0079] Fig.12 is a table including the performance of different models on class-imbalanced datasets;

[0080] Fig.13 is a table including the performance of different models on class-balanced data;

[0081] Fig.14 is a table including the performance of different numbers of baseline machine learning models; and

[0082] Fig.15 is a table including contributions of different types of features according to non-limiting embodiments or aspects of the present disclosure. DETAILED DESCRIPTION

[0083] For the purposes of the following description, the terms "end," "upper," "lower," "right," "left," "vertical," "horizontal," "top," "bottom," "lateral," "longitudinal," and their derivatives shall relate to the orientation of the embodiments in the accompanying drawings. However, it shall be understood that the present disclosure may employ various alternative variations and step sequences, unless expressly specified to the contrary. It shall also be understood that the specific devices and processes shown in the drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects of the disclosed subject matter. Accordingly, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein shall not be considered limiting.

[0084] Some non-limiting embodiments or aspects are described herein in conjunction with threshold values. As used herein, satisfying a threshold value may refer to a value greater than a threshold value, more than a threshold value, above a threshold value, greater than or equal to a threshold value, less than a threshold value, less than a threshold value, below a threshold value, less than or equal to a threshold value, equal to a threshold value, etc.

[0085] Aspects, parts, elements, structures, actions, steps, functions, instructions and / or the like used herein should not be understood as critical or necessary unless explicitly described as such. Moreover, as used herein, the article "one" is intended to include one or more items, and can be used interchangeably with "one or more" and "at least one". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items and / or the like), and can be used interchangeably with "one or more" or "at least one". In the case of wishing only one item, the term "one" or similar language is used. Moreover, as used herein, the term "having" and / or the like is intended to be an open term. In addition, unless otherwise explicitly stated, the phrase "based on" is intended to mean "based at least in part". In addition, the reference to the action of "based on" a condition may refer to the action being "in response to" the condition. For example, in some non-limiting embodiments or aspects, the phrases "based on" and "in response to" may refer to the conditions (e.g., specific operations of electronic devices such as computing devices and processors) that automatically trigger an action.

[0086] As used herein, the term "communication" may refer to the reception, acceptance, transmission, transmission, provision, etc. of data (e.g., information, signals, messages, instructions, commands, etc.). A unit (e.g., a device, a system, a component of a device or system, a combination thereof, and / or the like) communicating with another unit means that the unit is able to directly or indirectly receive information from the other unit and / or send information to the other unit. This may refer to a direct or indirect connection (e.g., a direct communication connection, an indirect communication connection, and / or the like) that is wired and / or wireless in nature. In addition, although the information sent may be modified, processed, relayed, and / or routed between the first unit and the second unit, the two units may also communicate with each other. For example, even if the first unit passively receives information and does not actively send information to the second unit, the first unit may communicate with the second unit. As another example, if at least one intermediate unit processes the information received from the first unit and transmits the processed information to the second unit, the first unit may communicate with the second unit. In some non-limiting embodiments or aspects, a message may refer to a network packet (e.g., a data packet and / or the like) comprising data. It will be appreciated that many other arrangements are possible.

[0087] As used herein, the term "computing device" may refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, a display, a memory, an input device, a network interface, and / or the like. A computing device may be a mobile device. As an example, a mobile device may include a cellular phone (e.g., a smartphone or a standard cellular phone), a portable computer, a wearable device (e.g., a watch, glasses, lenses, clothing, etc.), a personal digital assistant (PDA), and / or other similar devices. A computing device may also be a desktop computer or other form of non-mobile computer.

[0088] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communications and processing by multiple parties in a network environment such as the Internet, but it should be understood that communications may be facilitated through one or more public or private network environments, and that various other arrangements are possible. In addition, multiple computing devices (e.g., servers, point of sale (POS) devices, mobile devices, etc.) that communicate directly or indirectly in a network environment may constitute a "system."

[0089] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processors, servers, client devices, software applications, components of such devices, and / or the like). As used herein, references to a "device," "server," "processor," and / or the like may refer to a previously described device, server, or processor that is described as performing a previous step or function, a different device, server, or processor, and / or a combination of devices, servers, and / or processors. For example, as used in the specification and claims, a first device, first server, or first processor that is described as performing a first step or a first function may refer to the same or a different device, server, or processor that is described as performing a second step or a second function.

[0090] As used herein, the term "real-time" refers to performing one or more tasks during or before another process is completed. For example, real-time reasoning can be reasoning obtained from a model before a payment transaction is authorized, completed, etc.

[0091] Ensemble learning combines the advantages of multiple baseline machine learning models and is widely used in classification tasks. Traditional methods consider all baseline machine learning models in the ensemble and use the same structure for the classification of each sample, which is called static ensemble. However, due to the changing data pattern, the appropriate base classifiers for different samples are usually different. Past studies have shown that the selective ensemble process generally provides better prediction performance than static ensemble. One of the most popular series for selective ensemble learning is called dynamic selection (DS). DS takes one or several models based on some competence measures, and only uses the selected classifiers to perform the ensemble, rather than using all baseline machine learning models.

[0092] Many DS methods have been developed in the literature. Early studies aimed to find the best single classifier for each new sample from a candidate pool, which is called dynamic classifier selection (DCS). There are two main limitations of DCS: 1) there may be more than one model that performs well for a given sample, so there is no need to select only one base classifier, and 2) selecting a single model may lead to high local sensitivity, especially when the data is unbalanced or has a skewed feature distribution. Later studies have solved these problems by selecting multiple models with good performance for integration. This method is called dynamic ensemble selection (DES). Different DES methods use fundamentally different algorithms to measure the competence level of each base classifier for a given sample. The competence level usually depends on the accuracy of each model's prediction of the neighbors of the target sample. Once a number of base classifiers are selected based on the measured competence levels, the final prediction is made by aggregating the outputs from these models.

[0093] In past studies, DES methods have shown their advantages in terms of prediction accuracy. However, there are three problems that may limit the application of these methods in practice. First, the time and space complexity of popular DES methods is high, and therefore, it will be challenging to deploy them for large-volume or real-time classification tasks (e.g., real-time payment risk assessment, etc.). The complexity mainly comes from the adjacent sample search step, which requires storing all training / validation data in space and sorting the distance between the target sample and the training / validation data. In addition, it takes extra time to rank the performance of all base classifiers, especially when there are many candidate models. This computational complexity problem has attracted attention in several recent studies. Second, ensemble methods are usually limited to voting or weighted averaging after finding the most competent base classifier. This is because the baseline machine learning model combination changes sample by sample. It is difficult to use more flexible ensemble options such as stacked classifiers on top of the changing baseline machine learning model combination. Third, DES methods usually select competent baseline machine learning models based on their accuracy on certain training or validation samples. However, accuracy is not always a good measure for ranking models, especially when the data is class-imbalanced and the cost of false positive to false negative predictions is different. Some studies (e.g., DES-MI, etc.) have tried to weight or resample different classes when measuring the level of model competence. However, more flexible options are needed that allow easy integration of any popular evaluation metric (e.g., precision-recall, F1 score, etc.) during selective integration.

[0094] Non-limiting embodiments or aspects of the present disclosure provide methods, systems, and computer program products for ensemble learning, the methods, systems, and computer program products: (i) for each baseline machine learning model in a set of baseline machine learning models, using at least one processor: training the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; for the baseline machine learning model, generating a rejection region associated with at least one data type among the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; and using the baseline machine learning model to generate a rejection region associated with at least one data type among the plurality of different data types. The method further comprises: (i) processing a subset of second training samples in a plurality of second training samples outside a rejection region of the baseline machine learning model using a baseline machine learning model to generate a subset of second predictions for the subset of second training samples in a plurality of second training samples outside a rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions for the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each rejection flag in the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within a rejection region of the baseline machine learning model; and (ii) determining a baseline model prediction performance metric for the baseline machine learning model based on a baseline model prediction performance metric for the baseline machine learning model. type to generate a global rejection region associated with one or more of the multiple different data types; (iii) training an ensemble machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of the multiple second training samples outside the global rejection region; (b) a plurality of rejection flags for the multiple second samples associated with each baseline machine learning model; (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein training the ensemble machine learning model generates a subset of ensemble predictions for another subset of the second training samples outside the global rejection region, and And wherein the ensemble model prediction performance metric is determined based on a subset of the ensemble predictions; and (d) multiple distance metrics for multiple second samples are associated with each baseline machine learning model, and a subset of the ensemble predictions is provided as output; (iv) the set of baseline machine learning models is updated based on the baseline model prediction performance metric for each baseline machine learning model; and (v) (i)-(iv) is repeated until there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the ensemble model prediction performance metrics satisfies a threshold ensemble model prediction performance, the ratio of multiple second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

[0095] In this way, the non-limiting embodiments or aspects of the present disclosure provide a new selective ensemble learning method that solves the above-mentioned limitations of the existing DES method. The non-limiting embodiments or aspects of the present disclosure take the concept of "classification with rejection" into consideration in ensemble learning. Classification with rejection was originally proposed to deal with scenarios where incorrect predictions lead to much worse consequences than not making predictions. Such scenarios are quite common in practice (for example, when evaluating transaction risks with high payment amounts, when diagnosing critical illnesses, etc.). The non-limiting embodiments or aspects of the present disclosure define a rejection region for each baseline machine learning model based on the model performance of different data patterns. Any common evaluation metric can be easily adopted in this step, rather than using only accuracy. Each derived rejection region represents a set of data in which the corresponding baseline machine learning model has low confidence. A global rejection region is then developed, in which no baseline machine learning model can provide a credible prediction for samples within the global rejection region. This global rejection region enables the non-limiting embodiments or aspects of the present disclosure to avoid risk predictions on highly uncertain sample patterns. The non-limiting embodiments or aspects of the present disclosure further consider data beyond the global rejection region for ensemble machine learning modeling. Specifically, non-limiting embodiments or aspects of the present disclosure use two types of rejection-related metrics, and build a metamodel on top of these two types of rejection-related metrics to make final predictions. These new metrics capture 1) the rejection state of each baseline machine learning model and 2) the uncertainty in the derivation of the rejection region. The metamodel can be any classifier or any voting / bagging algorithm. In this way, non-limiting embodiments or aspects of the present disclosure enable the integrated machine learning model to learn how to use base classifiers on different data patterns, which avoids the complexity of ranking baseline machine learning models and also avoids limitations in output aggregation.

[0096] Thus, non-limiting embodiments or aspects of the present disclosure (i) enable new selective ensemble methods with rejection options, which significantly reduce the space and time complexity required to make predictions (a major limitation of popular DES methods); (ii) enable any common evaluation metric to be used for baseline machine learning model competence measures, rather than just accuracy used in popular DES methods, which may be particularly useful for situations like imbalanced data classification where the costs of false positives and false negatives are typically different; (iii) develop a global rejection region that indicates whether an ensemble machine learning model is able to make a credible prediction on a given sample, rather than just providing a classification score; (iv) and generate two types of rejection-related metrics for ensemble machine learning modeling that quantify the competence level of each baseline machine learning model for any given sample; and / or (v) provide a meta-model that provides greater flexibility for baseline machine learning model aggregation.

[0097] Reference now Figure 1 , Figure 1 An electronic payment processing network 100 is shown according to a non-limiting embodiment or aspect. The payment processing network can be used in conjunction with the systems and methods described herein. It should be understood that the specific arrangement of the electronic payment processing network 100 shown is for exemplary purposes only, and various arrangements are possible. The transaction processing system 101 (e.g., a transaction processor) is shown as communicating with one or more issuer systems (e.g., such as an issuer system 106) and one or more acquirer systems (e.g., such as an acquirer system 108). Although only a single issuer system 106 and a single acquirer system 108 are shown, it should be understood that the transaction processing system 101 can communicate with multiple issuer systems and / or acquirer systems. In some embodiments, the transaction processing system 101 can also work as an issuer system, so that the transaction processing system 101 and the issuer system 106 are both single systems and / or controlled by a single entity.

[0098] In some non-limiting embodiments or aspects, the transaction processing system 101 can communicate directly with the merchant system 104 through a public or private network connection. Additionally or alternatively, the transaction processing system 101 can communicate with the merchant system 104 through a payment gateway 102 and / or an acquirer system 108. In some non-limiting embodiments or aspects, an acquirer system 1108 associated with the merchant system 104 can operate as a payment gateway 102 to facilitate the transmission of transaction requests from the merchant system 104 to the transaction processing system 101. The merchant system 104 can communicate with the payment gateway 102 through a public or private network connection. For example, a merchant system 104 including a physical POS device can communicate with the payment gateway 102 through a public or private network to conduct card-present transactions. As another example, a merchant system 104 including a server (e.g., a web server) can communicate with the payment gateway 102 through a public or private network such as a public Internet connection to conduct card-not-present transactions.

[0099] In some non-limiting embodiments or aspects, upon receiving a transaction request from a merchant system 104 that identifies an account identifier of a payee (e.g., such as an account holder) associated with an issued payment device 110, the transaction processing system 101 may generate an authorization request message to be transmitted to the issuer system 106 that issued the payment device 110 and / or the account identifier. The issuer system 106 may then approve or deny the authorization request and, based on the approval or denial, generate an authorization response message that is transmitted to the transaction processing system 101. The transaction processing system 101 may transmit the approval or denial to the merchant system 104. When the issuer system 106 approves the authorization request message, it may then clear and settle the payment transaction between the issuer system 106 and the acquirer system 108.

[0100] Figure 1 The number and arrangement of systems and / or devices shown in the FIG. are provided as examples. Figure 1 There may be additional systems and / or devices, fewer systems and / or devices, different systems and / or devices, and / or systems and / or devices arranged in a different manner than the systems and / or devices shown in the drawings. Furthermore, a single system and / or device may be implemented Figure 1 Two or more systems or devices shown in Figure 1 The single system or device shown in the embodiment may be implemented as multiple distributed systems or devices. Additionally or alternatively, a group of systems (e.g., one or more systems) and / or a group of devices (e.g., one or more devices) of system 100 may perform one or more functions described as being performed by another group of systems or another group of devices of system 100.

[0101] Reference now Figure 2 , a diagram of example components of a device 200 according to a non-limiting embodiment is shown. As an example, the device 200 may correspond to a transaction processing system 101, a payment gateway 102, a merchant system 104, an issuer system 106, an acquirer system 108, and / or a consumer device 110. In some non-limiting embodiments, such a system or device may include at least one device 200 and / or at least one component of the device 200. The number and arrangement of the components shown are provided as examples. In some non-limiting embodiments, the device 200 may include additional components, fewer components, different components, or differently arranged components compared to those shown. Additionally or alternatively, a group of components (e.g., one or more components) of the device 200 may perform one or more functions described as being performed by another group of components of the device 200.

[0102] like Figure 2As shown, the device 200 may include a bus 202, a processor 204, a memory 206, a storage component 208, an input component 210, an output component 212, and a communication interface 214. The bus 202 may include components that allow communication between components of the device 200. In some non-limiting embodiments, the processor 204 may be implemented in hardware, firmware, or a combination of hardware and software. For example, the processor 204 may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), etc.), a microprocessor, a digital signal processor (DSP), and / or any processing component that can be programmed to perform a function (e.g., a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), etc.). The memory 206 may include a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by the processor 204.

[0103] Continue to refer Figure 2 , the storage component 208 may store information and / or software related to the operation and use of the device 200. For example, the storage component 208 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, a solid-state disk, etc.) and / or another type of computer-readable medium. The input component 210 may include a component that allows the device 200 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, a button, a switch, a microphone, etc.). In addition or alternatively, the input component 210 may include a sensor for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, an actuator, etc.). The output component 212 may include a component that provides output information from the device 200 (e.g., a display, a speaker, one or more light-emitting diodes (LEDs), etc.). The communication interface 214 may include a transceiver-type component (e.g., a transceiver, a separate receiver and a transmitter, etc.) that enables the device 200 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 214 may permit the device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, Interface, cellular network interface, etc.

[0104] Device 200 can perform one or more processes described herein. Device 200 can perform these processes based on processor 204 executing software instructions stored by computer-readable media such as memory 206 and / or storage component 208. Computer-readable media may include any non-transient memory device. Memory device includes a memory space located in a single physical storage device or a memory space extended across multiple physical storage devices. Software instructions can be read from another computer-readable medium or from another device to memory 206 and / or storage component 208 via communication interface 214. When executed, software instructions stored in memory 206 and / or storage component 208 can enable processor 204 to perform one or more processes described herein. In addition or alternatively, hard-wired circuit system can be used in place of or in conjunction with software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any specific combination of hardware circuit system and software. As used herein, the term "configured to" can refer to the arrangement of software, equipment and / or hardware for performing and / or realizing one or more functions (such as actions, processes, steps of processes and / or the like). For example, "a processor configured to" may refer to a processor executing software instructions (eg, program code) that cause the processor to perform one or more functions.

[0105] Dynamic Ensemble Selection (DES) aims to find the most competent baseline machine learning model from all candidates for each sample to be classified. Figure 4 , Figure 4 The process of a typical DES method in the inference phase is shown. The common DES method starts by defining a local area around the target sample to be predicted (e.g., a neighbor from training or validation data, etc.). Different algorithms are then used to define and rank the competence levels of the base classifiers.

[0106] Consider C = [C1, ..., C M ], as features and labels of samples from the training or validation set (i=1,...,N, where N is the sample size) (x i ,y i ) and as the jth sample S to be predicted test,j The characteristic of x test,j The first step in the common DES method is to find s test,j Methods such as K-nearest oracle-Eliminate (KNORA-E), K-nearest oracle-Union (KNORA-U), and Diversity-based Enhanced KNN Selection (DES-KNN) are based on the similarity of feature values ​​(e.g., x test,j with [x1,...,xN ] to determine neighboring samples. Methods such as K-nearest Output Profiles (KNOP) perform neighbor searches based on the similarity of decisions from a baseline machine learning model. Alternatively, probabilistic methods such as randomized reference classifiers (RRC) and Kullback-Leibler divergence-based selection (DES-KL) use kernel functions (e.g., Gaussian or exponential kernels, etc.) to quantify s test,j The distance between the training or validation samples. In addition to different options for defining neighbors using sample-level metrics, several studies have sought to reduce the computational burden using cluster-level neighbor search (e.g., K-means-based selection and fuzzy hyperbox-based selection). As a tradeoff for the reduced computational cost, past studies have pointed out that the output from cluster-level selection may not be as accurate as sample-level neighbor selection.

[0107] Let s=[s1,...,s K ] represents the neighboring samples selected above. The next step of DES is to use s to measure and rank the competence level of each baseline machine learning model in C. Again, multiple algorithms have been proposed for this step. Probabilistic methods (e.g., RRC, DES-KL, etc.) estimate the performance of the model on s weighted by a predefined kernel function. test,j The probability of making an accurate prediction from each baseline machine learning model on s. KNOP, KNORA-E, KNORA-U, and variants of these methods measure the competence of baseline machine learning models based on the number of accurate predictions made from each model on samples in s. In addition, similarity-based weights can be considered in the accuracy measure when ranking model competence to capture the similarity between each sample in s and the target point S test,j The pattern differences between them. Meta-learning ensemble selection (META-DES) builds a separate learner to rank the competence levels of baseline machine learning models. Features considered in the learner include the accuracy of each baseline machine learning model on its neighbors s and the level of inconsistency of the baseline machine learning model in predicting s. Entropy measures (e.g., cross entropy of predictions from each baseline machine learning model, etc.) are also used to quantify the competence of baseline machine learning models. Entropy measures consider the same contribution of each class and may not be suitable for class-imbalanced ensemble selection.

[0108] After the competency ranking, Expressed as for sample S test,j The K baseline machine learning models selected can aggregate the scores from C by using weighted majority voting (if the output is a hard classification) or weighted average score (if the output is a soft classification). test,j The output of s is used to make an integrated prediction. Taking soft classification as an example,test,j The above prediction from a typical DES method can be formulated according to the following equation (1):

[0109]

[0110] in represents the classification score (e.g., a scalar for binary classification, etc.), and w k is the weight of the baseline machine learning model k, usually based on the competency metric (e.g., higher weights for baseline machine learning models with higher competency rankings, etc.). Special case w k = 1 corresponds to the mean of all base model outputs.

[0111] We now summarize the limitations of existing DES methods that follow the three stages reviewed in this paper. In the neighbor selection stage, existing DES methods need to find test,j The most similar sample point, which requires S test,j The similarity between the training and validation samples is calculated and ranked. When the sample space or feature dimension is large, the computational cost may be huge. In the baseline machine learning model competence ranking stage, additional time cost is required to calculate the similarity between each new sample s test,j The competence levels of baseline machine learning models are measured and ranked. In addition, only accuracy is used to rank the competence levels using adjacent samples, and the resulting model combination for the next step of integration may be suboptimal (for example, especially for imbalanced data classification, etc.). In the output aggregation stage, only voting or weighted averaging is used in existing methods. It is difficult to apply more flexible integration techniques on changing baseline machine learning model combinations. In practice, computational efficiency and flexibility are useful when deploying classification models. Non-limiting embodiments or aspects of the present disclosure provide a more efficient and flexible solution for selective ensemble learning, which includes the concept of classification with rejection introduced herein.

[0112] Rejection is an option to improve the confidence of the developed model. Instead of providing a class or classification score for each sample, a classifier with the rejection option first determines whether the model can provide a credible prediction for the sample. Only when the model has sufficient confidence, a prediction is provided. Rejected samples are those samples that are associated with low model performance and, therefore, can be handled by other methods. There are two methods for deriving the rejection region of a model, namely, classifier-based rejection and confidence-based rejection. Now refer to Figure 5 , Figure 5 is a flow chart showing the logic behind classifier-based rejection and confidence-based rejection. The difference between classifier-based rejection and confidence-based rejection is whether the rejection region is determined during the training process of the classifier or derived using the output from the trained classifier.

[0113] When training a classification model, classifier-based rejection considers "rejection" as possible outputs other than the normal class. A modified loss function that incorporates the rejection loss is used to train such a classifier. The training process learns what samples the model should reject so that the total loss across all considered samples is minimized. For any training sample s n , the general loss L can be defined according to equation (2) n :

[0114]

[0115] where q n are hard classifications from a given model; l T is the loss for correct predictions, usually set to 0; l F is the loss for incorrect predictions, usually set to a positive value; and l R Is based on the decision to reject n The loss satisfies l T <l R <l F , so that the cost of making an incorrect prediction is higher than the cost of not making a prediction. There is a general loss L described in the paper n There are several variants of . For example, a cost-sensitive loss is one where rejection of different classes (e.g., minority vs. majority class for an imbalanced dataset, etc.) is associated with different costs. The advantage of classifier-based rejection is that both class predictions and rejection / acceptance decisions can be obtained within one learning process. However, the loss l can be carefully designed and tuned. R , because some metrics (e.g., rejection rate, model performance, etc.) may have an impact on the loss l R The value is sensitive.

[0116] The confidence-based rejection method trains the classifier without considering any rejection-related losses. The rejection region is determined by the rejector based on the classification score, which reflects the confidence of the classifier for a given sample. The general decision process for the rejector can be defined according to the following equation (3):

[0117]

[0118] where R is the range of scores to reject from making a prediction (i.e., the rejection region); and I(P n ) is the fraction P n The rejection region R can be defined based on the prediction performance of the classifier.

[0119] Reference now Figure 6 , Figure 6is a graph showing how to use example scores for binary classification to determine the rejection region R. Figure 6 As shown, the classifier has low confidence for medium score samples and high confidence for low score samples or high score samples. The prediction performance of the low confidence region of the classifier is much lower than that of the high confidence region. The rejector can reject samples with scores between a certain range based on the change in prediction performance after rejecting these samples and the rejection rate. For example, the rejection region [t lb ,t ub ]:

[0120]

[0121] where t lb <t ub ∈ left <τ left ∈ right <τ right (4)

[0122] where ∈ left and ∈ right Respectively represent scores less than t lb or higher than t ub The inaccuracy rate of the sample; τ left and τ right The corresponding is the tolerable inaccuracy. Given the baseline accuracy after rejection, equation (4) finds the minimum score range to be rejected. Several other options for defining the rejection region by similar goals (e.g., achieving a promised prediction performance after rejecting a small number of uncertain samples, etc.) can be found in the existing literature. Note that the model confidence discussed in this section is different from the confidence of the classification score or probability. For the latter case, measures such as confidence intervals can be used to capture the uncertainty in the decision. However, the rejection region is derived using a point estimate of the classification score.

[0123] Reference now Figure 3A and Figure 3B , a flow chart of a method 300 for integrated learning is shown according to some non-limiting embodiments or aspects. The steps shown in FIG. 3 are for example purposes only. It will be appreciated that in some non-limiting embodiments or aspects, additional, fewer, different and / or different order of steps may be used. In some non-limiting embodiments or aspects, steps may be automatically performed in response to the execution and / or completion of previous steps.

[0124] like Figure 3AAs shown, at step 302, method 300 includes, for each baseline machine learning model in a set of baseline machine learning models, training the baseline machine learning model based on multiple first training samples. For example, the transaction processing system 101 may train each baseline machine learning model in a set of baseline machine learning models based on multiple first training samples. The multiple first training samples may include multiple different data types. Training the baseline machine learning model may generate multiple first predictions for the multiple first training samples. For example, each baseline machine learning model may be trained using machine learning or other artificial intelligence techniques to accept multiple first training samples (e.g., feature representations associated with the multiple first training samples, etc.) as input and provide multiple first predictions for the multiple first training samples as output.

[0125] In some non-limiting embodiments or aspects, the plurality of first training samples may include a plurality of payment transactions, such as payment transactions configured to be processed in the electronic payment processing network 100. As an example, the payment transactions may include transaction parameters and / or features associated with the payment transactions.

[0126] Transaction parameters and / or features (e.g., categorical features, numerical features, local features, graphical features or embeddings, etc.) associated with a payment transaction may include transaction parameters of the transaction, features determined based thereon (e.g., using feature engineering, etc.), and / or the like, such as an account identifier (e.g., PAN, etc.), transaction amount, transaction date and / or time, type of product and / or service associated with the transaction, currency conversion rate, type of currency, merchant type, merchant name, merchant location, and / or the like. However, non-limiting embodiments or aspects are not limited thereto, and transaction parameters and / or features of a transaction may include any data, including any type of parameters associated with any type of transaction.

[0127] like Figure 3A As shown, at step 304, method 300 includes, for each baseline machine learning model in a set of baseline machine learning models, generating, for the baseline machine learning model, a rejection region associated with at least one data type in a plurality of different data types based on a plurality of first predictions for a plurality of first training samples. For example, the transaction processing system 101 may generate, for each baseline machine learning model in a set of baseline machine learning models, a rejection region associated with at least one data type in a plurality of different data types based on a plurality of first predictions for a plurality of first training samples.

[0128] Non-limiting embodiments or aspects of the present disclosure provide a selectively rejected ensemble learning method for binary classification. Non-limiting embodiments or aspects of the present disclosure can define a rejection region for each base classifier, generate a global rejection region, and incorporate the rejection / acceptance state into the ensemble machine learning modeling.

[0129] For each baseline machine learning model in the set of baseline machine learning models, the transaction processing system 101 may generate, for the baseline machine learning model, a rejection region associated with at least one of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples by optimizing an objective function defined according to the following equation (5) and any other constraints of interest:

[0130]

[0131] Where 0<t lb <1; N rej / N=a%; (5)

[0132] where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of multiple samples outside the rejection region (e.g., a model performance evaluation metric of interest calculated using data within the acceptance region, etc.); N rej is the number of first samples that are rejected due to the score being in the rejection region (e.g., the number of rejected samples, etc.); N is the total number of first samples; and a% is a user-defined rejection rate limit for the baseline machine learning model.

[0133] There can be only one parameter t lb To calculate, given t lb A certain value and a rejection rate a%, the upper limit of the rejection area t ub can be fixed. Therefore, t can be easily obtained using a search or optimization algorithm. lb As pointed out in prior studies on classification with rejection, the post-rejection performance (i.e., T acc ) is usually improved by a%. This is because the more uncertain samples are removed from the decision, the higher the confidence the classifier has in the remaining samples (and therefore the better the performance). This hyperparameter a% is related to both the performance of the integrated machine learning model and the global rejection rate introduced herein. Non-limiting embodiments or aspects of the present disclosure can test different values ​​of a% and determine a% by examining how the key metrics change due to different choices. Popular hyperparameter tuning methods can also be applied.

[0134] In some non-limiting embodiments or aspects, tlb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

[0135] Once t is obtained for each base classifier lb The optimal value of , the non-limiting embodiments or aspects of the present disclosure can obtain the rejection region of each classifier. Unlike the traditional classification method with rejection that stops at this step, the non-limiting embodiments or aspects of the present disclosure move to define global rejection and acceptance regions as described herein, which can be used for integrated machine learning modeling.

[0136] like Figure 3A As shown, in step 306, method 300 includes, for each baseline machine learning model in a set of baseline machine learning models, using the baseline machine learning model to process a subset of second training samples in a plurality of second training samples outside a rejection region of the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model. For example, the transaction processing system 101 may, for each baseline machine learning model in a set of baseline machine learning models, use the baseline machine learning model to process a subset of second training samples in a plurality of second training samples outside a rejection region of the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model. A baseline model prediction performance metric for the baseline machine learning model may be determined based on the subset of second predictions for the baseline machine learning model. The plurality of second training samples may be associated with a plurality of rejection flags for the baseline machine learning model. Each of the plurality of rejection flags may indicate whether a corresponding second sample in the plurality of second samples is within a rejection region of the baseline machine learning model.

[0137] In some non-limiting embodiments or aspects, the plurality of second training samples may include a plurality of payment transactions, such as payment transactions configured to be processed in the electronic payment processing network 100. As an example, the payment transactions may include transaction parameters and / or features associated with the payment transactions.

[0138] like Figure 3A As shown, at step 308, method 300 includes generating a global rejection region associated with one or more of the plurality of different data types based on the rejection region associated with each baseline machine learning model. For example, transaction processing system 101 may generate a global rejection region associated with one or more of the plurality of different data types based on the rejection region associated with each baseline machine learning model.

[0139] R1, ..., R M The non-limiting embodiments or aspects of the present disclosure represent the rejection regions of the M base classifiers, and the global rejection region R G It is defined as the intersection of the rejection regions of all baseline machine learning models. G Indicates the type of data that no baseline machine learning model will include in its decision making. G , then the baseline machine learning models have low confidence in making predictions about it. As a result, such samples can be processed by other methods to avoid any biased performance metrics. Conversely, if the sample falls into R G The complementary region A G , then there is at least one baseline machine learning model that considers this sample in its decision. It is therefore called the global acceptance region.

[0140] Compared to the rejection region from a base classifier, R G Returns a significantly reduced rejection rate (e.g., equivalent to an increased global acceptance rate, etc.). Now refer to Figure 7 , Figure 7 is a graph showing the relationship between the base and global rejection regions using three base classifiers as examples, including R G The relationship between the rejection region and the baseline machine learning model. Non-limiting embodiments or aspects of the present disclosure provide an intelligent way to integrate baseline machine learning models taking into account separate rejection / acceptance states so that the final model can have much higher coverage while also having a global confidence measure (i.e., A) for the samples to be predicted in addition to the assigned classification scores. G and R G ).

[0141] like Figure 3AAs shown, at step 310, method 300 includes training an integrated machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples in the second training samples in the plurality of second training samples outside the global rejection region, (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model, and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model. For example, the transaction processing system 101 may train an integrated machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples in the second training samples in the plurality of second training samples outside the global rejection region, (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model, and (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model. Training the integrated machine learning model may generate a subset of integrated predictions for another subset of second training samples in the plurality of second training samples outside the global rejection region. An integrated machine learning model prediction performance metric may be determined based on the subset of integrated predictions. For example, an ensemble machine learning model can be trained using machine learning or other artificial intelligence techniques to accept as input a) another subset of second training samples from a plurality of second training samples outside the global rejection region, (b) a plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model, (c) a subset of second predictions for the subset of second training samples generated for each baseline machine learning model, and / or (d) a plurality of distance measures for the plurality of second samples associated with each baseline machine learning model, and provide as output a subset of ensemble predictions.

[0142] In some non-limiting embodiments or aspects, for each baseline machine learning model in a set of baseline machine learning models, multiple second training samples are associated with multiple distance metrics, and each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model. In such an example, the transaction processing system 101 can further train the integrated machine learning model based on (d) multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

[0143] As described herein, the problem with existing DES methods is the lack of flexibility in aggregating baseline machine learning model outputs. To solve this problem, non-limiting embodiments or aspects of the present disclosure may use a meta-model for integration when considering rejection-based features as input, and non-limiting embodiments or aspects of the present disclosure provide two types of new features to capture the rejection status of each base learner, including: 1) rejection flag and 2) distance to the rejection boundary. For any sample, the rejection flag feature can be defined according to the following equation (6):

[0144]

[0145] Where m=1, ..., M (i.e., one flag for each baseline machine learning model). This feature indicates whether the sample is within the local rejection region of the candidate model being considered for the ensemble. For each baseline machine learning model, this is a hard boundary that separates the sample into two clusters. In order to capture uncertainty in the separation (e.g., samples outside the rejection region but very close to the boundary may be associated with low prediction confidence), non-limiting embodiments or aspects of the present disclosure may provide another type of feature as defined according to the following equation (7):

[0146]

[0147] The feature d m is a distance measure that shows how far a sample is towards the rejection boundary in the space of classification scores.

[0148] Using a metamodel as an ensemble approach enables non-limiting embodiments or aspects of the present disclosure to include different kinds of features in the ensemble learning process (which is not available in most DES methods). Figure 8 Example input features for meta-modeling according to non-limiting embodiments or aspects of the present disclosure are shown. In addition to the two types of rejection-related features introduced herein, non-limiting embodiments or aspects of the present disclosure may also consider base classification scores and / or original features. Note that feature selection can be easily performed before and during integrated machine learning model training, which is the same as developing any traditional classification model. Therefore, non-limiting embodiments or aspects of the present disclosure can always start with a set of comprehensive features for integrated meta-modeling.

[0149] like Figure 3AAs shown, at step 312, method 300 includes updating a set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model. For example, the transaction processing system 101 can update a set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model. As an example, the transaction processing system 101 can remove the baseline machine learning model with the lowest or worst baseline model prediction performance metric from the set of baseline machine learning models.

[0150] Non-limiting embodiments or aspects of the present disclosure may determine which and how many baseline machine learning models to consider for ensemble learning. Like the baseline machine learning model rejection rate α%, there may be two useful metrics depending on this determination, including ensemble machine learning model performance and global rejection or acceptance rate. Non-limiting embodiments or aspects of the present disclosure may use a stepwise backward (or forward) baseline machine learning model selection process to optimize candidates for ensemble. Now refer to Fig. 9 , Fig. 9 The backward baseline machine learning model selection process for ensemble learning according to non-limiting embodiments or aspects of the present disclosure is shown. Fig. 9 As shown, starting with the use of all M candidate base classifiers, non-limiting embodiments or aspects of the present disclosure can train an integrated machine learning model and measure the prediction performance and global acceptance rate. Non-limiting embodiments or aspects of the present disclosure can remove the baseline machine learning model with the worst performance after rejection, use the remaining base classifiers to train the integrated machine learning model, and measure the same metrics. Non-limiting embodiments or aspects of the present disclosure can repeat the process until 1) only 1 baseline machine learning model remains, or 2) the prediction performance or global acceptance rate is below a tolerance or threshold level. Alternatively, non-limiting embodiments or aspects of the present disclosure can perform forward selection in a similar manner, but continue to add baseline machine learning models from 1 to M.

[0151] like Figure 3BAs shown, in step 314, method 300 includes determining whether there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the integrated machine learning model prediction performance metrics in the integrated machine learning model prediction performance metric satisfies the threshold integrated machine learning model prediction performance, the ratio of multiple second training samples outside the global rejection region satisfies the threshold ratio, or any combination thereof. For example, the transaction processing system 101 may determine whether there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the integrated machine learning model prediction performance metrics in the integrated machine learning model prediction performance metric satisfies the threshold integrated machine learning model prediction performance, the ratio of multiple second training samples outside the global rejection region satisfies the threshold ratio, or any combination thereof. As an example, the transaction processing system 101 may return processing to step 302 of method 300 in response to determining that there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the integrated machine learning model prediction performance metrics in the integrated machine learning model prediction performance metric satisfies the threshold integrated machine learning model prediction performance, the ratio of multiple second training samples outside the global rejection region satisfies the threshold ratio, or any combination thereof. As an example, the transaction processing system 101 may proceed to step 316 of method 300 in response to determining that there is no single baseline machine learning model in the set of baseline machine learning models or at least one ensemble machine learning model prediction performance metric in the ensemble machine learning model prediction performance metric that satisfies the threshold ensemble machine learning model prediction performance, the ratio of multiple second training samples outside the global rejection region satisfies the threshold ratio, or any combination thereof. In such an example, the transaction processing system 101 may repeat steps 302 to 312 of method 300 until there is a single baseline machine learning model in the set of baseline machine learning models or at least one ensemble machine learning model prediction performance metric in the ensemble machine learning model prediction performance metric that satisfies the threshold ensemble machine learning model prediction performance, the ratio of multiple second training samples outside the global rejection region satisfies the threshold ratio, or any combination thereof.

[0152] Fig.10 An example relationship between ensemble machine learning model performance, global acceptance rate, and the number of baseline machine learning models is shown. This relationship can be used to determine the final number of base classifiers. Given a tolerance level for performance and acceptance rate, Fig.10 The shaded area in can indicate what and how many baseline machine learning models should be used. If multiple options are available, a decision can be made by prioritizing performance or model coverage or considering more metrics. The performance of the developed meta-model can be evaluated in the same way as for any popular classification model.

[0153] like Figure 3BAs shown, at step 316, method 300 includes obtaining a current sample. For example, transaction processing system 101 may obtain a current sample. As an example, the current sample may include a current transaction. For example, transaction processing system 101 may receive a current payment transaction currently being processed in electronic payment processing network 100. As an example, transaction processing system 101 may receive transaction parameters and / or features associated with the current payment transaction.

[0154] like Figure 3B As shown, at step 318, method 300 includes determining whether the current sample is within the global rejection region. For example, transaction processing system 101 may determine whether the current sample is within the global rejection region. As an example, in response to determining that the current sample is outside the global rejection region, transaction processing system 101 may proceed to step 320 of method 300. As an example, in response to determining that the current sample is within the global rejection region, transaction processing system 101 may proceed to step 322 of method 300.

[0155] like Figure 3B As shown, at step 320, method 300 includes, in response to determining that the current sample is outside the global rejection zone, automatically processing the current sample using the integrated machine learning model to generate a current prediction for the current sample. For example, the transaction processing system 101 may automatically process the current sample using the integrated machine learning model to generate a current prediction for the current sample in response to determining that the current sample is outside the global rejection zone. As an example, the transaction processing system 101 may automatically process the current payment transaction to generate an authorization or rejection of the current payment transaction in the payment processing network 100 in response to determining that the current payment transaction is outside the global rejection zone.

[0156] like Figure 3B As shown, in step 322, method 300 includes automatically marking the current sample as unable to receive a trusted prediction from the integrated classifier in response to determining that the current sample is within the global rejection region. For example, the transaction processing system 101 may automatically mark the current sample as unable to receive a trusted prediction from the integrated classifier in response to determining that the current sample is within the global rejection region. As an example, the transaction processing system 101 may automatically mark the current payment transaction as unable to receive a trusted prediction from the integrated classifier in response to determining that the current payment transaction is within the global rejection region. In such an example, the transaction processing system 101 may automatically process the current payment transaction using a machine learning model different from the integrated machine learning model in response to marking the current payment transaction to generate an authorization or rejection of the current payment transaction in the payment processing network 100.

[0157] experiment

[0158] This section uses a number of datasets to compare non-limiting embodiments or aspects of the present disclosure against the DES method. Experiments on non-limiting embodiments or aspects of the present disclosure apply different methods in modeling twelve binary classification datasets from OpenML and UCI machine learning repositories, six of which are highly unbalanced (minority class ratios between 0.002 and 0.087), while the other six datasets are more balanced. Reference is now made to the table that includes a description of the experimental datasets. Fig.11 , Fig.11 The table lists the basic information of the dataset, including the source, sample size and feature dimension, minority class ratio and definition of each class. For the activity monitor and MNIST datasets, PCA is first performed to reduce the feature size, and the returned PCA features account for 80% of the variability of the original input. The PCA operation is to speed up the modeling process, which does not affect the non-limiting embodiment or the way the DES method of the present disclosure or any comparison works.

[0159] For each experiment, the data was randomly split into three groups: 50% for training the base classifier, 25% for training the integrated machine learning model, and 25% for testing. Different samples are used to train the base classifier and the integrated machine learning model to avoid overfitting. Next, six different base classifiers were trained, including two random forest models (RF1 and RF2), two gradient boosting models (GBM1 and GBM2), and two multilayer perceptron models (MLP1 and MLP2). For each algorithm (i.e., random forest, gradient boosting, and multilayer perceptron), the corresponding two models were built using different parts of the base classifier training samples. Specifically, the shuffled training data was shuffled, and the first 60% samples were used to train RF1, GBM1, and MLP1, and the last 60% samples were used to train RF2, GBM2, and MLP2. This portion of the sample (i.e., 60% each) was found to be large enough to provide stable results for the experiment. Note that other sampling strategies can also be applied. The purpose of using 60% random samples is to enhance the sample-level diversity of the developed base learner.

[0160] In order to perform selective ensemble machine learning modeling, the rejection region of each base classifier is first derived. As discussed herein, it is possible to consider the global acceptance / rejection rate and the ensemble machine learning model performance a% to tune the baseline machine learning model rejection rate a%. It is possible to consider a%=10% to use the best value searched by equation (5). Several different options (i.e., [2%, 5%, 10%, 15% and 20%]) can be tested, and this a%=10% provides a good balance between performance and acceptance rate for different experimental data sets. The best algorithm among random forests, gradient boosting and multilayer perceptrons is then used as a metamodel for each experiment. The input variables for metamodel training include original features, base classification scores and derived rejection-related features. Note that some features may not be important in the initial metamodel, and the final feature list is tuned based on the feature importance score. In addition, backward selection is performed to check how metamodel performance and global acceptance / rejection rate vary with different base classifier combinations. Regarding the performance evaluation of the baseline machine learning model, the area under the precision-recall curve (PR-AUC) is considered for the 6 imbalanced datasets, and the area under the ROC curve (AUC) is considered for the 6 balanced datasets.

[0161] For comparison purposes, eight different DES models were constructed and their performance on the test samples was measured in each experiment. The DES models considered here include: DES-KNN, KNOP, DESP, DESMI, KNORA-U, KNORA-E, META-DES and RRC. The key hyperparameters of these models (e.g., the number of neighbors for competency measures, etc.) are tuned using the same evaluation criteria as for the non-limiting embodiments of the present disclosure.

[0162] The prediction performance and inference time of different models are compared. For the six imbalanced datasets, the PR-AUC and the best F1 score from each model are calculated. For the balanced dataset, the AUC and the highest accuracy are calculated. The inference time is measured on a machine with a 2.3GHz 8-core processor and 32GB of memory. Fig.12 and Fig.13 , Fig.12 is a table including the performance of different models on class-imbalanced datasets, Fig.13 is a table showing the performance of different models on class-balanced data, Fig.12 Table and Fig.13The table of lists these metrics from different models for unbalanced and balanced datasets respectively (the best values ​​are in bold). Both the mean and standard deviation of the metrics are provided. With respect to non-limiting embodiments or the present disclosure, the metrics are provided for the data in the acceptance region (i.e., Proposed: Acp) and for the complete test data (i.e., Proposed: All), which enables examination of how the performance changes after the introduction of the global rejection / acceptance metric, which is a constituent element of the non-limiting embodiments or the present disclosure. Since the results of the non-limiting embodiments or the present disclosure vary by the number of candidate base classifiers (equivalent to the global rejection / acceptance rate), Fig.11 and Fig.12 The table in shows the output when all six base learners are included (except for the Water Quality dataset where only the first four models are selected as MLP1 and MLP2, which show much worse performance than the others). The global rejection rate returned varies between 0 and 0.04 across different datasets, but is below 0.01 in most cases. To verify that the rejection decision is reasonable, Fig.11 and Fig.12 The tables in also show the best performance from the base classifier on the rejected samples (bottom row in each table).

[0163] from Fig.11 and Fig.12 There are four noteworthy findings in the results in the table. First, the inference time of the non-limiting embodiments or aspects of the present disclosure is much lower than the inference time required by different DES methods. For example, the average inference time for the Land Cover dataset using the current DES method varies between 57 seconds and 73 seconds, while the average inference time using the non-limiting embodiments or aspects of the present disclosure is only about 1.3 seconds. In practice, the sample size or feature dimensionality may be much higher than the number tested here. The reduced computational cost in those scenarios may therefore be more significant. Second, the performance of the base classifier on the rejected samples ( Fig.11 and Fig.12The bottom row in each table of ) is much worse than the performance of different models on the accepted samples. This verifies that the judgment of global rejection / acceptance is reasonable. This judgment helps users avoid the cost of misleading predictions on highly uncertain samples. Third, the non-limiting embodiments or aspects of the present disclosure show the promised prediction performance for all data sets (both class-imbalanced data sets and class-balanced data sets). Even if the non-limiting embodiments or aspects of the present disclosure are applied on the complete test data (i.e., Proposed:All) including rejected samples, the performance is still comparable to 8 DES methods, but the cost is much lower. If we consider the properties of the non-limiting embodiments or aspects of the present disclosure and only apply the model on the accepted samples (i.e., Proposed:Acp), the performance is much better in almost all experiments. Finally, the improvement in performance from Proposed:Acp is more significant in class-imbalanced experiments. One reason is that the non-limiting embodiments or aspects of the present disclosure can use PR-AUC to define local rejection for six imbalanced data sets, but the compared DES methods can only use accuracy-related metrics in the ranking of baseline machine learning models.

[0164] The output of a non-limiting embodiment or aspect of the present disclosure is compared with a different number of baseline machine learning models. Reference is now made to a table including the performance of different numbers of baseline machine learning models. Fig.14 , taking the PaymentRisk dataset as an example, Fig.14 The table shows how different metrics vary with the number of base learners used for the ensemble. As a comparison, the best metric values ​​from eight DES models are listed. First, the inference time of all different options in the non-limiting embodiments or aspects of the present disclosure is four to five times lower than the best number from the compared DES model (i.e., DESMI). The PR-AUC and F1 scores are better than the highest values ​​from the considered DES models even at the minimum rejection rate (i.e., 0.001). When comparing the performance of non-limiting embodiments or aspects of the present disclosure with different options, the performance generally decreases as the global rejection rate decreases.

[0165] Similar findings were obtained in other experiments, in particular, the performance decreases as the rejection rate decreases. This decrease is mainly due to the fact that more relatively low-confidence samples are considered in the decision process when more base classifiers are included in the meta-modeling. The performance of the meta-model on samples accepted by only one or a few base classifiers may be lower than those accepted by all base classifiers. However, note that the performance variation also depends on many other factors, such as the diversity and quality of the base classifiers, the label distribution of the rejected samples, etc.

[0166] Reference is now made to a table of contributions including different types of features according to non-limiting embodiments or aspects of the present disclosure. Fig.15 , Fig.15 The contributions of three types of features (i.e., original features, base classification scores, and rejection-related metrics) in a non-limiting embodiment or aspect of the present disclosure are shown. The contributions are represented by feature importance scores obtained from a permutation-based approach. Fig.15 It is shown that it is important to reject the correlation measure in most experiments. At the same time, the contribution of the three types of features varies drastically across different datasets. For example, the original features contribute the most to the Payment Risk ensemble machine learning modeling, but contribute almost nothing to the Network Attack modeling, which suggests that it is important to consider various features as ensemble inputs at the beginning. This flexibility is not available in the reviewed DES methods that only work on the base classification scores.

[0167] Therefore, the non-limiting embodiments or aspects of the present disclosure provide an integrated learning method with selective rejection for binary classification. Compared with existing methods, the non-limiting embodiments or aspects of the present disclosure are more efficient in calculation because the non-limiting embodiments or aspects of the present disclosure avoid the space and time complexity when defining neighbors for each sample and ranking the base learner. By introducing a global rejection region, the non-limiting embodiments or aspects of the present disclosure reduce the risk of making wrong predictions on highly untrustworthy data patterns. Using a metamodel for integration, the non-limiting embodiments or aspects of the present disclosure provide higher flexibility for baseline machine learning model aggregation and enable a compromise between model performance and coverage. Experiments show that the non-limiting embodiments or aspects of the present disclosure significantly reduce the reasoning time while providing a promising result and a reasonable rejection judgment. The non-limiting embodiments or aspects of the present disclosure can be extended to multi-class modeling, which can include the derivation of the rejection region of the baseline machine learning model and the optimization of more boundary parameters. In the reasoning stage, there is no additional cost compared to binary classification, which implies that even for multi-class tasks, the computational cost may be much lower than the computational cost from existing methods. For example, each baseline machine learning model in the set of baseline machine learning models may include a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and where the rejection region of each baseline machine learning model includes q–1 bounds defining the rejection region.

[0168] The described aspects include artificial intelligence or other operations, whereby the system uses apparent intelligence to process inputs and generate outputs. Artificial intelligence can be implemented in whole or in part by a model. The model can be implemented as a machine learning model. Learning can be supervised learning, unsupervised learning, reinforcement learning, or hybrid learning, whereby a variety of learning techniques are adopted to generate the model. Learning can be performed as part of training. Training the model can include obtaining a set of training data and adjusting the characteristics of the model to obtain the desired model output. For example, three characteristics can be associated with a desired project location. In this case, training can include receiving three characteristics as inputs to the model and adjusting the characteristics of the model so that for each set of three characteristics, the output device state matches the desired device state associated with the historical data.

[0169] In some embodiments, training can be dynamic. For example, the system can use a set of events to update the model. Detectable properties from the events can be used to adjust the model.

[0170] The model can be an equation, an artificial neural network, a recursive neural network, a convolutional neural network, a decision tree, or other machine-readable artificial intelligence structure. The characteristics of the structure that can be used to adjust during training can vary based on the selected model. For example, if a neural network is the selected model, the characteristics can include input elements, network layers, node density, node activation thresholds, weights between nodes, input or output value weights, etc. If the model is implemented as an equation (e.g., regression), the characteristics can include weights for input parameters, thresholds or limits for evaluating output values, or criteria for selecting from a set of equations.

[0171] Once the model is trained, retraining can be included to refine or update the model to reflect additional data or specific operating conditions. Retraining can be based on one or more signals detected by the apparatus described herein or as part of the methods described herein. Upon detection of the indicated signal, the system can activate the training process to adjust the model as described.

[0172] Further examples of machine learning and modeling features that may be included in the embodiments discussed above are described in “Asurvey of machine learning for big data processing” by Qiu et al., EURASIP Journal on Advances in Signal Processing (2016), which is hereby incorporated by reference in its entirety.

[0173] Although the embodiments have been described in detail for the purpose of illustration, it should be understood that such details are used only for that purpose and that the present disclosure is not limited to the disclosed embodiments or aspects, but on the contrary, is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that the present disclosure contemplates that, to the extent possible, one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A method comprising: (i) for each baseline machine learning model in a set of baseline machine learning models, utilizing at least one processor: training the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; generating, for the baseline machine learning model, a rejection region associated with at least one of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; as well as processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions of the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each rejection flag of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model; (ii) generating, with the at least one processor, a global rejection region associated with one or more of the plurality of different data types based on the rejection region associated with each baseline machine learning model; (iii) training, with the at least one processor, an ensemble machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of the second training samples in the plurality of second training samples outside the global rejection region, (b) the plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model, and (c) the subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein training the ensemble machine learning model generates a subset of ensemble predictions for the another subset of the second training samples in the plurality of second training samples outside the global rejection region, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating, with the at least one processor, the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; as well as (v) using the at least one processor to repeat (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the integrated model prediction performance metrics satisfies a threshold integrated model prediction performance, the ratio of the multiple second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

2. The method of claim 1 , wherein for each baseline machine learning model in the set of baseline machine learning models, generating, for the baseline machine learning model, the rejection region associated with the at least one data type in the plurality of different data types based on the plurality of first predictions for the plurality of first training samples comprises optimizing according to the following equation: Where 0 <t lb <1; and the objective function defined by any other constraints of interest; where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of the plurality of samples outside the rejection region; N rej is the number of the plurality of first samples that are rejected due to scores within the rejection region; N is the total number of the plurality of first samples; and a % is a user-defined rejection rate limit for the baseline machine learning model.

3. The method of claim 2, wherein t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

4. A method as claimed in claim 1, wherein for each baseline machine learning model in the set of baseline machine learning models, the multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model, and wherein training the integrated machine learning model using the at least one processor is further based on (d) the multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

5. A method as claimed in claim 1, wherein each baseline machine learning model in the set of baseline machine learning models comprises a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and wherein the rejection region of each baseline machine learning model comprises q–1 bounds defining the rejection region.

6. The method of claim 1, wherein a metamodel is used to integrate the set of baseline machine learning models into the integrated machine learning model.

7. The method of claim 1, further comprising: (vi) obtaining, using the at least one processor, a current sample; (vii) determining, using the at least one processor, whether the current sample is within the global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically processing the current sample using the integrated machine learning model with the at least one processor to generate a current prediction for the current sample; as well as (ix) in response to determining that the current sample is within the global rejection region, automatically marking, using the at least one processor, the current sample as being unable to receive a credible prediction from the ensemble classifier.

8. A system comprising: at least one processor configured to: (i) For each baseline machine learning model in a set of baseline machine learning models: training the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; generating, for the baseline machine learning model, a rejection region associated with at least one of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; as well as processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions of the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each rejection flag of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model; (ii) generating a global rejection region associated with one or more of the plurality of different data types based on the rejection regions associated with each baseline machine learning model; (iii) training an ensemble machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of second training samples in the plurality of second training samples outside the global rejection region, (b) the plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model, and (c) the subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein training the ensemble machine learning model generates a subset of ensemble predictions for the another subset of second training samples in the plurality of second training samples outside the global rejection region, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) Repeat (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the ensemble model prediction performance metrics satisfies a threshold ensemble model prediction performance, the ratio of the multiple second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

9. The system of claim 8, wherein the at least one processor is configured to, for each baseline machine learning model in the set of baseline machine learning models, generate, for the baseline machine learning model, the rejection region associated with the at least one data type in the plurality of different data types based on the plurality of first predictions for the plurality of first training samples by optimizing an objective function defined according to the following equation and any other constraints of interest: Where 0 <t lb <1; where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of the plurality of samples outside the rejection region; N rej is the number of the plurality of first samples that are rejected due to scores within the rejection region; N is the total number of the plurality of first samples; and a % is a user-defined rejection rate limit for the baseline machine learning model.

10. The system of claim 9, wherein t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

11. A system as claimed in claim 8, wherein for each baseline machine learning model in the set of baseline machine learning models, the multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model, and wherein the at least one processor is further configured to train the integrated machine learning model based on (d) the multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

12. The system of claim 8, wherein each baseline machine learning model in the set of baseline machine learning models comprises a multi-class classification model for predicting one class among many classes q, where q is more than two classes, and wherein the rejection region of each baseline machine learning model comprises q–1 bounds defining the rejection region.

13. The system of claim 8, wherein a metamodel is used to integrate the set of baseline machine learning models into the integrated machine learning model.

14. The system of claim 8, wherein the at least one processor is further configured to: (vi) obtaining the current sample; (vii) determining whether the current sample is within the global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically processing the current sample using the integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically marking the current sample as unable to receive a credible prediction from the ensemble classifier.

15. A computer program product comprising at least one non-transitory computer readable medium, the at least one non-transitory computer readable medium comprising program instructions that, when executed by at least one processor, cause the at least one processor to: (i) For each baseline machine learning model in a set of baseline machine learning models: training the baseline machine learning model based on a plurality of first training samples, wherein the plurality of first training samples include a plurality of different data types, and wherein training the baseline machine learning model generates a plurality of first predictions for the plurality of first training samples; generating, for the baseline machine learning model, a rejection region associated with at least one of the plurality of different data types based on the plurality of first predictions for the plurality of first training samples; as well as processing a subset of second training samples in a plurality of second training samples outside the rejection region of the baseline machine learning model using the baseline machine learning model to generate a subset of second predictions for the subset of second training samples in the plurality of second training samples outside the rejection region of the baseline machine learning model, wherein a baseline model prediction performance metric for the baseline machine learning model is determined based on the subset of second predictions of the baseline machine learning model, and wherein the plurality of second training samples are associated with a plurality of rejection flags for the baseline machine learning model, wherein each rejection flag of the plurality of rejection flags indicates whether a corresponding second sample in the plurality of second samples is within the rejection region of the baseline machine learning model; (ii) generating a global rejection region associated with one or more of the plurality of different data types based on the rejection regions associated with each baseline machine learning model; (iii) training an ensemble machine learning model integrated based on the set of baseline machine learning models based on: (a) another subset of the second training samples in the plurality of second training samples outside the global rejection region, (b) the plurality of rejection flags for the plurality of second samples associated with each baseline machine learning model, and (c) the subset of second predictions for the subset of second training samples generated for each baseline machine learning model, wherein training the ensemble machine learning model generates a subset of ensemble predictions for the another subset of the second training samples in the plurality of second training samples outside the global rejection region, and wherein an ensemble model prediction performance metric is determined based on the subset of ensemble predictions; (iv) updating the set of baseline machine learning models based on the baseline model prediction performance metric for each baseline machine learning model; and (v) Repeat (i)-(iv) until there is a single baseline machine learning model in the set of baseline machine learning models or at least one of the ensemble model prediction performance metrics satisfies a threshold ensemble model prediction performance, the ratio of the multiple second training samples outside the global rejection region satisfies a threshold ratio, or any combination thereof.

16. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, cause the at least one processor, for each baseline machine learning model in the set of baseline machine learning models, to generate, for the baseline machine learning model, the rejection region associated with the at least one data type in the plurality of different data types based on the plurality of first predictions for the plurality of first training samples by optimizing an objective function defined according to the following equation and any other constraints of interest: Where 0 <t lb <1; where t lb is the lower limit of the rejection region; T acc is a model performance evaluation metric of the baseline machine learning model calculated using the first sample of the plurality of samples outside the rejection region; N rej is the number of the plurality of first samples that are rejected due to scores within the rejection region; N is the total number of the plurality of first samples; and a % is a user-defined rejection rate limit for the baseline machine learning model.

17. The computer program product of claim 16, wherein t lb The optimal solution of is obtained using at least one of the following search or optimization algorithms: grid search, Bayesian optimization, simulated annealing, genetic algorithm, particle swarm optimization, or any combination thereof.

18. A computer program product as described in claim 15, wherein for each baseline machine learning model in the set of baseline machine learning models, the multiple second training samples are associated with multiple distance metrics, wherein each distance metric in the multiple distance metrics indicates the distance of the corresponding second sample from at least one boundary of the rejection region of the baseline machine learning model, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to train the integrated machine learning model based on (d) the multiple distance metrics for the multiple second samples associated with each baseline machine learning model.

19. The computer program product of claim 15, wherein a metamodel is used to integrate the set of baseline machine learning models into the integrated machine learning model.

20. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: (vi) obtaining the current sample; (vii) determining whether the current sample is within the global rejection region; (viii) in response to determining that the current sample is outside the global rejection region, automatically processing the current sample using the integrated machine learning model to generate a current prediction for the current sample; and (ix) in response to determining that the current sample is within the global rejection region, automatically marking the current sample as unable to receive a credible prediction from the ensemble classifier.

Images