Machine learning model error detection

The method and system address the challenge of incorrect predictions in machine learning models by using explainable features' importance analysis and human evaluation to enhance model accuracy and reliability.

JP7580485B2Active Publication Date: 2024-11-11INTERNATIONAL BUSINESS MACHINE CORPORATION
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022564593
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-05-29
Filing Date
2021-05-18
Publication Date
2024-11-11
Estimated Expiration
2041-05-18

AI Technical Summary

Technical Problem

Existing machine learning models lack effective methods for identifying and correcting incorrect predictions, which hinders their reliability and usability.

Method used

A method and system that utilize global-level and local-level importance magnitudes and directions of explainable features to identify and correct errors in machine learning model predictions, involving data perturbation, feature importance calculation, and human evaluation to refine model outputs.

Benefits of technology

Enhances the accuracy and reliability of machine learning models by pinpointing and correcting erroneous predictions, improving their performance and user trust through informed decision-making.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580485000037
    Figure 0007580485000037
  • Figure 0007580485000038
    Figure 0007580485000038
  • Figure 0007580485000039
    Figure 0007580485000039
Patent Text Reader

Abstract

The system includes a memory having instructions therein and at least one processor in communication with the memory. The at least one processor is configured to execute the instructions to determine global-level importance magnitude values ​​for global-level importance of explainable features of the machine learning-based model based on a first prediction of the machine learning-based model. The at least one processor is further configured to execute the instructions to determine global-level importance directional labels for the global-level importance of the explainable features based on the first prediction. The at least one processor is further configured to execute the instructions to generate information for presentation to a user based on a second prediction of the machine learning-based model, the global-level importance magnitude values, and the global-level importance directional labels.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates generally to the field of artificial intelligence, and more particularly, to improving the accuracy of machine learning models. [Background technology]

[0002] Various capabilities of machines to acquire and apply knowledge and skills are categorized as artificial intelligence ("AI"). Machine learning is considered a form of AI. Machine learning uses algorithms and statistical models that enable computer systems to perform tasks based primarily on data patterns and associated inferences, rather than on explicit instructions. Some machine learning models perform classification or other predictive tasks, or a combination of both. The challenge is to provide actionable insight into when and how such models err. Summary of the Invention

[0003] A method for correcting an erroneous prediction of a machine learning based model for a user is disclosed. The method includes determining a global-level importance magnitude value of a global-level importance of an explainable feature of the machine learning based model for the machine learning based model based on a first prediction of the machine learning based model. The method further includes determining a global-level importance direction label of the global-level importance of the explainable feature of the machine learning based model for the machine learning based model based on the first prediction of the machine learning based model. The method further includes generating a communication for presentation to the user based on a second prediction of the machine learning based model, the global-level importance magnitude value, and the global-level importance direction label.

[0004] A method of alerting a user to an erroneous prediction of a machine learning based model is also disclosed. The method includes executing the machine learning based model on a first input data set to generate a pair of baseline predictions by the machine learning based model and determining a local-level importance of a first explainable feature of the machine learning based model to a predicted class of the machine learning based model. The method further includes receiving input data. The method further includes determining a global-level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local-level importance of the first explainable feature of the machine learning based model. The method further includes executing the machine learning based model on a second input data set to generate a new prediction by the machine learning based model. The method further includes determining an erroneousness designation for the new prediction based on the local-level importance of the first explainable feature of the machine learning based model and the global-level importance of the first explainable feature of the machine learning based model. The method further includes communicating the new prediction and an indication of the erroneousness designation for the new prediction for presentation to a user.

[0005] A system for correcting an erroneous prediction of a machine learning based model for a user is also disclosed. The system includes a memory having instructions therein and at least one processor in communication with the memory. The at least one processor is configured to execute the instructions to determine a global level importance magnitude value of a global level importance of the explainable feature of the machine learning based model to the machine learning based model based on a first prediction of the machine learning based model. The at least one processor is further configured to execute the instructions to determine a global level importance directional label of the global level importance of the explainable feature of the machine learning based model to the machine learning based model based on the first prediction of the machine learning based model. The at least one processor is further configured to execute the instructions to generate information for presentation to a user based on a second prediction of the machine learning based model, the global level importance magnitude value, and the global level importance directional label.

[0006] A system for alerting a user to an erroneous prediction of a machine learning based model is also disclosed. The system includes a memory having instructions therein and at least one processor in communication with the memory. The at least one processor is configured to execute the instructions to execute the machine learning based model on a first input data set to generate a pair of baseline predictions by the machine learning based model and to determine a local level importance of a first explainable feature of the machine learning based model to a predicted class of the machine learning based model. The at least one processor is further configured to execute the instructions to determine a global level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local level importance of the first explainable feature of the machine learning based model. The at least one processor is further configured to execute the instructions to execute the machine learning based model on a second input data set to generate a new prediction by the machine learning based model. The at least one processor is further configured to execute the instructions to determine an error assignment for the new prediction based on the local level importance of the first explainable feature of the machine learning based model and the global level importance of the first explainable feature of the machine learning based model. The at least one processor is further configured to execute the instructions to communicate the new prediction and an indication of an error designation for the new prediction for presentation to a user.

[0007] A computer program product is also disclosed for alerting a user to an erroneous prediction of a machine learning based model. The computer program product includes a computer readable storage medium having program instructions implemented thereon. The program instructions are executable by the at least one processor to cause the at least one processor to execute the machine learning based model on a first input data set to generate a pair of baseline predictions by the machine learning based model and determine a local level importance of a first explainable feature of the machine learning based model to a prediction class of the machine learning based model. The program instructions are further executable by the at least one processor to cause the at least one processor to determine a global level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local level importance of the first explainable feature of the machine learning based model. The program instructions are further executable by the at least one processor to cause the at least one processor to execute the machine learning based model on a second input data set to generate a new prediction by the machine learning based model. The program instructions are further executable by the at least one processor to cause the at least one processor to determine an error assignment for the new prediction based on a local-level importance of the first explainable feature of the machine learning based model and a global-level importance of the first explainable feature of the machine learning based model. The program instructions are further executable by the at least one processor to cause the at least one processor to communicate the new prediction and an indication of the error assignment for the new prediction for presentation to a user.

[0008] For a more complete understanding of the present disclosure, reference is now made to the following brief description, written in conjunction with the accompanying drawings and detailed description, in which like reference numerals represent like parts, and in which: [Brief description of the drawings]

[0009] [Figure 1] 1 is a table illustrating example local-level feature contributions or importance generated by LIME analysis of an example hypothetical example sentiment classification model (not shown), according to aspects of the present disclosure. [Diagram 2] FIG. 1 is a block diagram of a machine learning prediction system according to an aspect of the present disclosure. [Diagram 3] FIG. 13 is a data flow diagram illustrating how global-level importance magnitude and direction can be calculated for a unigram explainable feature, such as “underwhelming,” in some cases in accordance with an embodiment of the present disclosure. [Figure 4] 1 is a flow diagram of a machine learning prediction process according to an aspect of the present disclosure. [Diagram 5] FIG. 2 is a block diagram of a hardware architecture of a data processing system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] These diagrams are for illustration purposes only and are not intended to assert or imply limitations with respect to the environment, architecture, design or process in which different embodiments may be implemented.

[0011] Although the following provides exemplary implementations of one or more embodiments, it should be understood at the outset that the disclosed systems, computer program products, or methods, or combinations thereof, may be implemented using any number of technologies, whether currently known or in existence. The present disclosure is not limited to the exemplary implementations, drawings, and technologies described below, including the exemplary designs and implementations shown and described herein, but may be modified within the scope of the appended claims and the full scope of equivalents.

[0012] As used in the disclosure and claims herein, the terms "including" and "comprising" (and variations thereof) are used in an open-ended manner and should therefore be interpreted to mean "including, but not limited to." Unless otherwise indicated, throughout this document, "or" does not require mutual exclusivity, and the singular forms "a," "an," and "the" are intended to include the plural forms unless otherwise clear from the context.

[0013] As referred to herein, a "module" or "unit" (and variations thereof) includes one or more hardware or electrical components, such as an electrical circuit, a processor, and a memory, that may be specifically configured to perform a particular function. The memory may include volatile or non-volatile memory that stores data, such as, but not limited to, computer-executable instructions, machine code, and various other forms of data. The module or unit may be configured to use such data to execute one or more instructions to perform one or more tasks. In certain instances, a module or unit may refer to a particular set of functions, software instructions, or circuits configured to perform a particular task. For example, a module or unit may include software components, such as, but not limited to, data access objects, service components, user interface components, application programming interface ("API") components; hardware components, such as, but not limited to, an electrical circuit, a processor, and a memory; or a combination thereof. As referred to herein, computer-executable instructions may take any form, including, but not limited to, machine code, assembly code, and high-level programming code written in any programming language.

[0014] Additionally, as used herein, the term "communicate" (and variations thereof) refers to receiving and / or transmitting data or information over a communication link. The communication link may include both wired and wireless links, may include a direct link, or may include multiple links through one or more communication networks or one or more network devices, such as, but not limited to, routers, firewalls, servers, and switches. The communication network may include any type of wired or wireless network. The network may include a private network or a public network, such as the Internet, or both. Additionally, in some embodiments, the term communicate may further include internal communication between various components of the system, or with external input / output devices, such as keyboards or display devices, or both.

[0015] As used herein, "explainable features" (and variants thereof) refer to human-understandable representations of data. For text data, intuitive explainable features are words (or other unigrams) or phrases, although the corresponding respective actual machine learning models may be built with embeddings. For images, possible explainable features could be superpixels, but the corresponding respective models would be trained using pixel-level features.

[0016] As used herein, "local level explainability," "local explainability," "local level," "local," etc. (and variations thereof) refer to the interpretations used to justify why a respective machine learning model output a particular prediction for a single example. In some embodiments, aspects of the present disclosure use Local Interpretable Model-Agnostic Explanation ("LIME") techniques to determine local level explainability. LIME works by perturbing the explainable features of a given data input to a machine learning model and recording the resulting corresponding change in the machine learning model's prediction probability. Based on the perturbed data inputs and their respective corresponding prediction outputs, LIME uses linear regression to assign relative weights to the importance of the explainable features.

[0017] FIG. 1 is a table 100 illustrating exemplary local level feature contributions or importance generated by LIME analysis of an exemplary hypothetical sentiment classification model (not shown) according to aspects of the present disclosure. In the illustrated example, the model's prediction for sentence 2(a) "Panera gives me hiccups." is "positive." The unigram feature "panera" contributes positively to this positive prediction with a magnitude of 0.576, while "hiccups" contributes negatively to this positive prediction with a magnitude of 0.159. It should be appreciated that by observing these local level feature contributions, the model may be determined to provide a prediction error by considering the word "panera" as a significant indicator of positive polarity, thus leading the model to assign a positive label to this negative sentence. Various embodiments of the present disclosure may use LIME analysis to determine the local level importance of the explainable features of the machine learning model. However, it should also be appreciated that other embodiments may use any of a number of other suitable local level discriminability techniques. Non-limiting examples of suitable alternative approaches that provide local level interpretations include Shapley Additive exPlanations ("SHAP") and Black Box Explanations through Transparent Approximations ("BETA").

[0018] As used herein, the terms "global level explainability," "global explainability," "global level," "global," etc. (and variations thereof) refer to how an explainable feature affects the prediction of a corresponding respective machine learning model with respect to the entire or entire input dataset, as opposed to the effect of the explainable feature on predictions made from individual data instances of the entire dataset. In some cases, a machine learning model may be trained on millions of data instances, whereas a human may only be able to label with a much more limited number of inputs based on local explanations. It should be understood that global level explainability may enable an individual to achieve a more general understanding of a machine learning model without knowing the detailed instance level predictions, and in that sense, global explainability may help derive more refined knowledge for less human effort.

[0019] Aspects of the present disclosure provide a system for detecting one or more errors in a machine learning model. Given a pre-trained black-box machine learning model ("machine learning-based model" or "base model") and a first input dataset, the system runs the machine learning-based model on the first input dataset using a data perturbation process to generate baseline predictions by the machine learning-based model, and for each of the baseline predictions, determines a local level importance of the machine learning-based model's explainable features for each prediction class of the machine learning-based model. Each local level importance corresponds to a difference between a first prediction probability generated by the machine learning-based model by running the machine learning-based model on a first input including the corresponding respective explainable feature, and a second prediction probability generated by the machine learning-based model by running the machine learning-based model on a second input excluding the corresponding respective explainable feature, for each corresponding prediction class of the machine learning-based model. The system aggregates the local level importance of the explainable features of the machine learning-based model to determine a global level importance of the explainable features for the entire machine learning-based model. Each global-level importance of each corresponding explainable feature includes a magnitude value (a "global-level importance magnitude value") and a direction label (a "global-level importance direction label"). The global-level importance magnitude value corresponds to a relatively greatest magnitude of importance of each corresponding explainable feature among a predicted class of the machine learning based model, and the global-level importance direction label corresponds to a predicted class of the machine learning based model associated with the magnitude value. The system ranks the explainable features according to each corresponding global-level importance magnitude value of the explainable features, communicates one or more of the highest ranked ones of the explainable features, and communicates (for presentation to one or more human raters) a corresponding respective global-level importance direction label of each such one or more of the highest ranked ones of the explainable features.The system receives information on one or more of the highest ranked explainable features having an incorrect global-level importance direction according to a corresponding respective consensus human rating. The system further receives a second input dataset. The system executes the machine learning based model on the second input dataset to generate one or more new predictions by the machine learning based model. The system calculates a local error score for each new prediction as a normalization of an accumulated error induced in each new prediction by the highest ranked explainable feature having an incorrect global-level importance direction according to a corresponding respective consensus human rating. The system determines a corresponding respective error designation for each new prediction (e.g., "problematic" or "suspected error" as opposed to "no problem" or "no suspected error") based on whether the corresponding respective local error score for the new prediction is greater than a threshold. In some embodiments, the system communicates (for presentation to one or more system developers or one or more other users) an indication of each new prediction and the corresponding respective error designation for the new prediction. It should be appreciated that the one or more system developers or one or more other users may further analyze the problematic new predictions and / or analyze associated data or other characteristics of the machine learning based model, and may modify the machine learning based model and / or take one or more other remedial and / or corrective actions. In some embodiments, the system communicates (for presentation to the one or more users) the one or more corrected predictions. Each such corrected prediction is based on a corresponding respective new prediction of the one or more new predictions and an error designation for the new prediction.More specifically, when the error assignment for a new prediction is "questionable" or "suspected," each corresponding revised prediction is the inverse of the new prediction. Conversely, when the error assignment for a new prediction is "clean" or "not suspected," each corresponding revised prediction is the same as the new prediction.

[0020] FIG. 2 is a block diagram of a machine learning prediction system 200 according to an embodiment of the present disclosure. The machine learning prediction system 200 is configured to perform a machine learning prediction process 400 according to an embodiment of the present disclosure (the machine learning prediction process 400 itself is not explicitly shown in FIG. 2, see FIG. 4). The machine learning prediction system 200 includes a local level importance generation module 212. The local level importance generation module 212 is configured to receive information of a machine learning based model 216 and a first input dataset 220. It should be understood that the machine learning based model 216 can be a pre-trained classification model for tasks such as sentiment analysis, intent prediction, image classification, or other pre-trained machine learning models that can be treated as black boxes. It should also be understood that the machine learning based model 216 can include logistic regression, support vector machines ("SVM"), random forests, combinatorial neural networks ("CNN"), recurrent neural networks ("RNN"), and / or one or more other types of machine learning and / or deep learning models. The local level importance generation module 212 further calculates the local level importance for each data instance d i , N}, and determining an absolute change in the predicted probability of the machine learning based model 216 associated with each class label k ∈ {0, 1, ..., K} of the machine learning based model 216 by:

number

number

number

[0021] The local level importance generation module 212 further calculates the resulting set of

number

[0022] The machine learning prediction system 200 further includes a global level importance generation module 228. The global level importance generation module 228 is communicatively coupled to the local level importance generation module 212. The global level importance generation module 228 is configured to receive the information of the local level importance 224. The global level importance generation module 228 further generates a global level importance generation module 228 for all N data instances d i For all N data instances d iThe local level importance of explainable feature j for

number

number

number

number

[0023] 3 is a data flow diagram 300 illustrating how global-level importance magnitude and direction can be calculated for a unigram explainable feature, e.g., "underwhelming," in some cases according to aspects of the present disclosure. It should be understood that this global-level importance measure can be viewed as an aggregation of local-level importance, with the underlying assumption being that if removing a feature results in a large change in predicted probability, then the feature is important, and if removing a feature results in a small change in predicted probability, then the feature is important, but not important.

[0024] Referring again to FIG. 2, the global level importance generation module 228 further evaluates the resulting set of k * as corresponding respective global level importance directions 232, resulting in a set of related

number

[0025] The machine learning prediction system 200 further includes an explainable feature ranking module 240. The explainable feature ranking module 240 is communicatively coupled to the global level importance generation module 228. The explainable feature ranking module 240 is configured to receive information of the global level importance direction 232 and the global level importance magnitude 236. The explainable feature ranking module 240 is further configured to rank the explainable features j in descending order according to the corresponding respective global level importance magnitudes 236 of the explainable features j. The explainable feature ranking module 240 is further configured to communicate one or more T explainable features j of the highest ranked such explainable features j and their corresponding respective global level importance directions 232 as T highest ranked explainable features 244 and T global level importance directions 248, respectively, for final presentation to one or more human raters 252. In some embodiments, the explainable feature ranking module 240 may include one or more corresponding data processing systems, such as data processing system 500 (data processing system 500 itself is not explicitly shown in FIG. 2, see FIG. 5).

[0026] The human raters 252 may include experts, online crowd workers, or others who are able to identify explainable features 244 for which any one or more of the T top ranked explainable features 244 corresponding to each of the T global level importance directions 248 are not what the human rater 252 believes the global level importance directions 248 should be. For example, such an evaluation task may include the human rater 252 making a yes or no (e.g., positive or negative in the case of a sentiment analysis task) decision for each of the T top ranked explainable features 244 and each of the T corresponding global level importance directions 248 when asked whether or not they believe the corresponding respective global level importance directions 248 of the T global level importance directions 248 is correct. An example task question may be, "Is the word 'panera' a word of 'positive' psychopolarity?" to which evaluators are expected to select "no" as a response. The consensus of the human evaluators 252 may determine that one or more of the E explainable features 244 among the T top ranked explainable features 244 have corresponding respective global-level importance directions 248 among the T global-level importance directions 248 that are problematic or incorrect (i.e., not what the consensus of the human evaluators 252 believes their global-level importance directions 248 should be). This corresponding identification of the E problematic explainable features 260 may be used by the machine learning prediction system 200, as described further below.

[0027] The machine learning prediction system 200 further includes a new prediction generation module 264. The new prediction generation module 264 is configured to receive the machine learning based model 216 and the second input dataset 268. The new prediction generation module 264 is further configured to execute the machine learning based model 216 on the second input dataset 268 to generate a new prediction 272 by the machine learning based model 216 based on the second input dataset 268. The new prediction generation module 264 is further configured to communicate the new prediction 272. In some embodiments, the new prediction generation module 264 can include a corresponding one or more of a data processing system, such as the data processing system 500 (not explicitly shown in FIG. 2 , see FIG. 5 ).

[0028] The machine learning prediction system 200 further includes a local error score generation module 276. The local error score generation module 276 is communicatively coupled to the local level importance generation module 212, the global level importance generation module 228, and the new prediction generation module 264. The local error score generation module 276 is configured to receive the local level importance 224 information, the global level importance direction 232 information, the new prediction 272 information, and the identification of the E problematic explainable features 260 information. It should be understood that, although the T top ranked explainable features 244 may be useful in identifying problematic predictions for unlabeled instances, in some cases, it may be somewhat unreliable to flag an error occurrence at the individual instance level based solely on the problematic explainable features identified by the identification of the E problematic explainable features 260. For example, as shown in FIG. 1, noticing that “panera” is incorrectly learned as “positive” may help to accurately identify the erroneous prediction for sentence 2(a), while its erroneous impact for sentence 2(b) is disguised by the presence of other positive features “good” that may be correctly learned at the global level. Referring back to FIG. 2 again, to more accurately determine the problematic predictions, the local error score generation module 276 is further configured to calculate a measurement metric called a local error score e to take into account the relative impact of the global erroneous features on the local level. More specifically, the local error score generation module 276 is configured to calculate the local error score e of each of the new predictions 272 as a normalized cumulative error contribution induced in each of the new predictions 272 by the problematic explainable features identified by the identification of the E problematic explainable features 260, as follows:

number

number

number

[0029] 2, the local error score generation module 276 is further configured to associate each of the local error scores e with a corresponding respective one of the new predictions 272 and communicate the resulting set of predictions as e-scored predictions 280. In some embodiments, the local error score generation module 276 may include a corresponding one or more of data processing systems, such as data processing system 500 (not explicitly shown in FIG. 2, see FIG. 5).

[0030] The machine learning prediction system 200 further includes an output logic module 284. The output logic module 284 is communicatively coupled to the local error score generation module 276. The output logic module 284 is configured to receive information of the e-scored predictions 280. The output logic module 284 is further configured to determine a corresponding respective error designation for each of the e-scored predictions 280 (and thus for each of the associated new predictions 272) based on whether the corresponding respective local error score e of that one of the scored predictions 280 e-scored predictions 280 is greater than a predetermined threshold τ. For example, in some embodiments, the output logic module 284 is configured to designate each of the e-scored predictions 280 (and thus each of the associated new predictions 272) having a local error score e greater than a predetermined threshold τ as “problematic” or as a “problematic new prediction.” The output logic module 284 is further configured to automatically generate a corresponding respective revised new prediction 288 for each of the corresponding respective e-scored predictions 280 based on whether the one e-scored prediction 280 of the corresponding respective e-scored predictions 280 is greater than a predetermined threshold τ. For example, when a particular one e-scored prediction 280 of the e-scored predictions 280 is “no” or “negative” and the local error score e of the one e-scored prediction 280 of the e-scored predictions 280 is greater than a predetermined threshold τ, the output logic module 284 can automatically generate a “yes” or “positive” prediction as the corresponding one revised new prediction 288 of the revised new predictions 288.Conversely, when a particular one of the e-scored predictions 280 is “yes” or “positive” and the local error score e of that one of the e-scored predictions 280 is greater than a predetermined threshold τ, the output logic module 284 can automatically generate a “no” or “negative” prediction as a corrected new prediction 288 of a corresponding one of the corrected new predictions 288. The output logic module 284 is configured to make the corrected new prediction 288 of the one of the corrected new predictions 288 the same as the corresponding one of the new predictions 272 for each corrected new prediction 288 for which the corresponding one of the e-scored predictions 280 of the e-scored predictions 280 does not exceed the predetermined threshold τ.

[0031] The output logic module 284 is further configured to communicate the revised new predictions 288. The output logic module 284 is further configured to communicate any e-scored predictions 280 (and thus any associated new predictions 272) having a corresponding respective local error score e greater than a predetermined threshold τ as a corresponding set of problematic new predictions 292 consisting of one or more corresponding respective problematic new predictions 292. In some embodiments, the output logic module 284 may include a corresponding one or more of a data processing system, such as data processing system 500 (not explicitly shown in FIG. 2, see FIG. 5).

[0032] FIG. 4 is a flow diagram of a machine learning prediction process 400 according to an embodiment of the present disclosure. In some cases, one or more steps of the machine learning prediction process 400 can be performed by one or more components of the machine learning prediction system 200 and / or by one or more of the other systems, components, methods, and / or processes described herein. For clarity of explanation, the following description of the machine learning prediction process 400 may relate to one or more of such systems, components, methods, or processes, or combinations thereof. Nevertheless, it should be understood that the machine learning prediction process 400, or one or more of the specific steps of the machine learning prediction process 400, or both, can also be performed by any suitable system, component, method, or process, or combinations thereof. It should also be understood that the machine learning prediction process 400 can be performed simultaneously or substantially simultaneously with other methods or processes or both described herein.

[0033] At step 412, the machine learning prediction process 400 receives the machine learning based model. In response, in some cases, the local level importance generation module 212 and / or the new prediction generation module 264 may receive the machine learning based model 216. From step 412, the machine learning prediction process 400 proceeds to step 418 (following step 418).

[0034] At step 418, the machine learning prediction process 400 receives the first input dataset. In response, in some cases, the local level importance generation module 212 can receive information from the first input dataset 220. From step 418, the machine learning prediction process 400 proceeds to (follows) step 424.

[0035] At step 424, the machine learning prediction process 400 runs the machine learning based model on the first input dataset using a data perturbation process to generate baseline predictions by the machine learning based model, and for each baseline prediction, determines local level importance of the machine learning based model's explainable features for each prediction class of the machine learning based model. Accordingly, in some cases, the local level importance generation module 212 can run the machine learning based model 216 on the first input dataset 220 using a LIME data perturbation technique to determine the local level importance 224. From step 424, the machine learning prediction process 400 proceeds to step 430 (following step 430).

[0036] At step 430, the machine learning prediction process 400 aggregates the local level importance of the explainable features of the machine learning based model to determine a global level importance of the explainable features to the entire machine learning based model. Accordingly, in some cases, the global level importance generation module 228 can use the local level importance 224 to determine the global level importance direction 232 and the global level importance magnitude 236. From step 430, the machine learning prediction process 400 proceeds to (follows) step 436.

[0037] At step 436, the machine learning prediction process 400 ranks the explainable features according to their respective corresponding global-level importance magnitude values. In response, in some instances, the explainable feature ranking module 240 may rank the explainable features j in descending order according to their respective corresponding global-level importance magnitudes 236. From step 436, the machine learning prediction process 400 proceeds to (follows) step 442.

[0038] At step 442, the machine learning prediction process 400 communicates one or more of the highest ranked explainable features among the explainable features and communicates the corresponding respective global level importance direction labels of each such highest ranked explainable feature among the explainable features. In response, in some cases, the explainable feature ranking module 240 can communicate one or more of such highest ranked explainable features among the explainable features j and their corresponding respective global level importance directions 232 as the T highest ranked explainable features 244 and the T global level importance directions 248, respectively, for eventual presentation to one or more human raters 252. From step 442, the machine learning prediction process 400 proceeds to step 448 (following step 448).

[0039] At step 448, the machine learning prediction process 400 receives information about each of one or more of the highest ranked explainable features of the explainable features that have an incorrect global-level importance direction according to one or more consensus human ratings. In response, in some cases, the local error score generation module 276 may receive information about the identification of the E problematic explainable features 260. From step 448, the machine learning prediction process 400 proceeds to (follows) step 454.

[0040] At step 454, the machine learning prediction process 400 receives a second input dataset. In response, in some cases, the new prediction generation module 264 can receive information from the second input dataset 268. From step 545, the machine learning prediction process 400 proceeds to step 460 (following step 460).

[0041] At step 460, the machine learning prediction process 400 runs the machine learning based model on the second input dataset to generate one or more new predictions by the machine learning based model. In response, in some cases, the new prediction generation module 264 can run the machine learning based model 216 on the second input dataset 268 to generate new predictions 272. From step 460, the machine learning prediction process 400 proceeds to step 466 (following step 466).

[0042] In step 466, the machine learning prediction process 400 calculates a local error score for each new prediction of the machine learning based model as a normalization of the accumulated error induced in each new prediction by the highest ranked explainable feature among the explainable features having a corresponding respective erroneous global level importance direction according to the corresponding respective consensus human rating. Accordingly, in some instances, the local error score generation module 276 can calculate a local error score e for each new prediction 272 as a normalization of the accumulated error contribution induced in each new prediction 272 by the problematic explainable features identified by the identification of the E problematic explainable features 260. From step 466, the machine learning prediction process 400 proceeds to step 472 (following step 472).

[0043] At step 472, the machine learning prediction process 400 determines a corresponding respective error assignment for each new prediction of the machine learning based model based on whether the corresponding respective local error score of the new prediction is greater than a threshold. In response, in some instances, the output logic module 284 may determine a corresponding respective error assignment for each of the e-scored predictions 280 (and thus each associated new prediction 272) based on whether the corresponding respective local error score e of that one of the e-scored predictions 280 is greater than a predetermined threshold τ. From step 472, the machine learning prediction process 400 proceeds to step 478 (following step 478).

[0044] At step 478, the machine learning prediction process 400 communicates each new prediction of the machine learning based model and an indication of a corresponding respective error assignment for the new prediction, or communicates a corresponding respective corrected prediction, or both. In response, in some cases, the output logic module 284 can communicate the corrected new prediction 288, or can communicate one or more corresponding respective problematic new predictions 292, or both.

[0045] 5 is a block diagram of a hardware architecture of a data processing system 500 according to aspects of the disclosure. In some embodiments, a corresponding one or more of the data processing systems 500 may be used to implement one or more of the systems and / or components described herein (e.g., the machine learning prediction system 200 or one or more components thereof). Additionally, the data processing system 500 may be configured to store and execute one or more instructions to perform one or more steps of the machine learning prediction process 400 and / or one or more steps of other methods and / or processes described herein.

[0046] Data processing system 500 uses a hub architecture that includes a north bridge and memory controller hub ("NB / MCH") 506 and a south bridge and input / output ("I / O") controller hub ("SB / ICH") 510. Connected to NB / MCH 506 are processor 502, main memory 504, and graphics processor 508. Graphics processor 508 may be connected to NB / MCH 506 through an accelerated graphics port (AGP). Any type of communications fabric or architecture can be used to implement a computer bus, such as bus 532 or bus 534, that provides for the transfer of data between different components or devices attached to the communications fabric or architecture.

[0047] Attached to the SB / ICH 510 is a network adapter 516. Attached to the SB / ICH 510 are an audio adapter 530, a keyboard and mouse adapter 522, a modem 524, a read-only memory ("ROM") 526, a hard disk drive ("HDD") 512, a compact disk read-only memory ("CD-ROM") drive 514, a universal serial bus ("USB") port and other communications ports 518, and peripheral component interconnect / peripheral component interconnect express ("PCI / PCIe") devices 520 through buses 532 and 534. PCI / PCIe devices can include, for example, Ethernet adapters, add-in cards, and personal computing ("PC") cards for notebook computers. PCI uses a card bus controller, whereas PCIe does not. The ROM 526 may include, for example, a flash basic input / output system ("BIOS"). The modem 524 or network adapter 516 may be used to send and receive data over a network.

[0048] The HDD 512 and CD-ROM drive 514 connect to the SB / ICH 510 through a bus 534. The HDD 512 and CD-ROM drive 514 may use, for example, an Integrated Drive Electronics ("IDE") or a Serial Advanced Technology Attachment ("SATA") interface. In some embodiments, other forms of data storage devices may be used in place of the HDD 512, including, but not limited to, a solid-state drive ("SSD"). A super I / O ("SIO") device 528 may be connected to the SB / ICH 510. The SIO device 528 may include chips on the motherboard configured to help perform less demanding control functions for the SB / ICH 510, such as controlling a printer port, controlling a fan, or controlling small light emitting diodes ("LEDs") of the data processing system 500, or a combination thereof.

[0049] Data processing system 500 may include a single processor 502 or may include multiple processors 502. Additionally, processor 502 may have multiple cores. In some embodiments, data processing system 500 may use multiple processors 502, including hundreds or thousands of processor cores. In some embodiments, processors 502 may be configured to perform a coordinated set of calculations in parallel.

[0050] An operating system runs on data processing system 500 using processor 502. The operating system coordinates and provides control of various components within data processing system 500. Various applications and services may run in conjunction with the operating system. Instructions for the operating system, applications and other data may be located on a storage device, such as one or more of HDDs 512, and loaded into main memory 504 for execution by processor 502. In some embodiments, additional instructions or data may be stored on one or more external devices. The processes described herein for the exemplary embodiments may be executed by processor 502 using computer usable program code, which may be located in memory, such as main memory 504, ROM 526, or one or more peripheral devices.

[0051] The present invention may be a system, method, or computer program product, or combinations thereof, at any level of technical detail capable of integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to carry out aspects of the present invention.

[0052] The computer readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves having instructions recorded thereon, and any suitable combination thereof. As used herein, a computer-readable storage medium should not be construed as being, itself, a ephemeral signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating in a waveguide or other transmission body (e.g., a light pulse traveling in a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0053] The computer readable program instructions described herein may be downloaded from a computer readable storage medium into the respective computing / processing device or may be downloaded to an external computer or storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium in the respective computing / processing device for storage.

[0054] The computer readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data or configuration data for an integrated circuit, or may be source or object code written in any combination of one or more programming languages, including object oriented programming languages ​​such as Smalltalk®, C++, and procedural programming languages ​​such as the “C” programming language or the like. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or remote server. In the last scenario above, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (e.g., via the Internet using an Internet Service Provider). According to aspects of the present disclosure, an electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuit to perform aspects of the present invention.

[0055] Aspects of the present invention are described herein with reference to flowchart and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer readable program instructions.

[0056] These computer readable program instructions may be provided to a processor of a general purpose or special purpose computer, or other programmable data processing apparatus forming a machine, in such a manner that the instructions, executed by the processor of that computer or other programmable data processing apparatus, produce means for implementing the functions / operations specified in the blocks of the flowcharts and / or block diagrams. These computer readable program instructions may further be stored on a computer readable storage medium capable of directing a computer, programmable data processing apparatus or other device, or combination thereof, to function in a particular manner, in such a manner that the computer readable storage medium having instructions stored therein comprises an article of manufacture containing instructions for implementing aspects of the functions / operations specified in the blocks of the flowcharts and / or block diagrams.

[0057] These computer readable program instructions can further be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented method, in a manner such that the instructions executed on the computer, other programmable apparatus or other device perform the functions / operations specified in the blocks of these flowcharts and / or block diagrams.

[0058] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may be performed in an order different from the order shown in the figures. For example, two blocks shown in succession may actually be performed as one step, may be performed simultaneously, substantially simultaneously, partially or completely overlapping in time, or the blocks may sometimes be performed in reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a hardware-based dedicated system that performs the specified functions or operations or implements a combination of dedicated hardware and computer instructions.

[0059] The above description of various embodiments of the present invention has been provided for illustrative purposes, and is not intended to be exhaustive or to be limited to the disclosed embodiments. Many modifications and variations that do not depart from the scope and spirit of the described embodiments will be apparent to those skilled in the art. Moreover, the steps of the methods described herein may be performed in any suitable order, or simultaneously where appropriate. The terms used herein have been selected to best explain the principles, practical applications, or technical improvements of the embodiments over commercially available technology, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for correcting an erroneous prediction of a machine learning based model for a user, the method comprising: determining a global-level importance magnitude value of a global-level importance of the explainable feature of the machine learning based model for the machine learning based model based on a first prediction of the machine learning based model; determining a global-level importance directional label for the global-level importance of the explainable feature of the machine learning based model for the machine learning based model based on the first prediction of the machine learning based model; and generating information for presentation to the user based on the second prediction of the machine learning based model, the global-level importance magnitude values ​​and the global-level importance directional labels; The method includes:

2. The method of claim 1 , wherein generating the information for presentation to the user comprises receiving an error assessment of the global-level importance directional label.

3. 3. The method of claim 2, further comprising: revising the erroneous prediction based on the global-level importance directional label, the error assessment of the global-level importance directional label, and a local-level importance of the explainable feature of the machine learning based model to a predicted class of the machine learning based model.

4. The method of claim 3 , wherein receiving the error assessment of the global level importance directional labels comprises receiving a human error assessment of the global level importance directional labels.

5. 1. A method for alerting a user to an erroneous prediction of a machine learning based model, the method comprising: running the machine learning based model on a first input dataset to generate a pair of baseline predictions by the machine learning based model and determining a local level importance of a first explainable feature of the machine learning based model to a predicted class of the machine learning based model; determining a global level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local level importance of the first explainable feature of the machine learning based model; running the machine learning based model on a second input dataset to generate new predictions by the machine learning based model; determining an error assignment for the new prediction based on the local-level importance of the first explainable feature of the machine learning based model and the global-level importance of the first explainable feature of the machine learning based model; and communicating the new prediction and an indication of the error assignment for the new prediction for presentation to the user. The method includes:

6. determining a rank of the first explainable feature of the machine learning based model relative to a second explainable feature of the machine learning based model based on a global-level importance magnitude value; communicating the first explainable feature of the machine learning based model based on the rank of the first explainable feature of the machine learning based model; communicating global level importance directional labels; receiving information about the first explainable feature of the machine learning based model; and calculating a local error score for the new prediction as a normalization of the error induced in the new prediction by the first explainable feature of the machine learning based model; Further comprising: the global level importance of the first explainable feature of the machine learning based model comprises the global level importance magnitude value and the global level importance direction label; the global-level importance directional label is incorrect according to human evaluation; determining the error assignment for the new predictions includes determining the error assignment for each new prediction based on whether the local error score for the new prediction is greater than a threshold. The method according to claim 5.

7. 7. The method of claim 6, wherein running the machine learning based model on the first input dataset to generate the pair of baseline predictions by the machine learning based model and determining the local level importance of the first explainable feature of the machine learning based model to the prediction class of the machine learning based model comprises using a data perturbation process to generate the pair of baseline predictions by the machine learning based model and determine the local level importance of the first explainable feature of the machine learning based model to the prediction class of the machine learning based model.

8. generating the pair of baseline predictions by the machine learning based model using the data perturbation process and determining the local level importance of the first explainable feature of the machine learning based model to the predicted class of the machine learning based model; [0010] Calculating where j represents the first explainable feature, In the above formula, d i , i ∈ {0, 1, . . . , N} represents a first data instance in the first input data set; In the above formula, d i includes the first explainable feature j, In the above formula, [0025] is the first data instance d excluding the first explainable feature j i Represents all of the above. where k ∈ {0, 1, . . . , K} represents the corresponding predicted class of the machine learning based model; In the above formula, P(y=k|d i ) applies the machine learning based model to the first data instance d i represents a first predicted probability for the corresponding predicted class k of the machine learning based model generated by the machine learning based model by running In the above formula, [0030] The machine learning based model [0045] represents a second predicted probability for the corresponding predicted class k of the machine learning based model generated by the machine learning based model by running In the above formula, [0050] is the first data instance d i represents the local level importance of the first explainable feature j associated with the corresponding predicted class k with respect to The method of claim 7.

9. determining the global level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local level importance of the first explainable feature of the machine learning based model; [006] Calculating where N represents the cardinality of a set of data instances in the first input data set; wherein each data instance in the set of data instances comprises the first explainable feature; In the above equation, k * represents the global-level importance directional label of the global-level importance of the first explainable feature, The method according to claim 8.

10. determining the global level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local level importance of the first explainable feature of the machine learning based model; [0070] Calculating In the above formula, [0080] represents a third predicted probability magnitude associated with the global-level importance directional label of the global-level importance of the first explainable feature generated by the machine learning based model; In the above formula, [0097] represents the global-level importance magnitude value of the global-level importance of the first explainable feature, 10. The method of claim 9.

11. Calculating the local error score of the new prediction [0010] Calculating In the above formula, ##EQU00011## represents the contribution of each corresponding first explainable feature j to a prediction probability generated by the machine learning based model by running the machine learning based model on a second data instance in the second input dataset; where each global-level importance directional label in the set of global-level importance directional labels is erroneous according to the human evaluation; where m represents the cardinality of the set of global-level importance directional labels; In the above formula, ##EQU00012## represents the contribution of each corresponding second explainable feature i to the predicted probability generated by the machine learning based model by running the machine learning based model on the second data instances in the second input dataset; where the contribution direction of each corresponding second explainable feature i is consistent with the new prediction; where n represents the cardinality of the set of explainable features, where the contribution direction of each explanatory feature in the set of explanatory features is consistent with the new prediction. The method of claim 10.

12. 1. A system for correcting an erroneous prediction of a machine learning based model for a user, the system comprising: a memory having instructions therein; at least one processor in communication with the memory; wherein the at least one processor executes the instructions to determining a global-level importance magnitude value of a global-level importance of the explainable feature of the machine learning based model for the machine learning based model based on a first prediction of the machine learning based model; determining a global-level importance directional label for the global-level importance of the explainable feature of the machine learning based model for the machine learning based model based on the first prediction of the machine learning based model; generating information for presentation to the user based on the second prediction of the machine learning based model, the global-level importance magnitude value, and the global-level importance directional label; A system configured as follows.

13. The at least one processor further executes the instructions to: receiving an error estimate of the global level importance directional label; generating the information for presentation to the user based on the error assessment; The system according to claim 12, configured to:

14. 14. The system of claim 13, wherein the at least one processor is further configured to execute the instructions to correct the erroneous prediction based on the global level importance directional label, the error assessment of the global level importance directional label, and a local level importance of the explainable feature of the machine learning based model to a predicted class of the machine learning based model.

15. 15. The system of claim 14, wherein the at least one processor is further configured to execute the instructions to receive a human error assessment of the global-level importance directional labels.

16. 1. A system for alerting a user to an erroneous prediction of a machine learning based model, the system comprising: a memory having instructions therein; at least one processor in communication with the memory; wherein the at least one processor executes the instructions to running the machine learning based model on a first input dataset to generate a pair of baseline predictions by the machine learning based model and determining a local level importance of a first explainable feature of the machine learning based model to a predicted class of the machine learning based model; determining a global level importance of the first explainable feature of the machine learning based model to the machine learning based model based on the local level importance of the first explainable feature of the machine learning based model; running the machine learning based model against a second input dataset to generate new predictions by the machine learning based model; determining an error assignment for the new prediction based on the local-level importance of the first explainable feature of the machine learning based model and the global-level importance of the first explainable feature of the machine learning based model; communicating the new prediction and an indication of the error assignment for the new prediction for presentation to the user. A system configured as follows.

17. The at least one processor further executes the instructions to: determining a rank of the first explainable feature of the machine learning based model relative to a second explainable feature of the machine learning based model based on a global-level importance magnitude value; communicating the first explainable feature of the machine learning based model based on the rank of the first explainable feature of the machine learning based model; Communicate global level importance directional labels; receiving information about the first explainable feature of the machine learning based model; calculating a local error score for the new prediction as a normalization of the error induced in the new prediction by the first explainable feature of the machine learning based model; determining the error assignment for each new prediction based on whether the local error score for the new prediction is greater than a threshold; It is structured as follows: the global level importance of the first explainable feature of the machine learning based model comprises the global level importance magnitude value and the global level importance direction label; the global level importance directional label is identified as erroneous; 17. The system of claim 16.

18. 20. The system of claim 17, wherein the at least one processor is further configured to execute the instructions to use a data perturbation process to generate the pair of baseline predictions by the machine learning based model and to determine the local level importance of the first explainable feature of the machine learning based model to the predicted class of the machine learning based model.

19. The at least one processor further executes the instructions to: ##EQU00013## is configured to calculate where j represents the first explainable feature, In the above formula, d i , i ∈ {0, 1, . . . , N} represents a first data instance in the first input data set; In the above formula, d i includes the first explainable feature j, In the above formula, ##EQU00014## is the first data instance d excluding the first explainable feature j i Represents all of the above. where k ∈ {0, 1, . . . , K} represents the corresponding predicted class of the machine learning based model; In the above formula, P(y=k|d i ) applies the machine learning based model to the first data instance d i represents a first predicted probability for the corresponding predicted class k of the machine learning based model generated by the machine learning based model by running In the above formula, ##EQU00015## The machine learning based model ##EQU00016## represents a second predicted probability for the corresponding predicted class k of the machine learning based model generated by the machine learning based model by running In the above formula, ##EQU00017## is the first data instance d i represents the local level importance of the first explainable feature j associated with the corresponding predicted class k with respect to 20. The system of claim 18.

20. The at least one processor further executes the instructions to: [0018] is configured to calculate where N represents the cardinality of a set of data instances in the first input data set; wherein each data instance in the set of data instances comprises the first explainable feature; In the above equation, k * represents the global-level importance directional label of the global-level importance of the first explainable feature, 20. The system of claim 19.

21. The at least one processor further executes the instructions to: [0019] is configured to calculate In the above formula, [0020] represents a third predicted probability magnitude associated with the global-level importance directional label of the global-level importance of the first explainable feature generated by the machine learning based model; In the above formula, ##EQU00021## represents the global-level importance magnitude value of the global-level importance of the first explainable feature, 21. The system of claim 20.

22. The at least one processor further executes the instructions to: [0022] is configured to calculate In the above formula, [0023] represents the contribution of each corresponding first explainable feature j to a prediction probability generated by the machine learning based model by running the machine learning based model on a second data instance in the second input dataset; where each global-level importance directional label in the set of global-level importance directional labels is incorrect according to human evaluation, where m represents the cardinality of the set of global-level importance directional labels; In the above formula, [0024] represents the contribution of each corresponding second explainable feature i to the predicted probability generated by the machine learning based model by running the machine learning based model on the second data instances in the second input dataset; where the contribution direction of each corresponding second explainable feature i is consistent with the new prediction; where n represents the cardinality of the set of explainable features, where the contribution direction of each explanatory feature in the set of explanatory features is consistent with the new prediction.

22. The system of claim 21.

23. 12. A computer program for alerting a user to erroneous predictions of a machine learning based model, the computer program causing at least one processor to perform the method according to any one of claims 5 to 11.

24. 5. A computer program for correcting erroneous predictions of a machine learning based model for a user, the computer program causing at least one processor to perform the method according to any one of claims 1 to 4.

Citation Information

Patent Citations

  • Method and system for constructing adaptive prediction model

    JP2004086896A

  • Prediction characterization for black box machine learning models

    US20190122135A1

  • Model interpretation

    US20190325333A1

  • Information processing device, information processing method, and program

    WO2019130974A1