Method and apparatus for evaluating model interpretation tools

Abstract

The embodiments of this specification provide an evaluation method and device for model interpretation tools. A specific embodiment of the method includes: determining a potential trigger for a pre-trained target model with a backdoor, wherein the output of the target model for sample data carrying a backdoor trigger is a target label, and the potential trigger has a target A function of the model outputting a target label; in response to determining that the potential trigger and the backdoor trigger satisfy a preset condition, determining the target model as a model for evaluation; based on the model for evaluation and preset sample data for evaluation, using the model interpretation tool to be evaluated Generate salience information, in which a backdoor trigger is added to the sample data for evaluation, and the saliency information is used to show the impact of each part of the sample data for evaluation on the output of the model for evaluation; according to the saliency information and the backdoor trigger, determine the evaluation Index, where the evaluation index is used to evaluate the model interpretation tool.

Description

technical field [0001] The embodiments of this specification relate to the technical field of machine learning, and in particular to an evaluation method and device for model interpretation tools. Background technique [0002] At present, machine learning is widely used in various fields. In essence, a machine learning model uses a complex function to fit the relationship between data and targets. A machine learning model is very different from some simple rules, which specify the relationship between data and targets, but a machine learning model is a black box with only input and output. People often cannot understand the decision-making process and basis of the machine learning model, so they cannot truly trust the model. In order to explain machine learning models, researchers at home and abroad have conducted a lot of research on the interpretability of machine learning models, and proposed a variety of model interpretation tools for explaining machine learning models....

AI Technical Summary

Problems solved by technology

However, there is currently a lack of practical performance evaluation methods to judge the pros and cons of a model interpretation tool, which poses great difficulties to the interpretability research of machine learning models.

Images