Method and system for characterization diagnosis and explanation, model comparison and training sample collection facing black box model

A model and black box technology, applied in computational models, neural learning methods, biological neural network models, etc., can solve problems such as the inability to objectively and concisely explain the internal logic of the black box model, the complexity of the explanation, and the inability to explain the degree of interpretability.

CN114462291APending Publication Date: 2022-05-10SHANGHAI JIAO TONG UNIV
0 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2022-05-10

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention relates to the technical field of machine learning, and discloses a black box model-oriented representation diagnosis, model comparison and training sample collection method and system, which can realize automatic explanation of internal logic of a black box model to obtain an AND-OR graph model representing the internal logic of the black box model, and improve the accuracy of the internal logic of the black box model. Therefore, black box model-oriented characterization diagnosis, model comparison and training sample collection are realized. The method comprises the following steps: providing a black box model; inputting a sample into the black box model, wherein the sample comprises an input unit with a certain dimension; based on the middle-layer output characteristics of the black box model, modeling is carried out on the interaction cooperation effect between the input units of the sample, the interaction strength of combinations formed by the input units is calculated, and the black box model is expressed as the'and plus relationship 'between the combinations of the input units; the reference value on each input unit in the interactive calculation is optimized, so that the expression of the'and addition relationship 'obtained from the black box model is more concise; and establishing an AND-OR graph model for expressing the internal logic of the black box model based on the relationship.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the technical field of machine learning, in particular to a method and system technology for black-box model-oriented representation diagnosis, model comparison, and training sample collection. Background technique

[0002] At present, many models with black-box properties have shown strong performance in various fields such as images and texts, but its black-box properties still restrict its wide application in many sensitive fields. Many studies have started to focus on the interpretability of black-box models. However, on the one hand, these techniques lack objectivity and cannot account for the degree of explainability, and on the other hand, there is a problem that the explanation is too complicated. That is, the existing technology cannot explain the internal logic of the black box model objectively and concisely.

[0003] Therefore, it is an urgent problem to explain the internal logic of the black-box model objectivel...

Examples

Embodiment

[0114] The first embodiment of the present invention relates to a method for explaining a black-box model, the process of which is as follows figure 1 As shown, the method includes the following steps:

[0115] In step 101: provide a black-box model to be explained.

[0116] Any black-box model with input and output can be used as the input original black-box model to be explained in the present invention, such as but not limited to convolutional neural network, deep neural network, etc. The present invention does not limit the internal structure of the black-box model, that is, the black-box model of the present invention can adopt various internal structures.

[0117] Afterwards, enter step 102: input a certain sample into the black-box model to be interpreted, the sample contains features of a certain dimension. The sample can be any data that fits the input of the black box model to be interpreted. Optionally, the sample is a tabular dataset, an image dataset, or a text...