An explainability method and system for convolutional neural networks

Abstract

The application provides an explainability method and system for a convolutional neural network, and belongs to the fields of software technology, computer vision and machine learning. The application extracts non-class data of each class based on an original task data set, and constructs an irrelevant feature test data set. Based on a convolutional neural network model for image classification, an explainable model composed of a feature extractor and an explainable classifier is constructed by modifying the architecture. The original task data set and the irrelevant feature test data set are used to train the explainable model. After the training of the explainable model is completed, each test sample is processed, and a prediction result and a corresponding decision explanation result are output. The application can provide detailed attribution explanation for each input feature, and users can obtain feature information that plays a key role in classification decision, so that clear visual explanation is obtained.

Description

Technical Field

[0001] This invention belongs to the fields of software technology, computer vision and machine learning, and specifically relates to a method and system for interpreting classification models based on convolutional neural networks (CNN). Background Technology

[0002] In recent years, neural networks, especially deep neural networks (DNNs), have become an advanced technology for solving complex visual recognition problems. They have demonstrated outstanding performance in areas such as image classification, object detection, and semantic segmentation. By simulating the hierarchical structure of the human nervous system, DNNs can automatically learn image features, from simple edge detection to complex object recognition. This ability to automatically learn features from data has led to the widespread application of DNNs in many industrial and research fields, such as autonomous vehicles, medical image analysis, and natural language processing.

[0003] While DNNs boast impressive performance, their decision-making processes often lack intuitive explanation, which can pose a significant obstacle in some applications. For example, in medical diagnostics or legal decision support systems, users require not only reliable results but also an understanding of the model's reasoning process. This lack of transparency leads to the need for interpretability of neural networks—the ability to explain the reasons behind the model's output. However, existing methods often rely on post-processing steps after model training to provide explanations, which can result in inconsistent explanation quality and a lack of synchronization with the model's decisions. Furthermore, the inherent mechanisms of existing methods lack guarantees of interpretability themselves; they cannot guarantee that the explanations accurately reflect the model's decision-making process. Summary of the Invention

[0004] The purpose of this invention is to provide an innovative interpretability method by reconstructing a convolutional neural network architecture and training an interpretable model using a specially constructed test dataset. This method can provide detailed attributional explanations for each input feature, allowing users to obtain feature information that plays a key role in classification decisions, thereby obtaining a clear visual explanation.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] An interpretability method for convolutional neural networks includes the following steps:

[0007] Based on the original task dataset, extract non-category data for each category and construct an irrelevant feature test dataset;

[0008] The convolutional neural network model based on image classification is modified to construct an interpretable model consisting of a feature extractor and an interpretable classifier.

[0009] The interpretable model is trained using the original task dataset and the test dataset with irrelevant features.

[0010] After the interpretable model is trained, each test sample is processed to output the prediction result and the corresponding decision explanation result.

[0011] Furthermore, when constructing the irrelevant feature test dataset, the extracted data is randomly pruned.

[0012] Furthermore, the steps for constructing an interpretable model include:

[0013] The convolutional neural network model for image classification is decomposed into a feature extractor and a classifier, with the last layer of the feature extractor being a convolutional layer.

[0014] The classifier is made interpretable by modifying it. If the classifier is a linear layer, it remains unchanged; otherwise, it is replaced with a new linear layer whose input dimension matches the output dimension of the feature extractor. The output dimension is the number of categories, thus obtaining an interpretable classifier.

[0015] By combining the feature extractor and the interpretable classifier, an interpretable model is obtained.

[0016] Furthermore, the steps for training an interpretable model include:

[0017] Randomly select an irrelevant feature from the irrelevant feature dataset and overlay it at a random position on the training sample x to generate an interpretability evaluation sample x. exp ;

[0018] The training sample x and the interpretability evaluation sample x exp Input the data into the interpretable model for training, and calculate the interpretability loss l. exp ;

[0019] Calculate the cross-entropy loss CEL, and then calculate the cross-entropy loss CEL and the interpretability loss l. exp Optimization is performed through backpropagation, the model parameters are updated, and the optimization loss is calculated.

[0020] Furthermore, x exp = (1-m)⊙x+m⊙x′, where x′ is the corresponding sample when generating the irrelevant feature, m is the mask used to cover the irrelevant feature, the value of the irrelevant feature is 1, and the rest are 0.

[0021] Furthermore, the interpretability loss l is calculated. exp The formula is as follows:

[0022] l exp =softmax(f exp (x)) C ·Ipos +α·softmax(f exp (x exp )) C ·I neg

[0023]

[0024]

[0025]

[0026] Wherein, softmax(f exp (x)) C Represents an interpretable model f exp The probability, I, of the predicted outcome for input x belonging to class C. pos I represents the model's explanatory loss on the relevant features. neg Let m(i,j) represent the model's explanation loss on irrelevant features, and e(i,j) represent the mask and explanation results when evaluating input x, respectively. exp The values ​​of positions i and j, N is the total number of pixels in the input; upsample is used to upsample the input to the same size as x, norm means to normalize the input to between 0 and 1; h(x) represents the classifier; α represents the weight of the linear classifier h for the class C of the k-th convolutional kernel; α is the weighting coefficient.

[0027] Furthermore, the formula for optimizing the loss is as follows:

[0028] loss = CEL(f exp (x), C)+CEL(f) exp (x exp ), C)+λ·l exp

[0029] Where CEL represents the cross-entropy loss, λ is a hyperparameter, and f exp For an interpretable model, C represents the category.

[0030] Furthermore, the interpretable model processes the test sample x and outputs the prediction result softmax(f) exp (x)), where the predicted probability of category C is softmax(f exp (x)) C ; and according to The output decision interpretation result is e(x), where norm represents the normalization operation, upsample represents the upsampling operation, and h(x) represents the feature extractor. This represents the weight of the linear classifier g for the class C of the k-th convolutional kernel.

[0031] Furthermore, the decision interpretation results are presented in the form of a matrix of the same size as the input sample, with element values ​​ranging from 0 to 1.

[0032] An interpretable system for convolutional neural networks includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the steps of the method described above.

[0033] Compared with the prior art, the present invention has the following positive effects:

[0034] 1. This invention first reconstructs the convolutional neural network (CNN) model used for image classification, decomposing it into two parts: a feature extractor and an interpretable linear classifier. This architecture allows the linear classifier to provide explicit explanations based on feature attribution for any input image.

[0035] 2. This invention improves the interpretability of the model by constructing a test dataset of irrelevant features related to the task and training the model in combination with the original dataset. This strategy not only improves the transparency of the decision-making process, but also ensures that the generation of explanations is closely integrated with the model training process, thereby improving the accuracy and relevance of the explanations.

[0036] 3. This invention employs an interpretability-based design loss function to ensure that the output of the interpretive model accurately reflects the input features that are crucial to decision-making, thus laying a solid foundation for the reliability and practicality of the interpretive model.

[0037] 4. The explanation method proposed in this invention ensures the real-time nature and consistency between the explanation and the model's decision-making, enhances model transparency, and provides accurate and reliable explanation results. Through this method, users can place greater trust in the model's decisions, especially in critical application scenarios requiring model explanation, such as medical diagnosis and legal review, where this method plays a crucial role. Attached Figure Description

[0038] Figure 1 This is an overall flowchart of an interpretability method for convolutional neural networks according to the present invention.

[0039] Figure 2 This is a flowchart of the interpretable model training process. Detailed Implementation

[0040] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0041] This invention proposes an innovative interpretability method that aims to create inherently interpretable models by reconstructing the model architecture and employing specialized training techniques, such as... Figure 1 As shown, the specific steps include:

[0042] 1. Construction of the irrelevant feature test dataset: Using the existing model task dataset D, for each category, construct irrelevant features relative to other categories to form the irrelevant feature test dataset D. exp These irrelevant features should not be considered key features of interpretability methods, thus hindering the real-time evaluation of the interpretability of interpretable models.

[0043] The steps involved in constructing the irrelevant feature test dataset include:

[0044] 1.1 From the existing dataset D of the model task, for each category C, extract the data of non-C categories, denoted as DD. C ; where D C This represents data for category C.

[0045] 1.2 For category C, iterate through DD C The data in the dataset is then subjected to a random cropping operation with width w and height h to generate a test dataset of irrelevant features for category C, denoted as .

[0046] 2. Interpretable Model Construction: Based on the image classification convolutional neural network model f, its architecture is modified to transform it into a combined model of a feature extractor and an interpretable classifier, thus enabling the interpretation of model f. exp This modification allows the model to be used for subsequent self-explanatory training.

[0047] The steps for building an interpretable model are as follows:

[0048] 2.1 Select a convolutional neural network as the image classification model f to process the input image and perform classification prediction; the prediction model can use any feature extractor, and its last layer is a convolutional neural network layer.

[0049] 2.2 The model f is decomposed into a feature extractor h and a classifier g. The last layer of the feature extractor h is a convolutional layer, which is achieved by selecting the last convolutional neural network layer in model f.

[0050] 2.3 Interpretation of classifier g: If g is a linear layer, it remains unchanged; otherwise, it is replaced with a new linear layer whose input dimension matches the output dimension of the feature extractor h, with the output dimension being the number of classes. The feature extractor h is then combined with the modified classifier g to form an interpretable model f. exp ,Right now

[0051] 3. Interpretable model training: using datasets D and D exp For the interpretability model f exp Initialize the model, evaluate its interpretability in real time, optimize its performance, and improve its interpretability and classification task performance, such as... Figure 2 As shown.

[0052] The steps and loss function for training an interpretable model are as follows:

[0053] 3.1 Interpretability Evaluation Sample Generation: For each training sample x and its correct label C, from the irrelevant feature dataset... Randomly select an irrelevant feature and overlay it at a random position in sample x to generate an interpretability evaluation sample x. exp Here, sample x and x' exp Paired input model.

[0054] 3.2 Model Interpretability Evaluation: Calculation of Interpretability Loss l exp as follows:

[0055] l exp =softmax(f exp (x)) C ·I pos +α·softmax(f exp (x exp )) C ·I neg (1)

[0056]

[0057]

[0058]

[0059] Wherein, softmax(f exp (x)) C Represents an interpretable model f exp The probability, I, of the predicted outcome for input x belonging to class C. pos I represents the model's explanatory loss on the relevant features. neg Let m(i,j) represent the model's explanation loss on irrelevant features, and e(i,j) represent the mask and explanation results when evaluating input x, respectively. exp The values ​​of positions i and j, and N is the total number of input pixels; upsample is used to upsample the input to the same size as x, and normalize is used to normalize the input to between 0 and 1 using the max-min normalization method; Let represent the weights of the linear classifier g with respect to the class C of the k-th convolutional kernel. Interpretability loss l exp The explanatory loss of relevant and irrelevant features is calculated by weighting coefficient α, and the explanatory loss is adjusted by the trend change of the target probability value of category C. The larger the probability value, the larger the loss value.

[0060] 3.3 Model Parameter Update: Calculate the model's cross-entropy loss and interpretability loss, and optimize them through backpropagation. The optimized loss is as follows:

[0061] loss = CEL(f exp (x), C)+CEL(f) exp (x exp ), C)+λ·l exp (5)

[0062] Where CEL represents cross-entropy loss, and λ is a hyperparameter used to balance interpretability loss.

[0063] 4. Model Decision Explanation: After the model training is completed, for each test sample x, the model generates a corresponding decision explanation result e(x). This decision explanation result is presented in the form of a matrix of the same size as the input sample, and the element values ​​in the matrix are between 0 and 1. The larger the value, the higher the importance of the feature in the decision-making process.

[0064] The steps involved in interpreting model decisions include:

[0065] 4.1 Given the test sample x as input, output the prediction result softmax(f) exp (x)), where the predicted probability of category C is softmax(f exp (x)) C .

[0066] 4.2 Calculate and output the interpretation result e(x) according to formula (4).

[0067] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. An interpretability method for convolutional neural networks, characterized in that, Includes the following steps: Based on the original task dataset, extract non-category data for each category and construct an irrelevant feature test dataset; The convolutional neural network model based on image classification is modified to construct an interpretable model consisting of a feature extractor and an interpretable classifier. The interpretable model is trained using the original task dataset and the test dataset with irrelevant features. The steps for training an interpretable model include: randomly selecting an irrelevant feature from the irrelevant feature dataset and overlaying it onto the training samples. Generate interpretability assessment samples from random locations. ; ,in It is the corresponding sample when that irrelevant feature was generated. It is a mask used to cover irrelevant features, with the value of 1 at the location of the irrelevant feature and 0 elsewhere; training samples and interpretability assessment sample Input the data into the interpretable model for training and calculate the interpretability loss. ; Calculate interpretability loss The formula is as follows: in, Representing an interpretable model For input training samples The prediction results belong to the category The probability, Representing an interpretable model Evaluation of input interpretability samples The prediction results belong to the category The probability, This represents the model's explanatory loss on the relevant features. This represents the model's explanatory loss on irrelevant features. This indicates that the mask m is at position The value at that location, Indicates the interpretability assessment sample Corresponding explanation results In position The value at that location, Indicates training samples Corresponding explanation results In position The value at that location, It is the total number of pixels input; Used to upsample the input to the same level Same size This means normalizing the input to a range between 0 and 1; Indicates a feature extractor; Represents the linear classifier for the th Types of convolutional kernels The weights; These are weighting coefficients; Calculate cross-entropy loss According to cross-entropy loss and interpretability loss Optimization is performed through backpropagation, the model parameters are updated, and the optimization loss is calculated. ; After the interpretable model is trained, each test sample is processed to output the prediction result and the corresponding decision explanation result.

2. The method as described in claim 1, characterized in that, When constructing the irrelevant feature test dataset, the extracted data is randomly pruned.

3. The method as described in claim 1, characterized in that, The steps to build an interpretable model include: The convolutional neural network model for image classification is decomposed into a feature extractor and a classifier, with the last layer of the feature extractor being a convolutional layer. The classifier is made interpretable by modifying it. If the classifier is a linear layer, it remains unchanged; otherwise, it is replaced with a new linear layer whose input dimension matches the output dimension of the feature extractor. The output dimension is the number of categories, thus obtaining an interpretable classifier. By combining the feature extractor and the interpretable classifier, an interpretable model is obtained.

4. The method as described in claim 1, characterized in that, Optimize loss The formula is as follows: in, Represents cross-entropy loss, For hyperparameters, For interpretable models, Indicates the category.

5. The method as described in claim 1, characterized in that, Interpretable model for test samples Process the data and output the prediction results. The predicted probability for category C is ; and according to Output decision explanation results ,in This indicates a normalization operation. Indicates an upsampling operation. Indicates feature extractor, Represents the linear classifier for the th Types of convolutional kernels The weight.

6. The method as described in claim 1, characterized in that, The decision interpretation results are presented in the form of a matrix of the same size as the input sample, with element values ​​ranging from 0 to 1.

7. An interpretable system for convolutional neural networks, characterized in that, It includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the steps of the method according to any one of claims 1-6.

Images