An interpretability calculation method for continuous learning of image recognition based on Shapley value

Abstract

The invention discloses a method for calculating the interpretability of continuous learning for image recognition based on Shapley values, which belongs to the field of machine learning. Under the paradigm of continuous learning for image recognition, in order to explain the decision-making of the model and understand its dependence on different features, the present invention introduces the SHAP method, performs feature attribution analysis on the shallow convolution of the network, and defines a new metric for quantifying changes in model knowledge. This metric combines the analysis of SHAP values, enabling developers to deeply understand the decision-making process of the model in continuous learning. In terms of evaluating model performance, the two interpretable forgetting indicators designed by the present invention are mean‑FC and final‑FC, which are used to measure the stability of the model's decision-making changes and the degree of forgetting at the interpretable level. The introduction of these indicators enables developers to more comprehensively evaluate the performance differences of different continuous learning strategies on different data sets, thereby better discovering potential problems of the model.

Description

Technical Field

[0001] The present invention belongs to the field of machine learning and is mainly used to measure the interpretability of continuous learning models in the absence of manually annotated data sets. Background Art

[0002] The field of machine learning continues to evolve. Early machine learning methods, such as decision trees, support vector machines, and clustering, achieved remarkable success in specific tasks. However, these models often perform poorly when faced with new tasks or data, necessitating constant manual tuning and retraining, which limits their effectiveness in practical applications. As machine learning applications continue to expand, the need for machine learning models to continuously learn and adapt to new knowledge is becoming increasingly urgent. The dynamic nature of data and tasks requires models to learn and adapt in an ever-changing environment, rather than being trained statically once. Continuous learning (CL) models aim to overcome the forgetting problem in traditional machine learning, enabling machine learning systems to learn new knowledge more flexibly and persistently, adapting to changing environments and tasks. Continuous learning models hold enormous potential in a wide range of fields, making them an important research direction in machine learning. Their development promises to make machine learning systems more flexible and intelligent. However, many challenges remain to be overcome, such as forgetting and sample efficiency.

[0003] Explainable machine learning plays a key role in continuous learning. It helps detect and fix model performance degradation or errors, monitor model evolution, explain model decisions, manage knowledge loss, support user interaction, and meet regulatory and compliance requirements. By providing transparency and operability, explainability techniques help make continuous learning systems more stable, controllable, and trustworthy, thereby increasing their practicality and acceptability in applications. To achieve this goal, researchers currently focus on interpretability methods, mainly local interpretability techniques (such as Local Linear Approximation (LIME)) and global interpretability techniques (feature importance ranking). Local interpretability techniques allow analysis of explanations for specific decisions or predictions; for example, they can reveal why a model made a certain prediction for a particular example. Global interpretability techniques aim to understand the behavior and decision rules of the entire model. Explainable machine learning not only helps improve model credibility but also helps identify and correct bias and unfairness within the model. Furthermore, it can promote model adoption, making it easier for users to trust understandable and explainable systems. Although interpretable machine learning has made significant progress in both academia and industry, challenges remain, such as improving interpretability while maintaining performance, and achieving interpretability on large-scale and high-dimensional data. Therefore, interpretable machine learning remains an actively developing research field that will continue to drive the development of artificial intelligence to ensure that the model's decisions can be reasonably explained and understood in various fields.

[0004] However, most of the current interpretability evaluations are qualitative and lack objectivity. Bau, David, et al. proposed a method to quantitatively measure the interpretability of a model, reference: Bau, David, et al. "Network dissection: Quantifying interpretability of deep visual representations." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017. However, this method still requires a large amount of human annotation information, and commonly used datasets such as MNIST, CIFAR10, and CIFAR100 do not have these detailed dataset annotations. This makes the quantitative indicator lack generalization ability and cannot be well applied to more common datasets. SHAP, a tool proposed by Lundberg, Scott M., et al. that uses Shapley values to attribute local features of a model, does not evaluate the overall interpretability of the model, but only performs feature attribution analysis at the level of a single sample. Reference: Lundberg, Scott M., and Su-In Lee. "A unified approach to interpreting model predictions." Advances in neural information processing systems 30 (2017). Traditional indicators for evaluating continuous learning models do not evaluate the interpretability of the model, which makes the credibility gap between different strategies ignored, while the credibility of the model has increasingly become a focus of attention in some fields.

[0005] Therefore, the present invention proposes to use the Shapley value framework to conduct an overall interpretability evaluation of the continuous learning model. The indicator proposed in the present invention does not require an artificially defined interpretable data set and can be applied to a more general data set. It is a more fair and objective evaluation indicator. This indicator will be conducive to promoting the development of the interpretability of continuous learning models and measuring the credibility of continuous learning models. Summary of the Invention

[0006] This method is a quantitative measure of the interpretability of continuous learning models. It uses Shapley values to visualize the semantics of intermediate layers in the network, thereby quantifying forgetting at the model's interpretability level through changes in network semantics. This method generalizes the local attribution of Shapley values to global model interpretation, and this interpretation does not rely on manual annotation, making it applicable to various datasets and continuous learning strategies.

[0007] The present invention first trains the models separately with different continuous learning strategies under the same continuous learning scenario and model framework. After the training is completed, a unified network feature extraction layer is selected, and the output features of this layer on each sample are attributed with the Shapley value, so as to explainably determine the semantics of this level. Whenever the model completes the training of a new task, the semantics of this level are recalculated on the samples of the previous task on the new model, and the difference between this semantics and the semantics on the previous model is compared, and the degree of forgetting of the model on this task is measured by this difference. This method uses the Shapley value with a more complete theoretical basis as a tool to visualize the semantics of the network level, making the semantic results more accurate, thereby improving the rationality and credibility of the indicator.

[0008] From the perspective of the interpretability of forgetting features in continuous learning, this method mainly accomplishes the following work: the conventional indicators of the current continuous learning model are not sufficient to evaluate the interpretability of the model, and the current quantitative evaluation indicators of the model interpretability need to be trained under a specific data set, which lacks generalization. The performance of the model on different data sets may be completely different, which makes the existing evaluation methods lack credibility. The present invention proposes a new indicator based on the Shapley value, attributes the characteristics of the sample, and evaluates the interpretability of the model through the attribution results. This indicator does not depend on a specific model and has better universality. Different data sets were tested under multiple continuous learning strategies, and the present method found credibility differences that cannot be reflected by traditional indicators, which will be beneficial to improving the interpretability and credibility of the model.

[0009] In order to improve the readability of this invention, some terms are defined and explained here.

[0010] Definition 1: Shapley value is an important concept in game theory and is also widely used in machine learning and explainable artificial intelligence (XAI). It is used to measure the contribution of each player in a game to the overall game. The expression is as follows:

[0011]

[0012] in represents the set of all participating players, v represents the value function, represents the set of all possible remaining players after removing player k, and the calculated Shapley value s k The higher this value is, the greater the contribution of player k.

[0013] Definition 2: Deep residual convolutional neural network (ResNet) is an important innovation in the field of deep neural networks. It improves the performance of deep convolutional neural networks by introducing residual connections. Residual connections allow information to flow more freely in the network, solving problems such as gradient disappearance and overfitting. The advantage of this network structure is that it can build a deeper network and improve the model's fitting ability and feature extraction ability. ResNet is widely used in computer vision tasks such as image classification, object detection, and semantic segmentation. Among them, the residual neural network used in the present invention is the Resnet18 network, which is a typical network in the ResNet series.

[0014] Definition 3: Intermediate-layer features in machine learning possess high-level representational capabilities. These features gradually abstract key information from the data within a multi-layer network, from low-level features (such as edges and colors) to high-level semantic features (such as objects and scenes). The extraction of intermediate-layer features is crucial to model performance because they contain semantic information about the data and can be used for tasks such as classification, detection, and generation. The training process of deep learning models achieves superior performance by optimizing intermediate-layer features through backpropagation, making them better suited to specific tasks.

[0015] Definition 4: Class-incremental learning is a continuous learning method in which the model is required to complete the corresponding task without the guidance of the task sequence. In class-incremental learning, the task of the model is to recognize and classify previously unseen categories in each new learning stage while maintaining the ability to recognize old categories. This learning method focuses more on the sustainability of the model.

[0016] Definition 5: SHAP (SHapley Additive exPlanation) value is a method for quickly calculating Shapley values. This method greatly improves the computational efficiency of Shapley values while ensuring a certain level of computational accuracy, making it feasible for large-scale calculation of Shapley values. The present invention adopts an approximate Shapley value algorithm based on path gradient integration. This method sets a reference eigenvalue, takes the reference eigenvalue as the starting point, and the sample eigenvalue as the end point, and performs forward propagation gradient integration on each feature on the path to obtain the SHAP value attribution result for each feature.

[0017] Definition 6: Feature attribution is a key concept in the field of machine learning interpretability, aiming to reveal the reasons behind model predictions. This approach quantifies the contribution of input features to model predictions, enabling understanding of the model's decision-making process. Common feature attribution methods include LIME, SHAP, and Grad-CAM. They provide different perspectives on explaining model predictions and help identify which features are critical to the model's decisions. This is crucial for improving model credibility and decision interpretability, especially in highly complex deep learning models.

[0018] The present invention is a method for calculating the interpretability of continuous learning for image recognition based on Shapley values, the method comprising:

[0019] Step 1: Normalize the data image;

[0020] Step 2: Build a continuous learning paradigm;

[0021] In continuous learning for image recognition, a model f with parameters θ is trained on a sequence of sequential image recognition tasks 1, ..., T; each task t consists of a different dataset. It contains samples Corresponding labels and task-specific categories In each task, the model f can only access the dataset of the current task and cannot access the data of previous tasks; the goal is to train the model by minimizing the loss function, where the loss function is usually expressed as cross-entropy loss; in class incremental learning, the model needs to accurately classify without task-specific category information;

[0022] Step 3: Build a continuous learning backbone network;

[0023] The backbone network is as follows: input layer Feature extraction module f consisting of convolutional layer, pooling layer and four residual blocks θ , average pooling layer, output layer; the mapping method in the residual block is identity mapping; each residual block contains multiple convolutional layers, and the output of each residual block is the sum of the output of the last convolutional layer and the input of the residual block; finally, the output features of the output layer are passed to the fully connected layer for classification, and the output result is

[0024] Step 4: Shapley value attributes features to the middle layer of the network;

[0025] For each input image The feature representation after the convolution layer is Calculate the Shapley value of feature z. The Shapley value is the same as the dimension of the feature and is expressed as

[0026] Step 5: Define changes in network semantics;

[0027] Calculate the FC feature. FC represents the feature consistency in decision making and is used to quantify the change in model knowledge during continuous learning. The calculation formula for the FC feature is as follows:

[0028]

[0029] In the formula, represents the Shapley value of the feature component at position (i, j) on the feature map of the first convolutional layer after the model has learned the t-th task of the input image x. In order to eliminate the dimensional differences when different models calculate the Shapley value of the same sample, Min-Max normalization is performed to limit the value of feature attribution to the range of [-0.5, 0.5]; for the simplicity of the formula, FC τ，t (x) represents the decision consistency of single-channel features in the τ-th task and the t-th task, where t>τ, while for multi-channel features, the result is the average of the channel dimensions; p t (x) represents the important feature region of sample x after the continuous learning model learns the t-th task. This region will affect the classification decision of the model. In this context, the important feature region is defined as the region whose Shapley value exceeds the threshold The present invention sorts the Shapley values of the feature points in descending order on each sample and selects the 30% Shapley value as the threshold;

[0030] Step 6: Calculate the explainable forgetting index;

[0031] Two interpretable forgetting metrics are defined: mean-FC and final-FC. Mean-FC evaluates the decision stability of the image recognition model across all tasks, while final-FC evaluates the degree of forgetting at the interpretable level after learning all tasks. These two metrics help to more fully understand the behavior and stability of the model and improve the interpretability of the model. The calculation methods for the two metrics are:

[0032]

[0033]

[0034]

[0035] Among them, Π τ，tIt represents the average decision change of the image recognition model on task τ after learning the t-th task; mean-FC is calculated once for each measurement task τ after the image recognition model learns each new task, and the average value is finally taken to obtain the average decision change of the task. The value of final-FC represents the final interpretability level of forgetting of the image recognition model after completing the continuous learning task. It does not show the process of model decision change, but focuses on the final decision difference. represents the sample data for task τ;

[0036] Step 7: Use the indicators obtained in step 6 and the accuracy (A) and forgetting rate (F) to judge the performance of the image recognition model. The higher the accuracy (A), mean-FC, and final-FC values, the better the image recognition model, while the lower the forgetting rate (F), the better the image recognition model.

[0037] The innovation of the present invention is:

[0038] 1) This invention emphasizes the need to evaluate the interpretability of continuous learning models, and points out that although the current continuous learning models perform well in conventional indicators, their interpretability is still an area that has not received sufficient attention. This demand means that the success of a continuous learning model lies not only in its performance, but also in making progress in understanding and explaining the decision-making process of the model. Therefore, this method introduces a new indicator based on the Shapley value, which is used to attribute the characteristics of the samples and evaluate the interpretability of the model through these attribution results. The innovation of this new indicator is that it does not depend on a specific model and data set, has better universality, and can be applied to various types of continuous learning models.

[0039] 2) This invention tested multiple continuous learning strategies using different datasets. This diverse testing demonstrated its ability to detect differences not captured by traditional metrics, enabling a more comprehensive assessment of the model's interpretability and credibility. This invention has broad research potential and can be used to reveal credibility gaps between different continuous learning strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Block diagram of the residual convolutional neural network of the method of the present invention.

[0041] Figure 2 The structure of the residual convolutional neural network Resnet18 and the network hierarchy for calculating the Shapley value of the method of the present invention.

[0042] Figure 3 The present invention uses the Shapley value feature attribution results to visualize the credibility gap between different strategies. DETAILED DESCRIPTION

[0043] Step 1: Experimental data preprocessing;

[0044] The experimental data for this paper is based on three widely used image datasets: CIFAR-10, CIFAR-100, and TinyImageNet. The CIFAR-10 dataset contains 10 different categories, each of which includes images of various animals and vehicles, totaling 60,000 images. The CIFAR-100 dataset is more finely classified into 100 categories, including specific types of animals and everyday objects. TinyImageNet is a subset of the ImageNet dataset, containing more categories and higher resolution images. To simulate continuous learning scenarios, these datasets are divided into multiple tasks, named Split-CIFAR10, Split-CIFAR100, and Split-TinyImageNet. Each task divides the dataset into different categories, namely 2, 10, and 10 categories respectively. This task division simulates the challenges of incremental learning, in which the model needs to gradually learn new categories while retaining previously learned knowledge. All datasets are normalized before training. This preprocessing step plays a positive role in improving training stability and accelerating model convergence.

[0045] Step 2: Build a continuous learning paradigm;

[0046] In continuous learning, a model f with parameters θ is trained on a sequence of ordered tasks {1, ..., T}, each task containing a different dataset It contains samples Corresponding labels and task-specific categories In task t, the dataset D t is presented to the model, and the previous dataset will not be accessible. The objective function is:

[0047]

[0048] Where l usually represents the cross entropy loss. In Class-IL, the model needs to learn the class without task-specific Accurately classify sample x into y under the condition of . The lack of task-specific categories makes incremental learning particularly challenging in the field of continuous learning. Effectively handling incremental learning requires the development of complex strategies that effectively retain prior knowledge while adapting to acquire new information related to unknown categories.

[0049] Step 3: Build a continuous learning backbone network;

[0050] The backbone network used in this invention is the residual convolutional neural network ResNet18, whose hierarchical structure includes the input layer x∈R H×W×C , the feature extraction module f consisting of convolutional layers, pooling layers and four core residual blocks θ Among them, the residual module overcomes the gradient vanishing problem in deep networks through identity mapping. Each residual block includes multiple convolutional layers, which helps to extract high-level features. Finally, the feature extraction module f θ The extracted features are outputted through the fully connected layer to complete the classification task z∈R d ResNet-18 is known for its depth and performance, and is suitable for computer vision tasks such as image classification. The innovation of this network structure is the introduction of residual blocks, which allows the construction of very deep neural networks while maintaining the performance and training efficiency of the model. Figure 1 , the residual convolutional neural network Resnet18 structure see Figure 2 ;

[0051] Step 4: Shapley value attributes features to the middle layer of the network;

[0052] For each input sample x∈R H×W×C , the feature dimension after convolution is recorded as z∈R W×Q×K , use SHAP of Definition 5 to perform attribution calculation on the feature set. Each feature will correspond to a SHAP value, so the SHAP value is the same as the feature dimension s∈R W×Q×K In this invention, the selected network middle layer is the output feature of the first convolutional layer of ResNet18, see Figure 2 , and its visualization results can be found in Figure 3 , the choice of output features can be flexibly selected according to the depth of the network.

[0053] Step 5: Define changes in network semantics;

[0054] In this step, inspired by the Intersection-over-union score (IoU), which is commonly used to evaluate the prediction performance of semantic segmentation networks by assessing the degree of overlap between the actual label and the prediction, this method proposes a new metric called Feature Consistency in Decision-Making (FC) to quantify the change in knowledge acquired by the continuous learner throughout the continuous learning process. The calculation formula of FC is as follows:

[0055]

[0056] It represents the SHAP value of the feature component at position (i, j) on the first convolutional layer feature map after the model has learned the t-th task of the input image x. t (x) is normalized by Min-Max:

[0057]

[0058] Thus, the feature attribution value is limited to the range of [-0.5, 0.5] to eliminate the dimensional differences caused by different models when calculating the SHAP value of the same sample. τ，t (x) represents the decision consistency of single-channel features in the τ-th task and the t-th task (t>τ), while for multi-channel features, the result is the average of the channel dimensions. t (x) represents the important feature region of sample x after the continuous learning model learns the t-th task, which will affect the classification decision of the model. In this context, the important feature region is defined as the region whose SHAP value exceeds a certain threshold. The location set composed of feature points. For each sample, the present invention sorts the SHAP values of the feature points in the sample from large to small, and sets a threshold to select the feature points with the top 30% SHAP values to determine the important feature areas:

[0059]

[0060] Step 6: Design an interpretable forgetting indicator;

[0061] In this step, this method designs two indicators for measuring explainable forgetting in continuous learning, namely mean-FC and final-FC. Mean-FC evaluates the decision stability of the model in all tasks by calculating the average change in the decision of the task after the model completes learning of multiple tasks. Final-FC is used to evaluate the degree of knowledge retention of the model after learning all tasks and the final change in feature differences in the model's decision compared to the first time the model learned the corresponding tasks. These two indicators will help better understand the behavior and stability of the model in continuous learning, thereby improving the interpretability of the model. The definition of mean-FC is as follows:

[0062]

[0063]

[0064] Defined intermediate variable ∏ t，τTo represent the change in the model's decision on task τ after learning the tth task, mean-FC builds on this. For each task, the model is tested once after learning a new task. The average of these tests is taken as the mean-FC for the average decision change on that task. Final-FC is intended to ultimately measure the degree of model decision drift, without necessarily capturing the process of model decision change. Therefore, to calculate this metric, the decision difference between the model that completed all tasks and the model that first learned each task is calculated. This decision difference is then averaged over the total number of tasks compared to obtain the final model decision change. The formula is as follows:

[0065]

[0066] A larger final-FC indicates that after completing the continuous learning task, the final decision of the continuous learning changes less, which means that the continuous learning strategy has a lighter impact on the decision of the model during the continuous learning process, making it easier for the model to maintain a stable decision.

[0067] Step 7: Use this metric to compare different continuous learning strategies;

[0068] The model trained in steps 1, 2, and 3 is tested for its general performance on a class-incremental task. The network intermediate feature attribution results calculated in step 4 are used to calculate two interpretable forgetting indices, mean-FC and final-FC, and the differences between different continuous learning strategies are compared. Experiments were conducted on seven continuous learning strategies, SI, LwF, A-GEM, iCaRL, DER, DER++, and BFP, and three datasets, CIFAR-10, CIFAR-100, and TinyImageNet. The test results are shown in Tables 1 and 2, where the higher the accuracy A, mean-FC, and final-FC, the better, while the lower the forgetting rate F, the better. Judging from the results, the traditional continuous learning strategy evaluation method only reflects the differences between different strategies, and the overall trend of its indicators is basically consistent across different datasets. However, the interpretability indicator proposed by this method not only reflects the differences between strategies, but also shows different indicator trends across different datasets. Therefore, this indicator can more comprehensively evaluate the performance differences of different strategies on different datasets. In practical applications, this method can discover the credibility gaps of the model in different scenarios, making it easier for developers to identify problems.

[0069] The results in Tables 1 and 2 show that while some strategies achieve high performance in conventional metrics such as accuracy and forgetting rate, they may lack interpretability, which in turn reduces the credibility of their models. The metrics proposed in this method, starting from the interpretability perspective, enable researchers to identify model issues that conventional continuous learning metrics cannot reveal. This provides a quantitative evaluation standard for model interpretability and credibility, and makes the evaluation of continuous learning models more comprehensive.

[0070] Table 1 Experimental results of the method of the present invention on Split-CIFAR10 and Split-CIFAR100.

[0071]

[0072] Table 2 Experimental results of the present invention on Split-TinyImageNet.

[0073]

Claims

1. A method for calculating the interpretability of continuous learning for image recognition based on Shapley values, the method comprising: Step 1: Normalize the data image; Step 2: Build a continuous learning paradigm; In continuous learning for image recognition, a model f with parameters θ is trained on a sequence of sequential image recognition tasks 1, ..., T; each task t consists of a different dataset. It contains samples Corresponding labels and task-specific categories In each task, model f can only access the dataset of the current task and cannot access the data of previous tasks; The goal is to train the model by minimizing the loss function, where the loss function is usually expressed as cross-entropy loss; in class-incremental learning, the model needs to accurately classify without task-specific category information; Step 3: Build a continuous learning backbone network; The backbone network is as follows: input layer Feature extraction module f consisting of convolutional layer, pooling layer and four residual blocks θ , average pooling layer, output layer; the mapping method in the residual block is identity mapping; each residual block contains multiple convolutional layers, and the output of each residual block is the sum of the output of the last convolutional layer and the input of the residual block; finally, the output features of the output layer are passed to the fully connected layer for classification, and the output result is Step 4: Shapley value attributes features to the middle layer of the network; For each input image The feature representation after the convolution layer is Calculate the Shapley value of feature z. The Shapley value is the same as the dimension of the feature and is expressed as Step 5: Define changes in network semantics; Calculate the FC feature. FC represents the feature consistency in decision making and is used to quantify the change in model knowledge during continuous learning. The calculation formula for the FC feature is as follows: In the formula, represents the Shapley value of the feature component at position (i, j) on the feature map of the first convolutional layer after the model has learned the t-th task of the input image x. In order to eliminate the dimensional differences when different models calculate the Shapley value of the same sample, Min-Max normalization is performed to limit the value of feature attribution to the range of [-0.5, 0.5]; for the simplicity of the formula, FC τ，t (x) represents the decision consistency of single-channel features in the τ-th task and the t-th task, where t>τ, while for multi-channel features, the result is the average of the channel dimensions; p t (x) represents the important feature region of sample x after the continuous learning model learns the t-th task. This region will affect the classification decision of the model. In this context, the important feature region is defined as the region whose Shapley value exceeds the threshold A location set composed of feature points; Step 6: Calculate the explainable forgetting index; Two interpretable forgetting metrics are defined: mean-FC and final-FC. Mean-FC evaluates the decision stability of the image recognition model across all tasks, while final-FC evaluates the degree of forgetting at the interpretable level after learning all tasks. These two metrics help to more fully understand the behavior and stability of the model and improve the interpretability of the model. The calculation methods for the two metrics are: Among them, Π τ，t It represents the average decision change of the image recognition model on task τ after learning the t-th task; mean-FC is calculated once for each measurement task τ after the image recognition model learns each new task, and the average value is finally taken to obtain the average decision change of the task. The value of final-FC represents the final interpretability level of forgetting of the image recognition model after completing the continuous learning task. It does not show the process of model decision change, but focuses on the final decision difference. represents the sample data for task τ; Step 7: Use the indicators obtained in step 6 and the accuracy (A) and forgetting rate (F) to judge the performance of the image recognition model. The higher the accuracy (A), mean-FC, and final-FC values, the better the image recognition model, while the lower the forgetting rate (F), the better the image recognition model.

2. The method for calculating the interpretability of continuous learning for image recognition based on Shapley values according to claim 1, wherein: In step 5, the Shapley values of the feature points are sorted in descending order on each sample, and the 30% Shapley value is selected as the threshold.

Images