Image recognition model misrecognition reason positioning method based on interpretability

Abstract

The invention discloses an interpretability-based misrecognition reason positioning method for an image recognition model, and relates to the technical field of interpretability of neural networks. According to the method, semantic concepts are combined with superpixels, so that provided explanations are feature information understood by the model and semantic concepts understood by human beings, and the model performance is helped to be efficiently improved; meanwhile, from the perspective of data, two aspects of sources of misrecognition positioning, namely a semantic concept of a misrecognition sample and a semantic concept of a category training set, are considered, which are embodied by optimizing a correction feature vector of the misrecognition sample, so that the misrecognition positioning accuracy is improved. Semantic concepts which have positive influence on correct classification and do not exist in misrecognition samples are learned by an image recognition model in a model activation space, and semantic concepts which have negative influence on correct classification in the misrecognition samples are deducted. On the basis, the weights and the corresponding semantic concepts after optimization of the concept activation vectors represent misrecognition reasons of the image recognition model, so that the specificity of a positioning result is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of interpretability of neural networks, and in particular to a method for locating causes of misrecognition of an image recognition model based on interpretability. Background Art

[0002] Generally speaking, neural network interpretability is about the ability to understand how a neural network model makes predictions or decisions. By improving the interpretability of the model, we can gain insight into the decision-making process within the model, reveal how the model processes input data, and help discover biases or errors in the model, thereby improving the accuracy and fairness of the model. In addition, interpretability can also make people more trusting of the model's prediction results, making them easier to accept and apply in actual scenarios. By understanding how neural networks make decisions, we can guide troubleshooting, locate model problems, and find room for improvement, thereby improving the robustness and performance of the model. Neural network interpretability also helps apply models to areas that require transparency and compliance, promote knowledge sharing and communication, and promote the sustainable development and application of artificial intelligence technology.

[0003] Deep Neural Network (DNN) models are widely used in various fields, from recommendation systems and voice assistants with low security requirements to autonomous driving, military applications, and smart medical care with high security requirements. However, despite the great success of DNN in these applications, the accuracy of its decisions still cannot meet perfect expectations. More importantly, even if a decision error occurs, the model cannot explain what factors lead to the wrong judgment, which may lead to serious and irreversible consequences. Therefore, an explainable artificial intelligence model is needed to present the wrong decision process made by the neural network model to the user in a transparent way, so as to facilitate the understanding of the reasoning basis of the system and improve its decision-making dependence.

[0004] When the model makes an incorrect decision on a sample that has not been intervened, from the perspective of data, the reasons for the model's decision error can be divided into two categories. The first category is the strong correlation information in the training set, that is, the model learns strong correlation or false correlation features with the category during the training process. The second category is the interference of non-generalized features within the sample, which are rare or abnormal in the training set. For a single sample, these two types of situations constitute a pair of binary mutually exclusive relationships, that is, the correlation binary of sample information, which unilaterally or multi-facetedly affects the model decision process. The root cause is the inaccurate learning and understanding of the model's true features. Since complex deep neural network models have different model structures and decision functions, the comprehensiveness and accuracy of interpretability are crucial. It is an important study to reveal the mutually exclusive relationships that cause model errors in a binary and visual way to provide explanations that humans trust.

[0005] In the prior art, for the location explanation of misidentification of image recognition models, heat maps or occlusion operations based on misidentified samples have become the mainstream operation. Under the premise of keeping the model's prior knowledge unchanged, the behavior pattern of the potential range of the location cause is made based on the attribute cause. Such behavior patterns can be used to locate the problem range in the misidentified samples. Counterfactual explanations are based on feature information that does not exist in the misidentified samples, providing generalization thinking of the training set.

[0006] However, from a result-oriented perspective, the explanation results of existing technologies focus on the feature analysis of misidentified samples or the generalization of training sets. For image recognition models, there is often pixel information corresponding to multiple semantics in images. From the perspective of feature analysis of misidentified samples or the generalization of training sets, there must be ambiguity in edge information in the form of heat maps or occlusions, which leads to the inability to fully and specifically explain what kind of pixel information of misidentified samples causes the misidentification results, and cannot effectively guide the performance improvement of image recognition models. At the same time, explaining the reasons for model misidentification is to guide model training to improve model performance and thus contribute to accuracy. However, previous work has ignored the semantics of the explanation, and cannot effectively guide the performance improvement of image recognition models when humans cannot understand it.

[0007] In summary, the current methods for locating the causes of misidentification in image recognition models are not comprehensive and specific enough, and lack semantic explanations, making it difficult to effectively guide the performance improvement of image recognition models in further applications. Summary of the invention

[0008] Based on this, it is necessary to provide a method for locating the cause of misidentification of image recognition models based on explainability to address the above technical problems.

[0009] The present invention adopts the following technical solutions:

[0010] The present invention provides a method for locating the cause of misrecognition of an image recognition model based on interpretability, comprising:

[0011] Step 1: obtaining misidentified samples from training samples for training an image recognition model and test samples for testing the trained image recognition model, and determining misclassification results corresponding to the misidentified samples;

[0012] Step 2: Perform superpixel segmentation on the training samples and misidentified samples respectively, and map each superpixel obtained by segmentation to a linear activation space through an image recognition model to obtain a mapping vector of each superpixel;

[0013] Step 3: Cluster the mapping vectors of each superpixel, and use each cluster obtained by clustering as a superpixel cluster corresponding to a semantic concept; determine the concept activation vector corresponding to the semantic concept based on each superpixel cluster through the concept activation vector algorithm;

[0014] Step 4: weight the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively by the initial weight, add the weighted concept activation vectors to the feature vectors obtained by extracting features from the misidentified sample by the image recognition model, and obtain the corrected feature vectors of the misidentified sample; classify the misidentified sample through the classification layer of the image recognition model according to the corrected feature vectors, and obtain the corrected classification results;

[0015] Step 5: Taking maximizing the difference between the misclassification results and the corrected classification results as the optimization goal, and taking the correlation between the weight of the concept activation vector of the semantic concept and the correct classification as the constraint, the weight of each concept activation vector is optimized and trained; according to the matching of the weight of each concept activation vector with the constraint conditions, the semantic concepts corresponding to the cause of misrecognition of the image recognition model in the training samples and the misrecognized samples are located.

[0016] Optionally, the trained image recognition model maps each superpixel obtained by segmentation to a linear activation space to obtain a mapping vector of each superpixel, specifically including:

[0017] Adjust the intensity of each RGB color channel of each superpixel obtained by segmentation to 128;

[0018] The adjusted superpixels are input into the trained image recognition model, and each superpixel is mapped to a linear activation space to obtain a mapping vector of each superpixel.

[0019] Optionally, clustering the mapping vectors of each superpixel, taking each cluster obtained by clustering as a superpixel cluster corresponding to a semantic concept, specifically includes:

[0020] The DBSCAN clustering algorithm is used to perform unsupervised division of the mapping vectors of each superpixel according to the density relationship between the mapping vectors of each superpixel; each cluster obtained by the division is regarded as a superpixel cluster corresponding to a semantic concept.

[0021] Optionally, determining the concept activation vector of the corresponding semantic concept by using a concept activation vector algorithm according to each superpixel cluster specifically includes:

[0022] For each superpixel cluster corresponding to a semantic concept, the superpixel cluster is used as a positive sample, and the superpixel that does not contain the meaning of the semantic concept is used as a negative sample;

[0023] The support vector machine is trained by using positive samples and negative samples until the classification accuracy of the support vector machine for the positive samples and the negative samples reaches a preset accuracy threshold;

[0024] The normal vector perpendicular to the optimal hyperplane of the trained support vector machine is obtained, and the normal vector is normalized, and the normalized normal vector is used as the concept activation vector of the semantic concept.

[0025] Optionally, the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample are weighted by initial weights using the following formula:

[0026] ζ(C x , C Y , W x , W Y )=C x ×W x +C Y ×W Y ;

[0027]

[0028] The corrected feature vector of the misidentified sample is classified through the classification layer of the image recognition model through the following formula to obtain the corrected classification result: alt =t[E+ζ(C x , C Y , W x , W Y )];

[0029] Among them, ζ() represents the combination of weight and concept, C x represents the concept activation vector of each semantic concept corresponding to the misrecognized sample, C Y represents the concept activation vector of each semantic concept corresponding to each training sample, W x is the weight matrix of the concept activation vectors of all semantic concepts contained in the misrecognized samples, is the weight of the concept activation vector of the i-th semantic concept contained in the misrecognized sample, is the number of all semantic concepts contained in the misidentified samples, W Y is the weight matrix of the concept activation vectors of all semantic concepts contained in each training sample, is the weight of the concept activation vector of the i-th semantic concept contained in each training sample, is the number of all semantic concepts contained in each training sample, galt is the corrected classification result of the misidentified sample, t() represents the classification layer output of the image recognition model, and E represents the feature vector of the misidentified sample.

[0030] Optionally, the optimization objective is to maximize the difference between the misclassification result and the corrected classification result, and the weight of each concept activation vector is optimized and trained based on the correlation between the concept activation vector of the semantic concept and the correct classification, specifically including:

[0031] The following formula is used to maximize the difference between the misclassification result and the correct classification result as the optimization goal, and the minimum maximization optimization method is used to restore the correct prediction of the image recognition model to the greatest extent with the least amount of semantic concepts, and the weights of the activation vectors of each concept are optimized and trained:

[0032]

[0033] G(C Y , C x , W Y , W x )=max[(g pre -g alt ), -τ];

[0034] ∈, μ1, μ2, τ ≥ 0;

[0035] The weights of the concept activation vectors of each semantic concept corresponding to the misidentified samples and each training sample are constrained by the following formula:

[0036]

[0037] in, Indicates that the concept activation vectors of semantic concepts that have a negative impact on correct classification in misidentified samples are suppressed at the feature level. Indicates that the concept activation vector of the semantic concept that has a positive impact on the correct classification but does not exist in the misidentified samples is added to the image recognition model at the feature layer; g pre is the misclassification result corresponding to the misidentified sample, g pre and g alt The difference of is controlled by the variable factor τ, and μ1, μ2, ∈ are regularization parameters.

[0038] The present invention provides a device for locating the cause of misidentification of an image recognition model based on interpretability, comprising:

[0039] An acquisition module, used to acquire misidentified samples that are misidentified in training samples for training an image recognition model and test samples for testing the trained image recognition model, and determine misclassification results corresponding to the misidentified samples;

[0040] The segmentation module is used to perform superpixel segmentation on the training samples and the misidentified samples respectively, and map each superpixel obtained by segmentation to a linear activation space through an image recognition model to obtain a mapping vector of each superpixel;

[0041] The concept activation module is used to cluster the mapping vectors of each superpixel, and each cluster obtained by clustering is a superpixel cluster corresponding to a semantic concept; the concept activation vector corresponding to the semantic concept is determined according to each superpixel cluster through the concept activation vector algorithm;

[0042] A weight module is used to weight the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively by using the initial weight, add the weighted concept activation vectors to the feature vectors obtained by extracting the features of the misidentified sample by the image recognition model, and obtain the corrected feature vectors of the misidentified sample; and classify the misidentified sample through the classification layer of the image recognition model according to the corrected feature vectors of the misidentified sample to obtain the corrected classification results;

[0043] The cause localization module is used to optimize the weight of each concept activation vector with the maximization of the difference between the misclassification result and the corrected classification result as the optimization goal, and the correlation between the weight of the concept activation vector of the semantic concept and the correct classification as the constraint; the semantic concept of the cause of misrecognition of the corresponding image recognition model in the training samples and the misrecognized samples is located according to the matching of the weight of each concept activation vector with the constraint conditions.

[0044] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0045] In order to improve the semantic understanding of positioning factors, the present invention combines semantic concepts with superpixels. The model uses the features of superpixels to locate which features correspond to the cause of misidentification. At the same time, the semantic concepts corresponding to superpixels can be used to locate which semantic concepts correspond to the cause of misidentification. The misidentification causes that meet the positioning requirements are both feature information understood by the model and semantic concepts understood by humans, which helps to improve the model performance efficiently.

[0046] In order to comprehensively locate the causes of misidentification, the present invention takes into account two sources of misidentification location from a data perspective, namely, the semantic concepts of the misidentified samples and the semantic concepts of the category training set. This is reflected in that by optimizing the corrected feature vectors of the misidentified samples, the semantic concepts learned by the image recognition model that have a positive impact on the correct classification but do not exist in the misidentified samples are incorporated into the model activation space, and the semantic concepts in the misidentified samples that have a negative impact on the correct classification are deducted. Based on this, the optimized weights of each concept activation vector and the corresponding semantic concepts characterize the causes of misidentification by the image recognition model, thereby achieving the specificity of the positioning results.

[0047] This improves the comprehensiveness, specificity and explainability of the method for locating the causes of misrecognition of image recognition models, enables users to understand the causes of misrecognition of image recognition models more accurately, and can effectively guide the performance improvement of image recognition models in further applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0049] Figure 1 A schematic diagram of a flow chart of a method for locating causes of misidentification of an image recognition model based on interpretability provided by the present invention;

[0050] Figure 2 A schematic diagram of an image recognition model provided by the present invention that is misclassified due to data generalization reasons;

[0051] Figure 3 A schematic diagram of a semantic concept library provided by the present invention;

[0052] Figure 4 A schematic diagram of a recovery model prediction process provided by the present invention;

[0053] Figure 5 A schematic diagram of the results of explaining the strong correlation provided by the present invention;

[0054] Figure 6 A schematic diagram of the results of explaining low connectivity provided by the present invention;

[0055] Figure 7 A schematic diagram of a device for locating causes of misidentification of an image recognition model based on explainability provided by the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0057] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0058] Figure 1 The present invention is a flowchart of a method for locating the cause of misidentification of an image recognition model based on interpretability, which specifically includes the following steps:

[0059] S101: Obtain misidentified samples that are misidentified in training samples for training an image recognition model and test samples for testing the trained image recognition model, and determine misclassification results corresponding to the misidentified samples.

[0060] S102: Perform superpixel segmentation on the training samples and the misidentified samples respectively, and map each superpixel obtained by the segmentation to a linear activation space through an image recognition model to obtain a mapping vector of each superpixel.

[0061] S103: Clustering the mapping vectors of each superpixel, taking each cluster obtained by clustering as a superpixel cluster corresponding to a semantic concept; and determining a concept activation vector corresponding to the semantic concept according to each superpixel cluster by a concept activation vector algorithm.

[0062] S104: weighting the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively by the initial weight, adding the weighted concept activation vectors to the feature vectors obtained by extracting features of the misidentified sample by the image recognition model, and obtaining the corrected feature vectors of the misidentified sample; classifying the misidentified sample through the classification layer of the image recognition model according to the corrected feature vectors of the misidentified sample, and obtaining the corrected classification results.

[0063] S105: Taking maximizing the difference between the misclassification result and the corrected classification result as the optimization goal, taking the correlation between the weight of the concept activation vector of the semantic concept and the correct classification as the constraint, the weight of each concept activation vector is optimized and trained; according to matching the weight of each concept activation vector with the constraint condition, the semantic concept corresponding to the cause of misrecognition of the image recognition model in the training samples and the misrecognized samples is located.

[0064] For the convenience of explanation, the following description is only based on the server as the execution subject. The server mentioned in the present invention can be a server set up on the business platform, or a device such as a desktop computer, a notebook computer, etc. that can execute the solution of the present invention.

[0065] Generally, in the process of training an image recognition model through sample images, the image recognition model can learn the mapping relationship between the image features of the sample images and the target to be recognized, so that it can be used to classify and recognize the image to be recognized.

[0066] However, when testing or applying trained image recognition models, the accuracy rate often cannot reach 100%. This is usually because in the process of learning sample images, the image recognition model inevitably learns feature information from the image to the classification result that may cause misidentification. Figure 2 A schematic diagram of misclassification of an image recognition model in the present invention due to data generalization reasons.

[0067] Based on this, when exploring the causes of misrecognition of image recognition models based on interpretability, we can first define the basic units of explanation, specifically, semantic concepts that humans can understand. Using visual concepts that humans understand as explanations of feature information learned by the model is the basis for semantic understanding information. For example, a group of meaningful image blocks containing "snow" represents the concept of "snow". Concepts that humans understand are easier to understand and accept because these concepts are associated with everyday experience and knowledge, making the explanation more intuitive and natural. Using concepts familiar to humans to explain neural network decisions can enhance the credibility and transparency of the model, making it easier for people to trust the model's decision-making process and avoiding information barriers.

[0068] For example, if the concepts are defined as "dog", "living room", etc., then training samples and corresponding category training sets for training the image recognition model to classify "dog" can be obtained; or training samples and corresponding category training sets for training the image recognition model to classify "living room". Of course, other corresponding semantic concepts can also be determined based on superpixels corresponding to other semantic concepts included in the training samples, which can be determined according to needs, and the present invention does not limit this.

[0069] Therefore, in one or more embodiments of the present invention, the server of the service platform may obtain training samples for training the image recognition model and misidentified samples in the test samples for testing the trained image recognition model. The detailed steps are as follows:

[0070] The superpixel segmentation (Simple Linear Iterative Clustering, SLIC) algorithm is used to obtain the superpixels of training samples and the superpixels of misidentified samples respectively.

[0071] The SLIC algorithm reduces the number of pixels that need to be processed by clustering pixels, thereby reducing computational complexity. The spatial position relationship between pixels is taken into account during the segmentation process, which helps to retain the edge information of the image and produce a more continuous and natural segmentation result. The superpixels generated by the SLIC algorithm are usually more compact than traditional pixel-level segmentation, reducing redundant information and facilitating subsequent image analysis and processing. While maintaining the compactness of the segmentation, the SLIC algorithm can better capture the texture and color information in the image, thereby improving the quality and accuracy of the segmentation.

[0072] By using the superpixel partitioning algorithm SLIC to automatically extract superpixels for acquiring concepts, the problems of fuzzy edge information and poor segmentation quality in conventional image segmentation algorithms can be solved.

[0073] Taking the training of image recognition model to classify "dog" as an example, the training sample set X1 and misidentified sample x1 used to train the image recognition model can be obtained. The color space values ​​{M, a, b} of the training samples and misidentified samples and the color space value {M θ ,a θ ,b θ}Calculate the color distance of the sample Spatial distance Iterative optimization is performed to obtain meaningful superpixels. For superpixel division, the number of superpixel divisions can be uniformly set to 120. This is only an example, and the specific number can be determined according to needs, and the present invention does not limit this.

[0074] The server can use the mapping vector of the superpixel block in the model feature space to cluster and divide it into multiple different superpixel clusters.

[0075] Specifically, the server may first adjust the intensity of each RGB color channel of each superpixel obtained by segmentation to 128. Then, each adjusted superpixel is input into the trained image recognition model, and each superpixel is mapped to a linear activation space to obtain a mapping vector of each superpixel.

[0076] The server can use the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) clustering algorithm to perform unsupervised partitioning of the mapping vectors of each superpixel according to the density relationship between the mapping vectors of each superpixel, so as to regard each cluster obtained by the partition as a superpixel cluster corresponding to a semantic concept.

[0077] The present invention uses the DBSCAN algorithm to cluster information in the feature space of the image recognition model with decision errors. The traditional Euclidean distance as a similarity index may be disturbed by meaningless outliers, and the division based on the neural network will increase the amount of calculation.

[0078] The present invention maps all superpixels to the activation space through the trained image recognition model, obtains the mapping vector, and then uses the density relationship between different mapping vectors to cluster similar superpixels into superpixel clusters corresponding to the same semantic concept. Since the density of DBSCAN can be calculated in a density-related manner, the density-based method helps to eliminate meaningless outliers. After clustering, each superpixel cluster obtained corresponds to a different semantic concept.

[0079] Afterwards, the server can obtain the concept activation vector of the corresponding semantic concept based on each superpixel cluster through the concept activation vector algorithm (Testing with Concept Activation Vectors, TCAV). Specifically, the server can use the superpixel cluster corresponding to each semantic concept as a positive sample and the superpixel that does not contain any semantic concept meaning as a negative sample. The support vector machine is trained with positive and negative samples until the classification accuracy of the support vector machine for positive and negative samples reaches a preset accuracy threshold. In this way, the normal vector perpendicular to the optimal hyperplane of the trained support vector machine is obtained, and the normal vector is normalized, and the normalized normal vector is used as the concept activation vector of the semantic concept.

[0080] The TCAV technology is used to associate the semantic concepts understood by humans with the feature information learned by the model. The support vector machine is trained by setting positive and negative samples to obtain the normal vector perpendicular to the optimal hyperplane, namely the concept activation vector.

[0081] Among them, the positive samples are the superpixel clusters obtained by the above clustering; the negative samples can be random superpixels that do not contain the meaning of the semantic concept; for each superpixel cluster, the normal vector perpendicular to the optimal hyperplane of the support vector machine is obtained, and the normal vector is normalized to obtain the concept activation vector of the superpixel cluster.

[0082] For example, for each semantic concept, in order to ensure the balance between independence and connectivity between semantic concepts, superpixels of training samples and misidentified samples are collected. As a positive sample, is one of the superpixel blocks; a random superpixel without conceptual meaning As a negative sample, A binary classifier is trained using IXY and PIXr, and a support vector machine is used to obtain the normal vector perpendicular to the optimal hyperplane of the support vector machine, and the normal vector is normalized to obtain the concept activation vector cav of the concept. i ∈R n (i∈{1, 2, 3, ..., n c}). Follow the decision function The accuracy of superpixels with real meaning is verified and fine-tuned to meet the accuracy rate of more than 90%, where g i is the Lagrange multiplier, L i is the positive and negative label, K is the linear kernel function, k is the intercept of svm, p x is the feature vector of the unlabeled sample, p i x is the feature vector of the i-th training sample.

[0083] The server may weight the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively through the initial weight.

[0084] Here, the full semantic concepts can be defined based on each superpixel cluster obtained by clustering, and the corresponding concept activation vectors are input into the linear feature layer of the network with learnable weights.

[0085] Among them, the concept of full volume can be divided into two dimensions: the overall training sample dimension and the misidentified sample dimension. The concepts of the two dimensions are managed in the form of libraries.

[0086] Additional learnable weights are applied to the linear layer of the network to enable it to act as a perturbation intervention model's predictions. The steps are as follows:

[0087] First define the learnable weights as follows:

[0088] Among them, W x is the weight matrix of the concept activation vectors of all semantic concepts contained in the misrecognized samples, is the weight of the concept activation vector of the i-th semantic concept contained in the misrecognized sample, is the number of all semantic concepts contained in the misidentified samples, W Y is the weight matrix of the concept activation vectors of all semantic concepts contained in each training sample, is the weight of the concept activation vector of the i-th semantic concept contained in each training sample, is the number of all semantic concepts contained in each training sample.

[0089] The combination of weights and concepts is then applied to the linear feature space of the model. The mathematical model is as follows: alt =t[E+ζ(C x , C Y , V x , W Y )],ζ(C x , C Y , W x , V Y )=C x ×W x +C Y ×V Y .

[0090] Among them, g alt is the corrected classification result of the misidentified sample, t() represents the classification layer output of the image recognition model, E represents the feature vector of the misidentified sample, ζ() represents the combination of weight and concept, and C xrepresents the concept activation vector of each semantic concept corresponding to the misrecognized sample, C Y Represents the concept activation vector of each semantic concept corresponding to each training sample.

[0091] Then, an optimization algorithm is used to maximize the category confidence difference and learn the weight distribution, so as to locate the cause of misidentification from a data perspective.

[0092] First, use the FISTA algorithm to optimize the maximum category confidence difference and learn the weight distribution, as follows

[0093]

[0094] G(C x , C y , W x , W y )=max[(g pre -g alt ), -τ];

[0095] ∈, μ1, μ2, τ ≥ 0;

[0096] In the formula, g pre is the misclassification result corresponding to the misidentified sample, g pre and g alt The variance of the scores is controlled by the variable factor τ. μ1, μ2, ∈ are regularization parameters that attempt to construct a new embedding vector and return it to the original model, learning and optimizing the constrained weights until the correct prediction of the model is restored.

[0097] When initializing the relevant weights and summarizing the weight learning process, the learning range of the relevant weights can be controlled by defining and utilizing the discrete limits of strong correlation and low correlation.

[0098] For strong correlation: if there are a large number of certain semantic concepts in the sample set, then it can be said that the semantic concept has a strong correlation with the target to be identified; for low correlation: if there are special concepts in the misidentified samples, then such concepts are not learned enough during the training process. The discrete weight learning range is set as follows:

[0099]

[0100] in, Indicates that the concept activation vectors of semantic concepts that have a negative impact on correct classification in misidentified samples are suppressed at the feature level. It means that the concept activation vector of the semantic concept that has a positive impact on the correct classification but does not exist in the misidentified samples is added to the image recognition model at the feature level; the semantic concepts are divided into three categories. The first category is negative factors. This type of semantic concept exists in the misidentified samples. This type of semantic concept may have a negative effect on the current prediction. It is meaningful to deduct it at the feature level. The second type of semantic concept is irrelevant concept. This type of semantic concept does not significantly help the prediction. The third category is positive concept, which exists in the semantic concept library of the training sample. This type of semantic concept may play a positive role in the prediction. Adding it at the feature level can positively promote the prediction.

[0101] In one or more embodiments of the present invention, considering binary computing, the source of explanation comes from a pair of mutually exclusive environments, namely, the strongly related concepts in the model training set and the ungeneralized concepts in the sample. For the former, the set of strongly related concepts, i.e., the sample category concept library C Y , for the latter, it exists in the sample ungeneralized concept set, namely the sample concept library C x For example, strong or weak correlations may lead to wrong conclusions or misleading insights, for example, a bed is strongly correlated with a bedroom and weakly correlated with a living room. By analyzing strong and weak correlations, we can avoid looking for associations in the data that appear to be related but are not actually causally related, which can lead to overfitting of the model or poor generalization. By eliminating false correlations, we can improve the robustness and generalization of the model.

[0102] Finally, according to the matching of the weight distribution information with the meaning of the optimized constraints, the semantic concepts corresponding to the reasons for the misidentification of the image recognition model in the training samples and the misidentified samples are located. The final optimization result is specifically reflected in the superpixel blocks corresponding to the concept activation vectors with non-zero weight values ​​obtained according to the optimized weights, which matches the meaning of the optimized constraints and finally locates the reasons for the misidentification of the model.

[0103] Based on the results of the explanation, the concepts are subdivided into strongly correlated concept sets and low-correlation concept sets. The concepts in the strongly correlated set can be used to detect the false correlation distribution in the training set; the concepts in the low-correlation set can be used to detect the low-connection concepts in the sample; the generalization of the data is analyzed in a binary way and a visual semantic cause location is provided, which helps improve the model and enhance the accuracy of the model.

[0104] Figure 3 is a schematic diagram of a semantic concept library in the present invention, Figure 3 The existence of the semantic concept library is shown in two dimensions: the dimension of error samples and the dimension of training sets. Analyzing the generalization of data helps people understand the relationship between data more deeply and identify and avoid misleading results that may occur during data analysis and modeling.

[0105] The present invention learns different weights of semantic concepts of training samples and misidentified samples. Figure 4 It is a flow chart of a recovery model prediction in the present invention. The fundamental purpose is to make the number of irrelevant factors as large as possible so that the decisive factors meet sparsity. The optimization algorithm FISTA is used to solve the sparse representation problem. The algorithm combines the characteristics of fast convergence and accelerates the optimization process by introducing acceleration technology. The advantages of FISTA lie in its fast convergence and sparsity processing capabilities. By introducing acceleration technology, FISTA can converge to a better solution within fewer iterative steps, and can effectively handle problems with sparse properties, such as sparse signal recovery problems in compressed sensing. This enables FISTA to find the global optimal solution or a near-optimal solution when solving convex optimization problems, with high precision and wide applicability. In addition, FISTA also shows good numerical stability and memory efficiency, making it an efficient, accurate and stable optimization algorithm for handling sparse representation problems.

[0106] Figure 5 and Figure 6 The explanatory results of the present invention are presented. Figure 5 This is a schematic diagram of the results of explaining the strong correlation in the present invention, Figure 6 This is a schematic diagram of the results of explaining weak correlation in the present invention. Figure 5 The results show that there are a large number of "snow" pixel blocks in the training set of the corresponding model, which causes the model to make wrong decisions when encountering a "dog" without "snow". Through manual observation of the training set, 83% of the samples have "snow" and the sample shooting location is Iceland. Based on the present invention, the following suggestions can be provided: It is recommended to observe the false correlation in the training set. Figure 6 The results show that the concept of "bed" in the sample has a significant impact on the decision. By manually observing the training set, it is found that the original category "living room" does not contain the concept of "bed". Based on the present invention, the following suggestions can be provided: It is recommended to analyze the special information in the picture. Comprehensively provide data generalization issues.

[0107] based on Figure 1 The method for locating the cause of misidentification of an image recognition model based on interpretability shown in the figure combines semantic concepts with superpixels for the semantic understanding of the positioning factor. The model uses the features of superpixels to locate which features correspond to the cause of misidentification. At the same time, the semantic concepts corresponding to the superpixels can be used to locate which semantic concepts correspond to the cause of misidentification. The misidentification cause that satisfies the positioning is both feature information understood by the model and semantic concepts understood by humans, which helps to efficiently improve the model performance.

[0108] In order to comprehensively locate the causes of misidentification, the present invention takes into account two sources of misidentification location from a data perspective, namely, the semantic concepts of the misidentified samples and the semantic concepts of the category training set. This is reflected in that by optimizing the corrected feature vectors of the misidentified samples, the semantic concepts learned by the image recognition model that have a positive impact on the correct classification but do not exist in the misidentified samples are incorporated into the model activation space, and the semantic concepts in the misidentified samples that have a negative impact on the correct classification are deducted. Based on this, the optimized weights of each concept activation vector and the corresponding semantic concepts characterize the causes of misidentification by the image recognition model, thereby achieving the specificity of the positioning results.

[0109] This improves the comprehensiveness, specificity and explainability of the method for locating the causes of misrecognition of image recognition models, enables users to understand the causes of misrecognition of image recognition models more accurately, and can effectively guide the performance improvement of image recognition models in further applications.

[0110] The present invention provides a comprehensive and detailed explanation based on concepts that humans understand. The key lies in creating a category concept library and constrained perturbations, combining superpixels with semantic concepts as a representation of the located causes of misrecognition, making it easier for users to understand and manage all features. Constrained perturbations reverse the model's incorrect decisions in a discrete manner, thereby clearly locating specific influencing factors. The generated explanations make specific suggestions for the generalization of model training, providing strong evidence for researchers to improve the model. This covers possible false correlations in the training set, the lack of key strongly correlated features in the samples, and the failure of the model to train the features required for generalization.

[0111] Using interpretable methods to analyze the errors in visual model decisions is an important research direction in the field of computer vision. This invention provides suggestions for the generalization ability of the model by detecting the source of errors. It can successfully explain the recognition errors of the model in the fields of autonomous driving, smart medical care, industrial production, etc., improve safety and production efficiency, and promote the progress and development of the field of computer vision.

[0112] The above is a method for locating the cause of misidentification of an image recognition model based on interpretability provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for locating the cause of misidentification of an image recognition model based on interpretability, such as Figure 7 shown.

[0113] Figure 7 A schematic diagram of a device for locating the cause of misidentification of an image recognition model based on interpretability provided by the present invention includes:

[0114] An acquisition module 201 is used to acquire misidentified samples that are misidentified in training samples for training an image recognition model and test samples for testing the trained image recognition model, and determine misclassification results corresponding to the misidentified samples;

[0115] The segmentation module 202 is used to perform superpixel segmentation on the training samples and the misidentified samples respectively, and map each superpixel obtained by the segmentation to a linear activation space through an image recognition model to obtain a mapping vector of each superpixel;

[0116] The concept activation module 203 is used to cluster the mapping vectors of each superpixel, and take each cluster obtained by clustering as a superpixel cluster corresponding to a semantic concept; and determine the concept activation vector corresponding to the semantic concept according to each superpixel cluster by using the concept activation vector algorithm;

[0117] The weight module 204 is used to weight the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively by the initial weight, add the weighted concept activation vectors to the feature vectors obtained by extracting features of the misidentified sample by the image recognition model, and obtain the corrected feature vectors of the misidentified sample; and classify the misidentified sample through the classification layer of the image recognition model according to the corrected feature vectors of the misidentified sample to obtain the corrected classification results;

[0118] The cause localization module 205 is used to optimize the weight of each concept activation vector by maximizing the difference between the misclassification result and the corrected classification result as the optimization goal, and taking the correlation between the weight of the concept activation vector of the semantic concept and the correct classification as the constraint; and locate the semantic concept of the cause of misrecognition of the image recognition model in the training samples and the misrecognized samples according to the matching of the weight of each concept activation vector with the constraint conditions.

[0119] Regarding the specific limitations of the device for locating the cause of misidentification of an image recognition model based on interpretability, please refer to the limitations of the method for locating the cause of misidentification of an image recognition model based on interpretability above, which will not be repeated here. Each module in the above-mentioned device for locating the cause of misidentification of an image recognition model based on interpretability can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

Claims

1. A method for locating the cause of misidentification of an image recognition model based on interpretability, characterized in that: include: Step 1: obtaining misidentified samples from training samples for training an image recognition model and test samples for testing the trained image recognition model, and determining misclassification results corresponding to the misidentified samples; Step 2: Perform superpixel segmentation on the training samples and misidentified samples respectively, and map each superpixel obtained by segmentation to a linear activation space through an image recognition model to obtain a mapping vector of each superpixel; Step 3: Cluster the mapping vectors of each superpixel, and use each cluster obtained by clustering as a superpixel cluster corresponding to a semantic concept; determine the concept activation vector corresponding to the semantic concept based on each superpixel cluster through the concept activation vector algorithm; Step 4: weight the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively by the initial weight, add the weighted concept activation vectors to the feature vectors obtained by extracting features from the misidentified sample by the image recognition model, and obtain the corrected feature vectors of the misidentified sample; classify the misidentified sample through the classification layer of the image recognition model according to the corrected feature vectors, and obtain the corrected classification results; Step 5: Taking maximizing the difference between the misclassification results and the corrected classification results as the optimization goal, and taking the correlation between the weight of the concept activation vector of the semantic concept and the correct classification as the constraint, the weight of each concept activation vector is optimized and trained; according to the matching of the weight of each concept activation vector with the constraint conditions, the semantic concepts corresponding to the cause of misrecognition of the image recognition model in the training samples and the misrecognized samples are located.

2. The method for locating the cause of misidentification of an image recognition model based on interpretability as claimed in claim 1, characterized in that: The image recognition model is used to map each superpixel obtained by segmentation to a linear activation space to obtain a mapping vector of each superpixel, specifically including: Adjust the intensity of each RGB color channel of each superpixel obtained by segmentation to 128; The adjusted superpixels are input into the trained image recognition model, and each superpixel is mapped to a linear activation space to obtain a mapping vector of each superpixel.

3. The method for locating the cause of misidentification of an image recognition model based on interpretability as claimed in claim 1, characterized in that: The step of clustering the mapping vectors of each superpixel, taking each cluster obtained by clustering as a superpixel cluster corresponding to a semantic concept, specifically includes: The DBSCAN clustering algorithm is used to perform unsupervised division of the mapping vectors of each superpixel according to the density relationship between the mapping vectors of each superpixel; Each cluster obtained by division is regarded as a superpixel cluster corresponding to a semantic concept.

4. The method for locating the cause of misidentification of an image recognition model based on interpretability as claimed in claim 1, characterized in that: Determining the concept activation vector of the corresponding semantic concept by using the concept activation vector algorithm according to each superpixel cluster specifically includes: For each superpixel cluster corresponding to a semantic concept, the superpixel cluster is used as a positive sample, and the superpixel that does not contain the meaning of the semantic concept is used as a negative sample; The support vector machine is trained by using positive samples and negative samples until the classification accuracy of the support vector machine for the positive samples and the negative samples reaches a preset accuracy threshold; The normal vector perpendicular to the optimal hyperplane of the trained support vector machine is obtained, and the normal vector is normalized, and the normalized normal vector is used as the concept activation vector of the semantic concept.

5. The method for locating the cause of misidentification of an image recognition model based on interpretability as claimed in claim 1, characterized in that: The concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample are weighted by the initial weights using the following formula: ζ(C x ,C Y ,W x ,W Y )=C x ×V x +C Y ×W Y ; The corrected feature vector of the misidentified sample is classified through the classification layer of the image recognition model through the following formula to obtain the corrected classification result: alt =t[E+ζ(C x , C Y , W x , W Y )]; Among them, ζ() represents the combination of weight and concept, C x Represents the concept activation vector of each semantic concept corresponding to the misidentified sample, C Y represents the concept activation vector of each semantic concept corresponding to each training sample, and W x is the weight matrix of the concept activation vectors of all semantic concepts contained in the misrecognized samples, is the weight of the concept activation vector of the i-th semantic concept contained in the misrecognized sample, is the number of all semantic concepts contained in the misidentified samples, W Y is the weight matrix of the concept activation vectors of all semantic concepts contained in each training sample, is the weight of the concept activation vector of the i-th semantic concept contained in each training sample, is the number of all semantic concepts contained in each training sample, galt is the corrected classification result of the misidentified sample, t() represents the classification layer output of the image recognition model, and E represents the feature vector of the misidentified sample.

6. The method for locating the cause of misidentification of an image recognition model based on interpretability as claimed in claim 5, characterized in that: The optimization objective is to maximize the difference between the misclassification result and the corrected classification result, and the weight of each concept activation vector is optimized and trained based on the correlation between the concept activation vector of the semantic concept and the correct classification, specifically including: The following formula is used to maximize the difference between the misclassification result and the correct classification result as the optimization goal, and the minimum maximization optimization method is used to restore the correct prediction of the image recognition model to the greatest extent with the least amount of semantic concepts, and the weights of the activation vectors of each concept are optimized and trained: G(C Y ,C x ,W Y ,W x )=max[(g pre -g alt ),-τ]; ∈, μ1, μ2, τ ≥ 0; The weights of the concept activation vectors of each semantic concept corresponding to the misidentified samples and each training sample are constrained by the following formula: in, Indicates that the concept activation vectors of semantic concepts that have a negative impact on correct classification in misidentified samples are suppressed at the feature level. Indicates that the concept activation vector of the semantic concept that has a positive impact on the correct classification but does not exist in the misidentified samples is added to the image recognition model at the feature layer; g pre is the misclassification result corresponding to the misidentified sample, g pre and g alt The difference of is controlled by the variable factor τ, and μ1, μ2, ∈ are regularization parameters.

7. A device for locating the cause of misidentification of an image recognition model based on interpretability, characterized in that: include: An acquisition module, used to acquire misidentified samples that are misidentified in training samples for training an image recognition model and test samples for testing the trained image recognition model, and determine misclassification results corresponding to the misidentified samples; The segmentation module is used to perform superpixel segmentation on the training samples and the misidentified samples respectively, and map each superpixel obtained by segmentation to a linear activation space through an image recognition model to obtain a mapping vector of each superpixel; The concept activation module is used to cluster the mapping vectors of each superpixel, and each cluster obtained by clustering is a superpixel cluster corresponding to a semantic concept; the concept activation vector corresponding to the semantic concept is determined according to each superpixel cluster through the concept activation vector algorithm; A weight module is used to weight the concept activation vectors of each semantic concept corresponding to the misidentified sample and each training sample respectively by using the initial weight, add the weighted concept activation vectors to the feature vectors obtained by extracting the features of the misidentified sample by the image recognition model, and obtain the corrected feature vectors of the misidentified sample; and classify the misidentified sample through the classification layer of the image recognition model according to the corrected feature vectors of the misidentified sample to obtain the corrected classification results; The cause localization module is used to optimize the weight of each concept activation vector with the maximization of the difference between the misclassification result and the corrected classification result as the optimization goal, and the correlation between the weight of the concept activation vector of the semantic concept and the correct classification as the constraint; the semantic concept of the cause of misrecognition of the corresponding image recognition model in the training samples and the misrecognized samples is located according to the matching of the weight of each concept activation vector with the constraint conditions.

Images