Black box classification interaction intervention method, electronic equipment and storage medium
By introducing concept bottleneck models and unsupervised concept extraction methods into deep learning image classification networks, combined with knowledge distillation technology, the concerns about reliability and decision-making security of deep learning image classification networks are solved, and efficient error recognition and correction are achieved , improves the interpretability and performance of the model.
Patent Information
- Application Number
- CN202411555728.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-11
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Due to the increase in parameter quantity and depth, deep learning image classification networks have caused users to worry about their reliability and decision security, and existing solutions suffer serious performance losses when improving interpretability.
A black box classification interactive intervention method is proposed, using the concept bottleneck model to confuse the local approximation of the black box classification network, combined with the deep learning neural network and the unsupervised concept extraction method, connecting the characteristics of the middle layer of the black box classification network with human-care Understand the concepts and transfer error correction knowledge through knowledge distillation.
This method ensures training speed and reliability of discovering error recognition patterns in black box classification networks, saving the time and labeling cost of training additional concept mapping functions. Users can clearly and accurately discover the error recognition patterns in black box models and modify the concepts by modifying the concepts. Weight matrix corrects wrong decision mode.
Smart Images
Figure CN119418121A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image classification, and in particular to a black box classification interactive intervention method, electronic equipment and storage medium. Background Art
[0002] With the development of deep learning algorithms, more and more image classification network architectures have emerged. These networks can generally achieve performance indicators far exceeding those of traditional methods in the context of related tasks. However, along with the performance improvement comes the geometric growth of the number of network model parameters and the depth and width. At the same time, it also raises users' concerns about its reliability and decision-making security.
[0003] Currently, one of the solutions is to start from the interpretable network itself, so that the decision-making reasoning structure of the network itself can be understood by humans, but this inevitably brings about performance loss. Summary of the invention
[0004] In order to solve the above technical problems, the present invention proposes a black box classification interactive intervention method, an electronic device and a storage medium.
[0005] The technical solution to achieve the purpose of the present invention is:
[0006] In a first aspect, the present invention provides a black box classification interactive intervention method, which comprises the following steps:
[0007] Analyze the dataset of the classification task to generate concept bottlenecks through deep learning neural networks;
[0008] Using a black box classification network trained by the data set, performing prediction and testing on the data set, generating a category confusion matrix based on confusion categories, using a depth-first search algorithm to traverse all confusion category combinations, sorting the confusion category combinations by confusion scores, selecting the confusion category combination with the highest confusion score and using it as the target confusion category combination;
[0009] Based on the concept bottleneck and the target confusion category combination, construct a concept bottleneck model;
[0010] Selecting object image samples correctly classified by the original black box classification network from the training set of the data set, randomly cropping the object image samples and inputting them into the original black box classification network and processing them using an unsupervised concept extraction method to obtain visual concepts of the intermediate layer of the input samples;
[0011] Taking the visual concept as input, using the concept bottleneck model to perform local distillation approximation on a black box classification network trained by the dataset, and outputting an interpretable visual concept;
[0012] performing the explainable visual concept intervention based on the prediction explanation of the concept bottleneck model on the data set;
[0013] The concept bottleneck model after concept intervention and the frozen copy of the original black-box classification network are used as teacher models, and the original black-box classification network is reversely distilled on the dataset to transfer the error correction knowledge.
[0014] In some embodiments, the specific steps of generating the concept bottleneck are as follows:
[0015] Step S11, using LLMs according to prompt templates corresponding to multiple categories in the data set, to generate multiple concept candidates for each category;
[0016] In step S12, all categories are first preliminarily screened, and the image embedding of each category is obtained using the image encoder. Then, all concept candidates of each category are processed by the text encoder and embedded into the same vector space. The mean similarity score between the concept embedding in each category and all image embeddings under the corresponding category is calculated to determine the final screening score of each concept candidate under the category.
[0017] Step S13, arranging the concept candidates under each category in descending order according to the screening scores, taking the first multiple concept candidates of each category, and merging the concept candidates of the taken category to form a concept bottleneck.
[0018] In some embodiments, the confusion score is obtained by dividing the sum of the average values of the confusion samples between the nodes in each confusion category combination by the total number of confusion category combinations.
[0019] In some embodiments, the specific steps of generating the visual concept are as follows:
[0020] Step S41, selecting object picture samples correctly classified by the original black box classification network from the training set of the data set as image concept candidates;
[0021] Step S42, for each type of image in the image concept candidate, randomly crop it and input it into the original black box classification network, and obtain the feature matrix of the middle layer, use the non-negative matrix decomposition method to decompose the feature matrix and set the number of feature bases, obtain the feature basis matrix and its projection matrix, and after algorithm iteration of the non-negative matrix decomposition method, obtain multiple groups of feature bases, each group of feature basis matrices can restore the image block concept of each category to the greatest extent through linear combination, that is, the visual concept of the middle layer of the input sample.
[0022] In some embodiments, generating the visual concept further comprises the following steps:
[0023] Step S43, for any complete object sample image input, the non-negative matrix decomposition can be restored by using the retained feature basis matrix of the corresponding output prediction category and the feature matrix of the middle layer of the black box classification network to obtain the image block concept corresponding to the input sample.
[0024] In some embodiments, the concept bottleneck model is used to perform local distillation approximation on a black box classification network trained by the data set, and the specific steps are as follows:
[0025] Using the original black box classification network as the teacher model, the constructed concept bottleneck model is locally distilled on the validation set in the dataset according to the selected confusion categories. The distillation loss is as follows:
[0026] Loss D-cbm =MSE(P cbm ,P org [mix_classes)
[0027] Among them, MSE is the mean square error loss function, is the prediction output of the concept bottleneck model, P org is the prediction output of the original black-box classification network, P org [mix_classes] means to intercept the prediction results of the selected confusion category, mix_classes is a length of N mix A list of category indexes.
[0028] In some embodiments, the interpretable visual concept intervention is performed based on the prediction explanation of the concept bottleneck model on the data set, and the specific steps are as follows:
[0029] According to the prediction results of the validation set of the local approximated concept bottleneck model in the data set, the category gradient activation vectors and of the true positive samples of the selected confusion category and the category gradient activation vectors and of the false positive samples are calculated, and the category gradient activation vectors and of the true positive samples are arranged in ascending order, and the category gradient activation vectors and of the false positive samples are arranged in descending order. Then, the number of intervention concepts is set, and the weights of the first predetermined concepts in the activation vector and are reset to 0; for the selected confusion category, the cumulative concept contribution scores of the correctly classified concepts and the cumulative concept contribution scores of the incorrectly classified concepts into other categories are calculated respectively; the matching score vector of the image block concept extracted by the middle layer of the original black box classification network and the concept bottleneck is determined by the image and text encoder, wherein the image block concept is the visual concept of the middle layer of the input sample.
[0030] In a second aspect, the present invention provides an electronic device. The electronic device comprises:
[0031] one or more processors;
[0032] A memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the black-box classification interactive intervention method of the first aspect and any one of the first aspects.
[0033] In a third aspect, the present invention provides a computer-readable storage medium, which includes a computer program, and when the computer program is executed on an electronic device, the electronic device executes the black-box classification interactive intervention method according to the first aspect and any one of the first aspects.
[0034] Compared with the prior art, the present invention has the following significant advantages:
[0035] (1) The present invention utilizes the concept bottleneck model to perform local approximation of the confusion categories of the black-box classification network. Compared with focusing on the recognition patterns of all categories, this method ensures the training speed and the reliability of discovering the erroneous recognition patterns of the black-box classification network.
[0036] (2) The present invention utilizes a deep learning neural network combined with an unsupervised concept extraction method to connect the features of the middle layer of the black box classification network with concepts that are understandable to humans, and eliminates the time cost and labeling cost of training additional concept mapping functions.
[0037] (3) The present invention combines the interpretability of the concept bottleneck model with the ability of human intervention in the loop and the knowledge distillation method, so that users can clearly and accurately find the wrong recognition patterns and corresponding related concepts of the black box model, and correct the wrong decision patterns by modifying the concept weight matrix, and finally use distillation to transfer the error correction knowledge to the original black box classification network. Therefore, this method can reverse the user's error correction process back to the black box for parameter optimization, not just for a single sample. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 is a flowchart of a black box classification interactive intervention method proposed in one embodiment of the present invention;
[0039] Figure 2 Schematic diagram of the interaction between the unsupervised concept extraction method and the concept bottleneck network in one embodiment of the present invention. DETAILED DESCRIPTION
[0040] The detailed description listed below is intended to describe various configurations of the technical solution of the present invention and does not represent the only configuration in which the technical solution of the present invention can be practiced. The attached drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details to provide a comprehensive understanding of the subject technology. However, the subject technology is not limited to the specific details specified herein and can be practiced using one or more other implementations. In one or more implementations, the structure and components are shown in block diagram form to avoid blurring the concepts of the subject technology.
[0041] With the development of deep learning algorithms, more and more image classification network architectures have emerged. These networks can generally achieve performance indicators that far exceed traditional methods in the context of related tasks. However, along with the performance improvement comes the geometric growth of the number of network model parameters and the depth and width. At the same time, it also raises users' concerns about its reliability and decision security. At present, one of the solutions is to start from the interpretable network itself, so that the decision reasoning structure of the network itself can be understood by people, but it inevitably brings performance loss.
[0042] In order to solve the above technical problems, an embodiment of the present invention provides a black box classification interactive intervention method, which uses a concept bottleneck model to locally approximate the black box classification network to confuse the category. Compared with focusing on the recognition mode of all categories, this method ensures the training speed and the reliability of discovering the wrong recognition mode of the black box classification network; this method uses a deep learning neural network combined with an unsupervised concept extraction method to connect the features of the middle layer of the black box classification network with the concepts that can be understood by people, and saves the time cost and annotation cost of training additional concept mapping functions; this method combines the interpretability of the concept bottleneck model and the intervention ability of people in the loop and the knowledge distillation method, so that users can clearly and accurately discover the wrong recognition mode of the black box model and the corresponding related concepts, and correct the wrong decision mode by modifying the concept weight matrix, and finally use distillation to migrate the error correction knowledge to the original black box classification network. Therefore, this method can reverse the user's error correction process back to the black box for parameter optimization, not just for a single sample.
[0043] The black box classification interaction intervention method described in this embodiment can be applied to electronic devices. In some embodiments, the electronic device can be, for example, a desktop computer, a portable computing device (such as a laptop), a smart phone, a peripheral device (such as a digital camera, a headset), a tablet device, a wearable device (such as a watch, a bracelet, etc.), etc. The electronic device can communicate with one or more external devices, and can also communicate with one or more devices that enable a user to interact with the electronic device, and / or communicate with any device that enables the electronic device to communicate with one or more other computing devices. The electronic device can also communicate with one or more networks (such as a local area network, a wide area network, and / or a public network) through a network adapter. The embodiment of the present invention does not impose any special restrictions on the specific form of the electronic device.
[0044] The electronic device includes: one or more processors, a memory and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device executes the black box classification interactive intervention method of the first aspect and any one of the first aspects.
[0045] The specific process of the black box classification interactive intervention method is described in detail below through a specific embodiment.
[0046] like Figure 1 As shown, the process of the black box classification interactive intervention method provided by the embodiment of the present invention specifically includes the following steps:
[0047] Step S1, analyzing the data set of the classification task through a deep learning neural network to generate a concept bottleneck;
[0048] In some embodiments, step S1 specifically includes:
[0049] Step S11, using LLMs to generate each category according to the prompt templates of n categories in the corresponding data set concept candidate;
[0050] The following is an example template:
[0051] 1.What's the shape of{object}?
[0052] 2. What's the color of {object}?
[0053] 3.What's the texture of {object}?
[0054] Step S12, first screen the concepts of n categories one by one, use the image data in the dataset (training set), stack them according to categories, and get the concepts of each object category after passing through the OpenAI-CLIP image encoder. image embeddings, and then LLMs are generated for each object category based on the prompt templates. Concept candidates are embedded in the same vector space using OpenAI-CLIP text encoder R d Finally, according to the object category, the average similarity score between each concept embedding in each category and all image embeddings in the corresponding category is calculated one by one, and the final screening score of each concept candidate in the category can be obtained. The screening score calculation process of the i-th concept candidate in category c is as follows:
[0055]
[0056] Where E img and E text They are the image and text encoders of OpenAI-CLIP, N img is the number of sample images under category c, represents the jth sample image under category c, represents the i-th natural language concept under category c, ⊙ represents matrix multiplication, is the screening score of the i-th natural language concept under category c.
[0057] Step S13: sort the candidate concepts in each category in descending order according to the screening scores, and take the top N concepts in each category. c / n concepts, and finally merge N object categories c These concepts constitute a conceptual bottleneck.
[0058] Step S2, using the black box classification network trained by the data set to perform prediction and testing on the data set, generating a category confusion matrix based on the confusion categories, using a depth-first search algorithm to traverse all confusion category combinations, sorting the confusion category combinations by confusion scores, selecting the confusion category combination with the highest confusion score and using it as the target confusion category combination;
[0059] In some embodiments, step S2 specifically includes:
[0060] According to the prediction results of the black box classification network trained on the data set in step S1, the confusion category is selected; the category confusion matrix M is obtained by testing the original black box classification network on the data set (validation set) n×n , n is the total number of categories of categorical data, M n×n Used to record the number of samples that are confused between categories, and set the number of confused categories N mix, use the depth-first search algorithm (DFS) to traverse all categories, the number of which is N mix The category combination, confusion score S mix The sum of the average values of the confusion samples between nodes in each combination is divided by N mix Find the average:
[0061]
[0062] Among them, i, j are the node indexes of the category, which are represented by N mix The category combination composed of categories is divided into confusion scores S mix Arrange in descending order, and the category combination with the highest score is the selected confusion category.
[0063] Step S3, construct a concept bottleneck model based on the concept bottleneck and the target confusion category combination; specifically, according to the number of concepts N in the concept bottleneck c and the number of confusion categories N selected in step S2 mix Constructed (N c ,N mix )-dimensional matrix W.
[0064] Step S4, such as Figure 2 As shown, object image samples that are correctly classified by the original black box classification network are selected from the training set of the data set, and the object image samples are randomly cropped and input into the original black box classification network and processed by an unsupervised concept extraction method to obtain the visual concept of the middle layer of the input sample;
[0065] In some embodiments, step S4 specifically includes:
[0066] Step S41, for the CRAFT method, select each type of object image sample in the data set (training set) that is correctly classified by the original black box classification network as an image concept candidate;
[0067] Step S42, for each type of image in the image concept candidate, after random cropping (image block concept), input into the original black box classification network, and obtain the feature matrix of the middle layer; use the non-negative matrix decomposition (NMF) method to decompose the feature matrix and set the number of feature bases r to obtain the feature basis matrix and its projection matrix. After the NMF algorithm is iterated, n groups of feature bases are obtained, and each group of feature basis matrices can restore the image block concept of each category to the greatest extent through linear combination (dot product projection matrix).
[0068] Step S43: for any complete object sample image input, the NMF decomposition can be restored by using the retained feature basis matrix of the corresponding output prediction category and the feature matrix of the middle layer of the black box classification network to obtain the N corresponding to the input sample. pThe concept of an image patch is the visual concept of the intermediate layer of the input sample.
[0069] Step S5, taking the visual concept as input, using the concept bottleneck model to perform local distillation approximation on the black box classification network trained by the dataset, and outputting an interpretable visual concept;
[0070] In some embodiments, step S5 specifically includes:
[0071] Using the original black box classification network as the teacher model, the constructed concept bottleneck model is locally distilled on the validation set in the dataset according to the selected confusion categories. The distillation loss is as follows:
[0072] Loss D-cbm =MSE(P cbm ,P org [mix_classes)
[0073] Among them, MSE is the mean square error loss function, is the prediction output of the concept bottleneck model, P org is the prediction output of the original black-box classification network, P org [mix_classes] means to intercept the prediction results of the selected confusion category, mix_classes is a length of N mix A list of category indexes.
[0074] Step S6, performing interpretable visual concept intervention based on the prediction explanation of the concept bottleneck model on the data set;
[0075] In some embodiments, step S6 specifically includes:
[0076] According to the prediction results of the locally approximated CBM (concept bottleneck model) on the data basis (validation set), the category gradient activation vector and Act of the true positive samples of the selected confusion category are calculated. Tp And the category gradient activation vector and Act of the false positive sample Fp , Act Tp Sort in ascending order, Act Fp Sort in descending order and set the number of intervention concepts N ic ,Act Fp and Act Tp The first N ic The concept weight is reset to 0 (eliminated). and They represent the cumulative contribution scores of concepts correctly classified into a selected confusion category and the cumulative contribution scores of concepts incorrectly classified into other categories, respectively. The calculation method is as follows:
[0077]
[0078] Where N is the total number of samples, i is the sample index, is the y-category column vector of the concept weight gradient matrix corresponding to the CBM output category y (true category) of sample i, is the y category column vector of the concept weight matrix, is the pred category column vector of the concept weight gradient matrix corresponding to the CBM output category pred (wrong prediction category) of sample i, is the pred category column vector of the concept weight matrix, is the matching score vector of the image block concept and the concept bottleneck extracted by the middle layer of the original black box classification network for the sample input with index i, and is calculated as follows:
[0079]
[0080] Where E img is the image encoder of OpenAI-CLIP, E text is the text encoder of OpenAI-CLIP, N p is the number of extracted image block concepts, is the kth image block concept extracted by the middle layer of the original black box classification network corresponding to sample i as input, and C is N in the concept bottleneck c Natural language concepts.
[0081] In step S7, the concept bottleneck model after concept intervention and the frozen copy of the original black-box classification network are used as teacher models to reversely distill the original black-box classification network on the dataset and transfer the error correction knowledge.
[0082] In some embodiments, step S7 specifically includes:
[0083] The CBM after intervention (output is N mix Confusion categories) and the original black box classification network (output is all n categories, intercepting other categories except the confusion categories, which is a copy of the original network and the parameters are frozen) as the teacher model, and the original black box classification network as the student model, and knowledge distillation is performed. The distillation loss is as follows:
[0084]
[0085] Among them, MCE is the multivariate cross entropy loss, is the softmax output of the original black box classification network and the distillation temperature is T s-org , is the joint softmax output of the concatenation of the intervening CBM and the frozen copy of the original black-box classification network, and the distillation temperature of the intervening CBM is Tt-cbm , the distillation temperature of the frozen copy of the original black-box classification network is T t-org , the splicing process is as follows:
[0086]
[0087] Among them, logits t-cbm and logits t-org are the unnormalized score outputs of the frozen copies of the intervened CBM and the original black-box classification network, respectively, and [~mix_classes] refers to the softmax output of the non-confused categories of the frozen copy of the original black-box classification network.
[0088] This black-box classification interactive intervention method uses a concept bottleneck model with an interpretable structure as an interactive link between the black-box classification network's understanding of classification decisions and the human recognition model's understanding of samples and their categories. The combination of unsupervised concept extraction methods and LLMs saves this method the additional cost of mapping function training and annotation costs. In addition, for the selection of unsupervised concept extraction methods, in addition to using the CRAFT method as a visual concept extraction module, other methods such as ACE and MOCE can also be selected. However, other methods similar to ACE only provide a conceptual explanation of "Why is this classification result?" It is not easy for humans to understand the simple use of concept explanations for text-image interaction, while the CRAFT method also gives a concept attribution explanation, telling users which concepts the black-box classification model relies on and at which positions in the input image to infer the classification results. On the other hand, CRAFT's concept extraction based on non-negative matrix decomposition is more balanced among methods such as ACE and PCA, and the extracted concepts are more accepted by people, and the algorithm is faster.
[0089] This black-box classification interactive intervention method uses the knowledge distillation method to transfer model knowledge and completes the local approximation of the black-box classification network's confused categories by CBM. Users can use the image block concepts of the middle layer of the original black-box classification network and the concept weight matrix of CBM to clearly and accurately judge the errors and false correlations generated by the black box in the decision of the confused categories in the human recognition mode. In addition to the CBM interpretable structure, other forms of network structures such as graph neural networks and decision trees can also provide approximate explanations and interventions. However, the selection of intervention concepts of CBM is more intuitive and convenient because natural language concepts are easier for humans to understand. Graph neural networks tend to use visual concept embeddings and their relationship weights to express reasoning logic. Decision trees can only provide reasoning paths from root nodes to leaf nodes, and their explanations are more abstract than the previous two. From the perspective of intervention methods, CBM is more efficient, fast and easy to explain in modifying concept weights, and the selection of intervention concepts is based on the global gradient contribution score of category concepts (Act Tp and ActFp ). On the contrary, other methods mostly modify concepts by replacing or deleting them based on prior knowledge, which may cause artificial classification bias.
[0090] Based on the classification decision errors caused by the black-box classification network's reliance on certain concepts discovered by users, the CBM's human-in-the-loop intervention feature makes it easy to remove (from the concept weight matrix) these "bad" concepts and correct the decision errors. This black-box classification interactive intervention method splices the probability distribution of the intervened CBM (confused categories) and the frozen copy of the original black-box classification network (non-confused categories), and reversely distills the original black-box classification network, thereby using the correct decision concepts to tune the black-box classification network parameters, so as to achieve the purpose of improving the classification performance, generalization performance and interpretability of the original black-box classification network.
[0091] This embodiment also provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the black box classification interactive intervention method in the above-mentioned embodiment. Computer-readable storage media may include but are not limited to magnetic storage devices, optical disks, digital versatile disks, smart cards, and flash memory devices. In addition, the readable storage medium of the present invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" includes but is not limited to wireless channels and various other media (and / or storage media) that can store, contain and / or carry codes and / or instructions and / or data.
[0092] This embodiment also provides a computer program product. When the computer program product runs on a computer, it enables the computer to execute the above-mentioned related steps to implement the black box classification interactive intervention method in the above-mentioned embodiment.
[0093] The foregoing is considered to be merely illustrative of the principles of the present invention. Furthermore, since many modifications and variations will readily occur to those skilled in the art, it is not desirable to limit the present invention to the exact structure and operation shown and described. Although preferred embodiments have been described, the details may be changed without departing from the essential concept of the present invention. Therefore, the technical solutions involved in the present invention include not only the technical solutions disclosed above, but also the technical solutions consisting of equivalent replacements of the technical features in the technical solutions involved in the present invention. The unfinished matters of the present invention belong to the common knowledge of those skilled in the art.
Claims
1. A black box classification interactive intervention method, characterized in that: The following steps are involved: Analyze the dataset of the classification task to generate concept bottlenecks through deep learning neural networks; Using a black box classification network trained by the data set, performing prediction and testing on the data set, generating a category confusion matrix based on confusion categories, using a depth-first search algorithm to traverse all confusion category combinations, sorting the confusion category combinations by confusion scores, selecting the confusion category combination with the highest confusion score and using it as the target confusion category combination; Based on the concept bottleneck and the target confusion category combination, construct a concept bottleneck model; Selecting object image samples correctly classified by the original black box classification network from the training set of the data set, randomly cropping the object image samples and inputting them into the original black box classification network and processing them using an unsupervised concept extraction method to obtain visual concepts of the intermediate layer of the input samples; Taking the visual concept as input, using the concept bottleneck model to perform local distillation approximation on a black box classification network trained by the dataset, and outputting an interpretable visual concept; performing the explainable visual concept intervention based on the prediction explanation of the concept bottleneck model on the data set; The concept bottleneck model after concept intervention and the frozen copy of the original black-box classification network are used as teacher models, and the original black-box classification network is reversely distilled on the dataset to transfer the error correction knowledge.
2. The black box classification interactive intervention method according to claim 1, characterized in that: The specific steps to generate the concept bottleneck are as follows: Step S11, using LLMs according to prompt templates corresponding to multiple categories in the data set, to generate multiple concept candidates for each category; In step S12, all categories are first preliminarily screened, and the image embedding of each category is obtained using the image encoder. Then, all concept candidates of each category are processed by the text encoder and embedded into the same vector space. The mean similarity score between the concept embedding in each category and all image embeddings under the corresponding category is calculated to determine the final screening score of each concept candidate under the category. Step S13, arranging the concept candidates under each category in descending order according to the screening scores, taking the first multiple concept candidates of each category, and merging the concept candidates of the taken category to form a concept bottleneck.
3. The black box classification interactive intervention method according to claim 1, characterized in that: The confusion score is the sum of the average values of the confusion samples between nodes in each confusion category combination and then divided by the total number of confusion category combinations to obtain the average value.
4. The black box classification interactive intervention method according to claim 3, characterized in that: The specific steps for generating the visual concept are as follows: Step S41, selecting object picture samples correctly classified by the original black box classification network from the training set of the data set as image concept candidates; Step S42, for each type of image in the image concept candidate, randomly crop it and input it into the original black box classification network, and obtain the feature matrix of the middle layer, use the non-negative matrix decomposition method to decompose the feature matrix and set the number of feature bases, obtain the feature basis matrix and its projection matrix, and after algorithm iteration of the non-negative matrix decomposition method, obtain multiple groups of feature bases, each group of feature basis matrices can restore the image block concept of each category to the greatest extent through linear combination, that is, the visual concept of the middle layer of the input sample.
5. The black box classification interactive intervention method according to claim 4, characterized in that: Generating the visual concept specifically includes the following steps: Step S43, for any complete object sample image input, the non-negative matrix decomposition can be restored by using the retained feature basis matrix of the corresponding output prediction category and the feature matrix of the middle layer of the black box classification network to obtain the image block concept corresponding to the input sample.
6. The black box classification interactive intervention method according to claim 1, characterized in that: The concept bottleneck model is used to perform local distillation approximation on the black box classification network trained by the dataset. The specific steps are as follows: Using the original black box classification network as the teacher model, the constructed concept bottleneck model is locally distilled on the validation set in the dataset according to the selected confusion categories. The distillation loss is as follows: Loss D-cbm =MSE(P cbm ,P org [ mix_classes) Among them, MSE is the mean square error loss function, P cbm ∈ [ 0,1 ] (Nmix , 1) is the prediction output of the concept bottleneck model, P org is the prediction output of the original black-box classification network, P org [mix_classes] means to intercept the prediction results of the selected confusion category, mix_classes is a length of N mix A list of category indexes.
7. The black box classification interactive intervention method according to claim 4, characterized in that: Based on the prediction explanation of the concept bottleneck model on the data set, the explainable visual concept intervention is performed, and the specific steps are as follows: According to the prediction results of the validation set of the local approximated concept bottleneck model in the data set, the category gradient activation vectors and of the true positive samples of the selected confusion category and the category gradient activation vectors and of the false positive samples are calculated, and the category gradient activation vectors and of the true positive samples are arranged in ascending order, and the category gradient activation vectors and of the false positive samples are arranged in descending order. Then, the number of intervention concepts is set, and the weights of the first predetermined concepts in the activation vector and are reset to 0; for the selected confusion category, the cumulative concept contribution scores of the correctly classified concepts and the cumulative concept contribution scores of the incorrectly classified concepts into other categories are calculated respectively; the matching score vector of the image block concept extracted by the middle layer of the original black box classification network and the concept bottleneck is determined by the image and text encoder, wherein the image block concept is the visual concept of the middle layer of the input sample.
8. An electronic device, characterized in that: include: one or more processors; Memory; And one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the black box classification interaction intervention method as described in any one of claims 1-7.
9. A computer-readable storage medium comprising a computer program, characterized in that: When the computer program runs on an electronic device, the electronic device executes the black-box classification interactive intervention method as described in any one of claims 1 to 7.