A human-model interaction explanation guiding method based on a visual concept map representation, an electronic device and a storage medium

Abstract

The application discloses a kind of based on visual concept map representation Man-model interaction explanation guide method, electronic equipment and storage medium, the method includes: with model as foundation extraction important image block as category concept, constructs structure concept map, to this as tool representation arbitrary image, again build feature flow transparent inference network simulation existing classification network, observe the contribution of each part of concept map, to realize the explainability of decision;On the other hand, people independently each component as a concept, attribute arrangement is carried out to component, and structure component diagram is constructed in combination with component relationship.Based on this, people can update structure concept map with the structure component diagram of category, retrain inference network, and then guide knowledge to original network by knowledge distillation.Based on this two-way interaction method, the application can provide reasoning logic that people can understand by model, realize the explainability of neural network;Meanwhile, allow people to provide feedback to network using prior knowledge, to improve the performance of network.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and artificial intelligence technology, and specifically relates to a human-model interaction explanation and guidance method based on visual concept map representation, an electronic device, and a storage medium. Background Technology

[0002] In recent years, the research and application of neural networks have become increasingly widespread, playing a crucial role in numerous fields. Given the broad application and importance of machine learning systems, ensuring the safety and reliability of these models has become particularly urgent. However, currently, neural networks still largely lack interpretability and transparency; they are essentially "black boxes," unable to provide explanations for decisions from a human-understandable perspective. Furthermore, humans cannot participate in the network's decision-making process, using prior knowledge and reasoning abilities to correct potential errors or improve the model to enhance network performance. Therefore, the interpretability of neural networks and error correction and performance improvement remain two key challenges currently facing machine learning models.

[0003] Research on the interpretability of neural networks has made some progress, such as a gradient-based interpretability method. This method calculates the impact of perturbations in each pixel on the final result using gradients, selects a set of pixels important to the decision, and thus provides an explanation of the decision from a pixel perspective. However, this type of method is limited to explaining low-level relationships and cannot intuitively demonstrate the network's decision-making logic from a human-understandable perspective. More importantly, it does not provide a channel for knowledge exchange between humans and the network, meaning humans cannot truly influence the network's decisions.

[0004] Therefore, questions such as how to construct a channel between humans and neural networks so that they can exchange knowledge effectively through a rule-based "language" structure, and how humans can use this structured language to "communicate," understand the network's decision-making logic, and guide the original neural network with knowledge, still need to be considered and explored. Summary of the Invention

[0005] To overcome the shortcomings of the existing technology, the present invention aims to provide a human-model interaction interpretation guidance method, electronic device, and storage medium based on visual concept map representation. The visual concept map serves as the "language" for knowledge exchange between humans and models. By observing and tracking visual concepts, the model provides humans with easily understandable decision-making reasoning logic, achieving the interpretability of neural networks. At the same time, it allows humans to provide feedback to the network through their own knowledge, thereby improving the network's performance and decision accuracy, and realizing two-way interaction between humans and models.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A human-model interaction explanation and guidance method based on visual concept map representation includes the following steps;

[0008] S1, Visual concept extraction: Based on multiple images for each category, a top-down gradient attention algorithm is used to filter out regions important to the network's classification decision. Then, a multi-resolution segmentation algorithm is used to obtain image patches, which are uniformly adjusted to the required size. The image patches are then converted into feature vectors by a given classification network and clustered in the vector space. Finally, a concept scoring method is used to obtain the importance score of the potential visual concepts in each cluster, thus obtaining the most important visual concepts for each category. Here, the visual concepts are represented as a set of pixels, i.e., image patches containing object parts. At the same time, the average concept feature vector of each visual concept is stored.

[0009] S2 uses an existing classification network to make classification decisions on the input random images for recognition;

[0010] S3, for the input image described in S2, a multi-resolution segmentation algorithm is used to obtain image patches. The features of the image patches are calculated and compared with the average concept feature vectors of the most important visual concepts for each category selected in S1. Concept matching is performed on the specific image, and finally, multiple structural concept graphs for different categories are generated for the same input image. Visual concepts are used as nodes of the structural concept graphs, and structural relationships between visual concepts are used as edges of the structural concept graphs. Then, graph convolution is used to learn the influence of visual concepts and their relationships on the final decision for each structural concept graph. All nodes and edges are concatenated into a vector, and the decision vector for all categories of interest is output through multilayer perceptron technology, thus completing the training process of the inference network. Finally, by observing the flow of each node and each edge in the structural concept graph and its contribution to the decision, an easily understandable decision explanation is given, completing the decision explanation process.

[0011] S4, based on multiple images of each category, segments each category object by component, treats each component as a visual concept, organizes and encodes each component's color, shape, quantity, and some special appearance attributes as node features, selects the most important component concepts, and constructs a structural component diagram through the spatial position and adjacency relationships between components.

[0012] In step S5, the structural concept graph in S3 is updated using the structural component graph obtained in S4. A new network is then trained using the modified data. This new inference network serves as teacher model 1, retraining the original classification network for classes where decisions were incorrect. The original classification network is used as teacher model 2, training the remaining classes. Simultaneously, a network with the same architecture as the original neural network but initialized using the original neural network's weights is constructed as the student model. Through knowledge distillation, the two teacher models provide soft labels to guide the original neural network's overall knowledge, while using the true categories as hard labels to provide stronger constraints. This approach enhances the neural network's correct understanding of misclassified categories by leveraging prior human knowledge without altering the original neural network structure, thereby improving the network's classification accuracy.

[0013] Furthermore, the visual concept extraction process includes a region constraint module, an image patch segmentation module, a vector space clustering module, and a visual concept scoring module;

[0014] The region constraint module is used to filter out the foreground portion, providing space for concept extraction;

[0015] The image block segmentation module is used to segment and obtain concept candidate blocks from multiple perspectives, including texture and shape.

[0016] The vector space clustering module is used to perform preliminary screening of the relevance and importance of image patches;

[0017] The visual concept scoring module is used to select the most important concepts for a specific category, thereby forming a visual concept library.

[0018] Furthermore, in step S3, the inference network training process uses knowledge distillation to simulate the decision-making of the original classification network, with the loss function being:

[0019] L=‖σ(g(H))-σ(F(I))‖1

[0020] Where σ is the normalization operation, g is the inference network, and H = {h1, h2, ... h} m} is a set of m constructed structural concept maps, where m is the total number of categories of interest, F is the classifier of the original classification network, and I is the input image.

[0021] Furthermore, the decision interpretation process is implemented based on gradient contribution calculation, specifically: for each class of interest c, there is a class prediction result p. c Calculate p c The gradient of the graph embedding relative to all hypotheses is given by the following formula:

[0022]

[0023] grad i That is, to indicate the hypothesis h i The contribution weight vector, where h i For the structural concept graph of the i-th class, G is a graph network module of the inference network. Then, for each hypothesis h... i Contribution score s i That is, grad i and G i (h i The weighted sum of )

[0024]

[0025] Use the calculated contribution score s i Furthermore, it indicates the positive or negative contribution of each concept or its spatial relationship to the decision made by the neural network.

[0026] Furthermore, in S3, concept matching is performed on specific images: if the Euclidean distance between the features of an image patch and the average feature vector of a concept is less than a specific threshold τ, the image patch is identified as a concept; for undetected concepts, virtual nodes are used to represent them, and finally, multiple structural concept graphs for different classes are generated for the same input image.

[0027] Furthermore, in S4, the process of constructing the structural component diagram extracts concepts based on components: for example, a pistol includes components such as the barrel, slide, receiver, grip, magazine, and trigger. Each component is treated as a visual concept, and each visual concept is a part with actual physical meaning. Moreover, the attributes and relationships of components can be fully explored and embedded by humans, so that it is fully imbued with human prior knowledge and rational logic.

[0028] Furthermore, in S5, the method for updating the structural concept diagram from the structural component diagram is as follows: For each node n in the structural concept diagram S... i Traverse all nodes in the structural component diagram S′ and find the node in S′ that corresponds to n. i The node with the closest features is used to pair n. i Perform the replacement and update; repeat this process until all nodes in S have been traversed; after traversing all nodes, traverse each edge e in S. i The same search, replace, and update process is then performed on it. The result is an updated category-specific structural concept map incorporating prior human knowledge.

[0029] Furthermore, in step S5, the overall knowledge guidance process for the original neural network is implemented through knowledge distillation, and the overall objective loss function is as follows:

[0030] Loss = αLosssoft +βLoss hard

[0031] Where α and β are the loss during the distillation process. soft and Loss hard The weights of each item, Loss soft Let be the first loss function, and This represents the cross-entropy between the combined soft labels and the probability distribution of the student model's output. Where N represents the total number of categories in the original neural network. The probability value of class c represents the combination of soft labels from the two teacher models. For temperature T s The probability value of class c predicted by the student network is as follows:

[0032]

[0033]

[0034]

[0035] Where z c , T1 and T2 represent the inputs to the student network, teacher model 1, and teacher model 2, respectively; T1 and T2 represent the temperature values ​​used by teacher model 1 and teacher model 2 during knowledge distillation, respectively; S represents the set of total categories in the original neural network, and N = |S| represents the total number of categories. p Represents the set of modified classes, n = |S p | Represents the number of modified classes; This represents the probability ratio of the n modified classes relative to all classes N in the original neural network.

[0036] Loss hard For the second loss function, and represents the cross-entropy between the true class label of the image and the probability distribution output by the student model at T=1. Where g c For the true category label of class c, Let be the probability value of class c in the student network prediction when T=1, and

[0037] Another objective of this invention is to provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus.

[0038] Memory, used to store computer programs;

[0039] The processor, when executing a program stored in memory, implements the above method steps.

[0040] Another objective of this invention is to provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps.

[0041] The beneficial effects of this invention are:

[0042] (1) This invention extracts concepts and structures images of categories, and constructs a reasoning module based on graph networks to simulate known networks. It then transparently tracks features in the structure graph to discover the contribution of different concepts to the decision-making process, thereby providing a conceptual-level explanation of the original network's classification decisions. This overcomes the problem of existing "black box" networks failing to provide interpretability and improves low-level pixel-based explanations, making this invention more interpretable and presenting the logic from a human-understandable perspective.

[0043] (2) This invention achieves full mining of image feature attributes by manually segmenting categorized images into components, treating each component as a "concept," and manually extracting features from each component concept. Then, it interacts with the model in the form of a structural component diagram. By updating the structural concept diagram, this knowledge is fed back into the original neural network, strengthening the neural network's correct recognition of misclassifications, thereby improving the original neural network's classification ability and accuracy. In this process, human reasoning ability and prior knowledge are incorporated, realizing human-guided knowledge of the model.

[0044] (3) This invention does not modify the structure or size of the original neural network, but uses knowledge distillation technology to guide the model with overall knowledge as a tool to improve its performance. Therefore, it is not limited by the structural complexity of the original model and does not affect the deployment of the original network. Attached Figure Description

[0045] Figure 1 This is an overall framework diagram of the human-model interaction explanation and guidance method based on visual concept map representation in an embodiment of the present invention.

[0046] Figure 2 This is a schematic diagram of visual concept extraction based on a model, according to an embodiment of the present invention.

[0047] Figure 3 This is a schematic diagram illustrating the construction of a reasoning network and the simulation and interpretation of decision-making on the original classification network according to an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram illustrating the overall knowledge guidance using knowledge distillation technology in an embodiment of the present invention. Detailed Implementation

[0049] The present invention will now be described in further detail with reference to the accompanying drawings.

[0050] like Figure 1 As shown, the human-model interaction explanation and guidance method based on visual concept map representation consists of two parts: the model provides reasoning explanations to the human in the form of a structural concept map, and the human provides overall knowledge guidance to the model in the form of a structural component diagram.

[0051] The model provides a graphical reasoning module that offers explanations in the form of a structural concept map, and includes the following steps:

[0052] Step 1: Based on multiple images for each category, extract important visual concepts related to that category, and represent the images as a structural concept map through the potential spatial structural relationships between these concepts, such as... Figure 2 As shown, the specific steps are as follows:

[0053] Step 11 involves selecting multiple images for each category to extract visual concepts. Then, a top-down gradient attention algorithm is used to filter out regions (mostly foreground) that are important to the network's decision-making, thus constraining the concept extraction regions and ensuring that the extracted concepts better represent the original network's understanding of that category. The method is as follows: Given an attention map M for a specific class, M is binarized using a selected threshold τ. (Pixel values ​​below τ are set to 0, and others are set to 1), then the processed image is generated using the following formula:

[0054]

[0055] Where I represents the original image. The image is the processed result; × represents element-wise multiplication.

[0056] Step 12: Within the specified area, the image is segmented using a multi-resolution segmentation algorithm to obtain image patches, which are then uniformly adjusted to the required size. The image patches are then converted into feature vectors using a given network and clustered in the vector space. Finally, the importance score of the potential visual concepts in each cluster is obtained using a concept scoring method. The most important visual concepts in each category are obtained, and the average concept feature vector of all visual concepts in each category is stored.

[0057] After extracting category-related visual concepts, we discover potential spatial structural relationships between them. Therefore, we use these visual concepts to construct a structural concept graph, where different nodes represent different visual concepts, and edges represent the spatial relationships between these concepts. Based on this, any image can be expressed as a structured concept graph corresponding to its category, serving as an important "language" for explaining neural network reasoning and decision-making.

[0058] Step 2, as follows Figure 3 As shown in the network decision module, an existing classification network is used to make classification decisions for a specific input image;

[0059] Step 3: Construct a graph reasoning module based on a graph network to make the feature flow more transparent, simulating the decision-making of the classification network in Step 2. Observe the flow of each node and edge in the structural concept graph and its contribution to the decision, thus providing an easily understandable explanation of the decision.

[0060] Step 31: For the input image, first construct an image-level structural concept graph following a process similar to Step 12: Given an image, use multi-resolution segmentation to obtain image patches, which are used as input to the original neural network. Calculate the features of the image patches and compare them with the average concept feature vectors of the most important visual concepts for each category selected in Step 12 to perform concept matching: If the Euclidean distance between the features of an image patch and the average feature vector of a concept is less than a specific threshold τ, the image patch is identified as a concept; undetected concepts are represented by virtual nodes. Then, using visual concepts as nodes and the spatial relationships between visual concepts as edges, construct a visual concept graph.

[0061] Therefore, for all m categories of interest, m structural concept maps for different categories were ultimately generated for the same input image.

[0062] Step 32, as follows Figure 3 As shown in the inference network module, graph convolution is used to learn the influence of visual concepts and relationships between them on the final decision for each structural concept graph. Important edges are selected, and the structural relationships between nodes are reconstructed. Finally, all points and edges are concatenated into a vector, and a multilayer perceptron technique is used to output the decision vector for all categories of interest.

[0063] Step 33, as Figure 3 As shown in the knowledge distillation module, knowledge distillation is used to ensure that the decisions of the concept graph reasoning network g are consistent with those of the original network, thus simulating the original network and providing decision premises for the final reasoning explanation. That is, we obtain:

[0064] g(H)=E(G(H))=F(I)

[0065] Where H = {h1, h2, ... h} m Let} be the set of m constructed structural concept graphs, where m is the total number of categories of interest, G is the graph network module of the inference network, e is the embedding network following the graph network, F is the classifier of the original classification network, and I is the original input image. The loss function constructed during knowledge distillation is:

[0066] L=‖σ(g(H))-σ(F(I))‖1

[0067] Here, σ represents the normalization operation. By minimizing the loss function, the inference network is trained to mimic the original classification network.

[0068] Step 34: Based on the transparent and easily traceable feature transfer of the concept reasoning network using a graph convolutional neural network, the contribution of each point and edge involved in the decision-making process to the network decision is calculated by observing the feature flow of the structural concept graph and using gradient-based analysis methods. Specifically, for each class of interest c, there is a class prediction result p. c Calculate p c The gradient of the graph embedding relative to all hypotheses is given by the following formula:

[0069]

[0070] grad i That is, to indicate the hypothesis h i The contribution weight vector, where h i This is a structural concept diagram for the i-th class. Then, for each hypothesis h... i Contribution score s i That is, grad i and G i (h i The weighted sum of )

[0071]

[0072] Use the calculated contribution score s i Furthermore, it indicates the positive or negative contribution of each concept or its spatial relationship to the decision made by the neural network.

[0073] On the other hand, humans provide prior knowledge to the model, creating a comprehensive knowledge guidance module for the original classification network, including the following steps:

[0074] Step 4: Manually segment the categorized images by components. Each component is treated as a visual concept. Each component is then categorized by color, shape, quantity, and other specific attributes, and encoded as node features. The most important component concepts are selected, and a structural component diagram is constructed based on the spatial relationships and adjacency relationships between components.

[0075] Step 41 uses an object-based concept construction method, leveraging appearance visibility and expert knowledge to segment the category objects by components. Each component is treated as a visual concept, and its color, shape, quantity, and other appearance-specific attributes are analyzed to extract the image data features as completely and thoroughly as possible. In this case, the visual concept is a physically meaningful part. An example is shown below: [

[0077] EVO3A1 submachine gun

[0078] Barrel: Cylindrical, black, 1 piece;

[0079] Gun body: Nearly rectangular, black, 1 trigger; two rows of button-like triggers at the front: "J" shaped, black, 1 trigger;

[0080] Stock: Scorpion tail shaped, black, 1 piece;

[0081] Magazine: Long, curved, semi-transparent, 1 piece; Plastic magazine grip: Nearly rectangular, black, 1 piece; Sloping rearward ]

[0083] Step 42: Encode the features of each component as feature vectors for nodes in the structural component diagram: For structured values ​​such as color, shape, and quantity, a one-bit effective encoding method can be used; for special attributes in text form, existing text encoding methods can be used, as shown in the following example:

[0084] [Long, curved | Semi-transparent | 1 piece | Plastic magazine]

[0085] ↓

[0086] [0010000|000100|00001|102, 3467, 908, 4597, 391]

[0087] Step 43: Using a concept importance scoring method similar to that in Step 12, the most important component concepts are selected, and then a structural component graph is constructed based on the spatial positional relationships and adjacency relationships between the components. Each node represents a component with component features represented in vector form, and each edge represents the adjacency and positional relationship between the two end components, thus providing a structural representation of the image from another perspective.

[0088] Step 5: When the neural network's decisions do not match human cognition, or when the neural network's classification performance is poor, humans can use their prior knowledge to modify the structural concept map, use the modified data to train a new network, and finally use this part of the data with human prior knowledge to train and constrain the original classification network through knowledge distillation.

[0089] Step 51: Humans evaluate the performance of the classification network. For categories that need modification, the structural concept map of the category is updated using the structural component map of that category.

[0090] Specifically, the process involves updating the category-specific structural concept graph S constructed in step 12 using the category structure component graph S′ constructed in step 43. The specific process is as follows: for each node n in S... iIterate through all nodes in S′ and find the node in S′ that is related to n. i The node with the closest features is used to pair n. i Perform the replacement and update; repeat this process until all nodes in S have been traversed; after traversing all nodes, traverse each edge e in S. i The same search, replace, and update process is then performed on it. The result is an updated category-specific structural concept map incorporating prior human knowledge.

[0091] Step 52: Because the structural concept map of the specific category has been modified, a new image-level structural concept map will be reconstructed in the inference network for each input image: similar to step 31, image patches are re-segmented, visual concepts are re-matched, and spatial structural relationships are relearned. This is used to train a new inference network that focuses on the modified category.

[0092] Step 53, as follows Figure 4 As shown, after training is complete, this new inference network is used as teacher model 1 to focus on the modified class S. p Training; using the original classification network as teacher model 2, focusing on unmodified class S\S p The network is trained using the same neural network architecture as the original neural network, but initialized with the same weights. Two teacher models are then used together to provide soft labels via knowledge distillation. Simultaneously, the true image category is used as a hard label to provide stronger constraints, thereby improving the network's classification performance.

[0093] Furthermore, the overall objective loss function for the knowledge distillation process described in step 53 is as follows:

[0094] Loss = αLoss soft +βLoss hard

[0095] Where α and β are the loss during the distillation process. soft and Loss hard The weights of each item, Loss soft Let be the first loss function, and This represents the cross-entropy between the combined soft labels and the probability distribution of the student model's output. Where N represents the total number of categories in the original neural network. The probability value of class c represents the combination of soft labels from the two teacher models. For temperature T s The probability value of class c predicted by the student network is as follows:

[0096]

[0097]

[0098]

[0099] Where z c , T1 and T2 represent the inputs to the student network, teacher model 1, and teacher model 2, respectively; T1 and T2 represent the temperature values ​​used by teacher model 1 and teacher model 2 during knowledge distillation, respectively; S represents the set of total categories in the original neural network, and N = |S| represents the total number of categories. p Represents the set of modified classes, n = |S p | Represents the number of modified classes; This represents the probability ratio of the n modified classes relative to all classes N in the original neural network.

[0100] Loss hard For the second loss function, and represents the cross-entropy between the true class label of the image and the probability distribution output by the student model at T=1. Where g c For the true category label of class c, Let be the probability value of class c in the student's network prediction when T=1, and

[0101] This invention extracts key visual concepts for specific categories based on models, enabling structured graph representations of specific categories and images. This graph representation serves as the "language" for interaction between the model and humans, constructing a graph-based inference network that transparently tracks features, allowing humans to easily understand the network's decision-making logic.

[0102] This invention is based on human understanding of specific categories and images, extracting concepts using components as the standard. This imbues the concepts with actual physical meaning, and allows humans to fully and conveniently observe and extract features from each component concept, enabling comprehensive mining of image data features. Furthermore, compared to abstract, model-based visual concepts, this invention provides a higher-level method for structured image representation using components as concepts.

[0103] This invention utilizes knowledge distillation technology to guide the model with overall knowledge through structural component graphs without altering the original network structure. This enables the invention to achieve better prediction results in tasks such as image classification, and is not limited by the complexity of the original model structure or deployment method.

[0104] The aforementioned human-model interaction explanation and guidance method based on visual concept map representation can be implemented as a computer program, stored on a hard disk, and loaded into a processor for execution to implement the method of the present invention.

[0105] This invention also provides a computer-readable medium storing computer program code, which, when executed by a processor, implements the human-model interactive explanation and guidance method based on visual concept map representation as described above.

[0106] When a human-model interactive explanation and guidance method based on visual concept map representation is implemented as a computer program, it can also be stored as an article of manufacture in a computer-readable storage medium. For example, computer-readable storage media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic stripes), optical discs (e.g., compact discs (CDs), digital multifunction discs (DVDs)), smart cards, and flash memory devices (e.g., electrically erasable programmable read-only memory (EPROM), cards, sticks, key drives). Furthermore, the various storage media described in the embodiments of this invention can represent one or more devices and / or other machine-readable media for storing information. The term "machine-readable medium" may include, but is not limited to, wireless channels and various other media (and / or storage media) capable of storing, containing, and / or carrying code and / or instructions and / or data.

[0107] It should be understood that the above embodiments are merely illustrative. The embodiments described in this invention can be implemented in hardware, software, firmware, middleware, microcode, or any combination thereof. For hardware implementation, the processing unit can be implemented within one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and / or other electronic units designed to perform the functions described in this invention, or combinations thereof.

[0108] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for human-model interaction explanation guidance based on visual concept map representation, characterized in that, The method comprises the following steps: S1, visual concept extraction is performed: according to multiple images of each category, important areas for network classification decision are screened out by using top-down gradient attention, and then image blocks are obtained by using a multi-resolution segmentation algorithm, and the image blocks are uniformly adjusted to a required size, the image blocks are converted into feature vectors by using a given classification network, and clustering is performed in a vector space; finally, the importance scores of potential visual concepts of each cluster are obtained by using a concept scoring method, and several most important visual concepts of each category are obtained, and the visual concept is represented as a set of pixel points, that is, an image block containing an object part, and meanwhile, the average concept feature vector of each visual concept is stored; S2, a given classification network is used to make classification decisions on input random images for identification; S3, for the input images in S2, image blocks are obtained by using a multi-resolution segmentation algorithm, the features of the image blocks are calculated, and the average concept feature vectors of several most important visual concepts of each category screened out in S1 are compared, concept matching is performed on the specific images, and finally multiple structure concept graphs for different categories are generated for the same input image, the visual concepts are used as nodes of the structure concept graph, and the structural relationship between the visual concepts is used as an edge of the structure concept graph; then, graph convolution is used to learn the influence of the visual concepts and the relationship therebetween on the final decision for each structure concept graph, all nodes and edges are concatenated into a vector, and a decision vector for all interested categories is output by using a multilayer perception technique, so that the training process of the reasoning network is completed, finally, a human understandable decision explanation is given by observing the flow and contribution to the decision of each node and each edge in the structure concept graph, and the decision explanation process is completed; S4, based on multiple images of each category, objects of each category are segmented by parts, each part is taken as a visual concept, color, shape and quantity of each part are arranged, and coding is performed, so that node features are obtained, several most important part concepts are screened out, and a structure part graph is constructed by using the spatial position relationship and the adjacency relationship between the parts; S5, the structure concept graph in S3 is updated by using the structure part graph obtained in S4, a new network is trained by using the modified data, the new reasoning network is taken as a teacher model 1, and some categories with incorrect decisions of the original classification network are retrained; the original classification network is taken as a teacher model 2, and the remaining categories are trained; meanwhile, a network with the same structure as the original neural network and initialized by using the weights of the original neural network is taken as a student model; by using the knowledge distillation method, the two teacher models provide soft labels together, and the original neural network is guided by the whole knowledge, and meanwhile, the real categories are used as hard labels to provide stronger constraints, so that the prior knowledge of human beings is used to strengthen the correct understanding of the neural network for the wrong categories, and the classification accuracy of the network is improved; in S5, the whole knowledge guiding process of the original neural network is realized by using knowledge distillation, and the whole target loss function is as follows: in and During the distillation process and The weights of each item, Let be the first loss function, and , representing the cross-entropy between the combined soft labels and the probability distribution of the student model output, where This represents the total number of categories in the original neural network. The class representing the combination of soft labels from two teacher models The probability value, For temperature Student network prediction The probability value of a class is in the following form: in , , These represent the inputs to the student network, teacher model 1, and teacher model 2, respectively. These represent the temperature values ​​used by Teacher Model 1 and Teacher Model 2 during knowledge distillation, respectively. This represents the set of all categories in the original neural network. Indicates the total number of categories. Represents a collection of modified classes. Represents the number of modified classes; , indicating that in the original neural network The modified class is relative to all classes. The probability ratio; is a second loss function, and represents a cross-entropy between the image true class label and the probability distribution output by the student model at ; wherein is a true class label of the class , is a probability value of the class predicted by the student network at , and .

2. The method of claim 1, wherein, The visual concept extraction process comprises a region constraint module, an image block segmentation module, a vector space clustering module and a visual concept scoring module; The region constraint module is used for screening out a foreground part and providing a space for concept extraction; The image block segmentation module is used for segmenting a concept candidate block from multiple angles of texture and shape; The vector space clustering module is used for preliminarily screening the relevance and importance of image blocks; The visual concept scoring module is used for selecting a concept most important to a specific category to form a visual concept library.

3. The method of claim 1, wherein, In the S3, the training process of the reasoning network uses a knowledge distillation method to simulate the decision of the original classification network, and the loss function is: wherein is a normalization operation, is an inference network, is a constructed set of structural concept graphs, is the total number of classes of interest, is a classifier of the original classification network, is an input picture.

4. The method of claim 1, wherein, In the S3, the decision explanation process is realized based on gradient contribution calculation, specifically: for each class of interest , there is a class prediction result , the calculation The gradient of all assumed graph embeddings is as follows: i.e. the contribution weight vector of hypothesis where is the structure concept graph for the th class, is the graph network module of the reasoning network, after which each hypothesis is assigned a contribution score i.e. the weighted sum of and . using the computed contribution scores further indicate the positive or negative contribution of each concept or its spatial relationship to the decision made by the neural network.

5. The method of claim 1, wherein, In S3, the specific image is conceptually matched: if the Euclidean distance between a certain image block feature and the average concept feature vector of a certain concept is less than a certain threshold i.e. the image block is identified as a concept; for undetected concepts, a virtual node is represented, resulting in multiple structure concept graphs for different classes for the same input image.

6. The method of claim 1, wherein, In the S4, the process of constructing the structure component graph extracts concepts based on components: a pistol contains components such as a barrel, a sleeve, a body, a handle, a magazine and a trigger, each component is taken as a visual concept, at this time each visual concept is a part with actual physical meaning, and can be artificially embedded with complete component attributes and component relationships, so as to fully bring human prior knowledge and rational logic.

7. The method of claim 1, wherein, In step S5, the method for updating the structural concept diagram from the structural component diagram is as follows: For the structural concept diagram... Each node in Traverse the structural component diagram Find all nodes in Zhongyu The node with the closest features is used to pair... Perform the replacement and update; repeat this process until all iterations are complete. All nodes in the list; after traversing all nodes, then traverse the list again. Each edge in The same search, replace, and update process is then performed on it; finally, an updated category-specific structural concept map with human prior knowledge is obtained.

8. An electronic device, comprising: The device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus. The memory is used for storing a computer program. The processor is used for executing the program stored in the memory to realize the method steps in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to realize the method steps in any one of claims 1-7.

Images